Skip to content

armgit / macro

Public Imported from GitHub
0

Macro is a unified workspace for teams: email, chat, docs, tasks, agents, calls, and CRM — @-linked together with shared AI memory.

feat(email): gate every Gmail request per mailbox (#8772)

teoc59227cadfb1
diff --git a/Cargo.lock b/Cargo.lock
index c1670e3cbc..2e7a5305db 100644
--- a/Cargo.lock
+++ b/Cargo.lock
@@ -3120,6 +3120,15 @@ dependencies = [
"tokio",
]
+[[package]]
+name = "backon"
+version = "1.6.0"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "cffb0e931875b666fc4fcb20fee52e9bbd1ef836fd9e9e04ec21555f9f85f7ef"
+dependencies = [
+ "fastrand 2.4.1",
+]
+
[[package]]
name = "base16ct"
version = "0.2.0"
@@ -7393,6 +7402,7 @@ dependencies = [
"models_properties",
"notification",
"once_cell",
+ "rand 0.9.4",
"redis",
"regex",
"reqwest 0.13.4",
@@ -16214,12 +16224,15 @@ version = "1.2.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "72d32a1ac9123f0d84fda64bfc02a271d9868483162dd2d9099b5c362ece064c"
dependencies = [
+ "arc-swap",
"arcstr",
"async-lock",
+ "backon",
"bytes",
"cfg-if",
"combine",
"crc16",
+ "futures-channel",
"futures-util",
"itoa",
"log",
diff --git a/crates/email/src/domain/mailbox/test.rs b/crates/email/src/domain/mailbox/test.rs
index 3accbfd03b..121dbba08b 100644
--- a/crates/email/src/domain/mailbox/test.rs
+++ b/crates/email/src/domain/mailbox/test.rs
@@ -6,7 +6,7 @@ use std::time::Duration;
fn provider_backoff_is_preserved_and_auth_failure_does_not_hot_loop() {
let error = EmailApiError::RateLimited {
retry_after: Some(Duration::from_secs(73)),
- origin: RateLimitOrigin::Provider,
+ origin: RateLimitOrigin::Provider { concurrency: false },
};
assert_eq!(retry_seconds(&error), 73);
assert!(retry_seconds(&EmailApiError::AuthRequired) >= 300);
diff --git a/crates/email_api_client/AGENTS.md b/crates/email_api_client/AGENTS.md
index d826233a69..0f9f294351 100644
--- a/crates/email_api_client/AGENTS.md
+++ b/crates/email_api_client/AGENTS.md
@@ -9,11 +9,12 @@ Outlook (Microsoft Graph) outbound adapters.
## Start here
- `src/domain/service.rs`: `EmailApiClientServiceImpl<R, T, L>` — the
- orchestration layer, generic over a repository, token source, and rate
- limiter port; submodules `attachments`/`blocklist`/`contacts`/`labels`/
+ orchestration layer, generic over a repository, token source, and request
+ gate port; submodules `attachments`/`blocklist`/`contacts`/`labels`/
`messages`/`send`/`subscription`/`sync` hold one capability each.
-- `src/domain/ports.rs`: `ProviderRateLimiter`, `ProviderTokenSource`, and the
- repository trait adapters implement.
+- `src/domain/ports.rs`: `ProviderRequestGate` (per-mailbox admission for the
+ service above), `ProviderRateLimiter` (the Outlook `MailboxApiService`'s),
+ `ProviderTokenSource`, and the repository traits adapters implement.
- `src/outbound/gmail.rs`: the Gmail adapter (feature `outbound-gmail`).
- `src/outbound/outlook.rs`: the Microsoft Graph adapter (feature
`outbound-outlook`); every Graph call goes through `outlook/transport.rs`,
@@ -47,13 +48,24 @@ Outlook (Microsoft Graph) outbound adapters.
## Gotchas
- `#![deny(missing_docs)]`.
-- The service is generic over `<R, T, L>` (repository, token source, rate
- limiter) — add a new dependency as a port + generic bound, not a concrete
+- The service is generic over `<R, T, L>` (repository, token source, request
+ gate) — add a new dependency as a port + generic bound, not a concrete
field.
+- Every provider call holds a `ProviderRequestGate` permit (`RequestPermit`)
+ for its duration and finishes it with the call's error, so a provider rate
+ limit cools the mailbox down for every caller. A permit dropped unfinished,
+ by a cancelled future or an undispatched `PreparedSendMessage`, releases
+ itself. New service methods go through `read_with_token` or
+ `call_with_token`; a call made with a bare token is invisible to the
+ in-flight cap. Their spans declare `gate_in_flight` and `gate_cap` as
+ empty fields, which admission fills with the mailbox's load so it sits next
+ to the call's outcome.
- Link-based reads go through `read_with_token`, which retries once with a
fresh token when the provider answers `AuthRequired`; a cached token can
outlive a revoked grant, and only the fresh fetch lets the token source mark
- the link for reauthorization. Writes use `prepare` and do not retry.
+ the link for reauthorization. Contact listings use `retry_with_fresh_token`
+ to keep their WARN-level logging. Writes use `call_with_token` and do not
+ retry.
- Graph answers a bodyless POST or PATCH without `Content-Length: 0` with 411
Length Required. `transport.rs` adds the header, so new Graph calls must go
through `request`/`conditional_request` rather than the raw client.
diff --git a/crates/email_api_client/src/domain/models/error.rs b/crates/email_api_client/src/domain/models/error.rs
index 83b1695c6a..c0722f13c8 100644
--- a/crates/email_api_client/src/domain/models/error.rs
+++ b/crates/email_api_client/src/domain/models/error.rs
@@ -72,15 +72,19 @@ impl EmailApiError {
/// Where a rate-limit refusal originated.
///
-/// Callers use this to distinguish a local budget refusal (no provider quota
-/// was consumed; a drop-and-wait policy may be appropriate) from the provider
-/// itself throttling the request (retrying preserves delivery guarantees).
+/// Callers use this to distinguish a local refusal (no provider request was
+/// made, so the mailbox needs no cooldown) from the provider itself throttling
+/// the request. Both defer the work; neither drops it.
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub enum RateLimitOrigin {
/// Refused by local quota policy before any provider call was made.
Local,
/// The provider itself throttled the request.
- Provider,
+ Provider {
+ /// The provider refused because the mailbox had too many requests in
+ /// flight at once, rather than for a quota.
+ concurrency: bool,
+ },
}
impl From<RateLimitRefusal> for EmailApiError {
diff --git a/crates/email_api_client/src/domain/models/mod.rs b/crates/email_api_client/src/domain/models/mod.rs
index 3d6b2c7416..398a57bd43 100644
--- a/crates/email_api_client/src/domain/models/mod.rs
+++ b/crates/email_api_client/src/domain/models/mod.rs
@@ -8,6 +8,7 @@ mod error;
mod mailbox;
mod mailbox_settings;
mod operation;
+mod permit;
mod provider;
mod send;
@@ -19,6 +20,7 @@ pub use error::{EmailApiError, RateLimitOrigin, RateLimitRefusal};
pub use mailbox::{CalendarPart, MessageWithCalendarParts, ProviderSubscription, ThreadListPage};
pub use mailbox_settings::*;
pub use operation::ApiOperationKind;
+pub use permit::{MailboxLoad, RequestPermit};
pub use provider::*;
#[cfg(feature = "ports")]
pub use send::PreparedSendMessage;
diff --git a/crates/email_api_client/src/domain/models/permit.rs b/crates/email_api_client/src/domain/models/permit.rs
new file mode 100644
index 0000000000..cbaf3f8df6
--- /dev/null
+++ b/crates/email_api_client/src/domain/models/permit.rs
@@ -0,0 +1,72 @@
+use uuid::Uuid;
+
+type Release = Box<dyn FnOnce(Uuid, Uuid) + Send + Sync>;
+
+/// How busy a mailbox was when the gate admitted a request.
+#[derive(Debug, Clone, Copy, PartialEq, Eq)]
+pub struct MailboxLoad {
+ /// The mailbox's requests in flight, counting this one.
+ pub in_flight: u32,
+ /// The mailbox's in-flight cap when the request was admitted.
+ pub cap: u32,
+}
+
+/// Admission for one provider request. A permit dropped before its gate
+/// finishes it, by a cancelled caller or an abandoned send, hands itself to
+/// the gate's release instead of holding its slot until the lease expires.
+pub struct RequestPermit {
+ link_id: Uuid,
+ id: Uuid,
+ load: Option<MailboxLoad>,
+ release: Option<Release>,
+}
+
+impl RequestPermit {
+ /// Creates a permit that runs `release` if it is dropped unfinished.
+ pub fn new(
+ link_id: Uuid,
+ id: Uuid,
+ release: impl FnOnce(Uuid, Uuid) + Send + Sync + 'static,
+ ) -> Self {
+ Self {
+ link_id,
+ id,
+ load: None,
+ release: Some(Box::new(release)),
+ }
+ }
+
+ /// Records how busy the mailbox was when this request was admitted.
+ pub fn with_load(mut self, load: MailboxLoad) -> Self {
+ self.load = Some(load);
+ self
+ }
+
+ /// How busy the mailbox was at admission, when the gate tracks it.
+ pub fn load(&self) -> Option<MailboxLoad> {
+ self.load
+ }
+
+ /// The mailbox this permit admits a request for.
+ pub fn link_id(&self) -> Uuid {
+ self.link_id
+ }
+
+ /// The gate's identifier for this admission.
+ pub fn id(&self) -> Uuid {
+ self.id
+ }
+
+ /// Records that the gate has finished this permit, so dropping it does nothing.
+ pub fn disarm(mut self) {
+ self.release = None;
+ }
+}
+
+impl Drop for RequestPermit {
+ fn drop(&mut self) {
+ if let Some(release) = self.release.take() {
+ release(self.link_id, self.id);
+ }
+ }
+}
diff --git a/crates/email_api_client/src/domain/models/send.rs b/crates/email_api_client/src/domain/models/send.rs
index e0ed6c91b2..18b38c86bf 100644
--- a/crates/email_api_client/src/domain/models/send.rs
+++ b/crates/email_api_client/src/domain/models/send.rs
@@ -86,8 +86,12 @@ impl SendRequest {
/// Locally prepared delivery. Tokens and message contents cannot be logged via
/// `Debug`, and callers cannot construct it without completing preparation.
+///
+/// It holds the send's admission permit until dispatch finishes it; a delivery
+/// dropped without dispatch releases the permit.
#[cfg(feature = "ports")]
pub struct PreparedSendMessage {
+ pub(crate) permit: super::RequestPermit,
pub(crate) access_token: AccessToken,
pub(crate) mime: Vec<u8>,
pub(crate) provider_thread_id: Option<String>,
diff --git a/crates/email_api_client/src/domain/ports.rs b/crates/email_api_client/src/domain/ports.rs
index 87c3e8739d..b1e3b29d89 100644
--- a/crates/email_api_client/src/domain/ports.rs
+++ b/crates/email_api_client/src/domain/ports.rs
@@ -13,8 +13,8 @@ use uuid::Uuid;
use super::models::{
AccessToken, ApiOperationKind, CalendarPart, ChangeBatch, EmailApiError,
- MessageWithCalendarParts, ProviderSubscription, RateLimitRefusal, SendRequest, SentIds,
- SyncCursor, ThreadListPage, TokenError, TokenFreshness,
+ MessageWithCalendarParts, ProviderSubscription, RateLimitRefusal, RequestPermit, SendRequest,
+ SentIds, SyncCursor, ThreadListPage, TokenError, TokenFreshness,
};
#[cfg(test)]
@@ -315,6 +315,27 @@ pub trait ProviderRateLimiter: Send + Sync + 'static {
) -> impl Future<Output = Result<(), RateLimitRefusal>> + Send;
}
+/// Admits provider requests for a linked mailbox. An admitted request holds a
+/// permit until it completes. A permit dropped unfinished releases itself, and
+/// a crashed process's permits expire on their own.
+pub trait ProviderRequestGate: Send + Sync + 'static {
+ /// Admits one operation, or reports when the mailbox may try again. An
+ /// adapter may hold the caller briefly for room before refusing.
+ fn acquire(
+ &self,
+ link_id: Uuid,
+ operation: ApiOperationKind,
+ ) -> impl Future<Output = Result<RequestPermit, RateLimitRefusal>> + Send;
+
+ /// Releases a permit. A provider rate limit in `error` makes every caller
+ /// for the mailbox wait out a shared cooldown.
+ fn finish(
+ &self,
+ permit: RequestPermit,
+ error: Option<&EmailApiError>,
+ ) -> impl Future<Output = ()> + Send;
+}
+
/// Mailbox adapter used when provider capabilities are intentionally unavailable.
#[derive(Debug, Clone, Copy, Default)]
pub struct NoOpMailboxClient;
@@ -557,3 +578,17 @@ impl ProviderRateLimiter for AlwaysAllowRateLimiter {
Ok(())
}
}
+
+impl ProviderRequestGate for AlwaysAllowRateLimiter {
+ async fn acquire(
+ &self,
+ link_id: Uuid,
+ _: ApiOperationKind,
+ ) -> Result<RequestPermit, RateLimitRefusal> {
+ Ok(RequestPermit::new(link_id, Uuid::nil(), |_, _| {}))
+ }
+
+ async fn finish(&self, permit: RequestPermit, _: Option<&EmailApiError>) {
+ permit.disarm();
+ }
+}
diff --git a/crates/email_api_client/src/domain/service.rs b/crates/email_api_client/src/domain/service.rs
index 925fad096e..5076b841f7 100644
--- a/crates/email_api_client/src/domain/service.rs
+++ b/crates/email_api_client/src/domain/service.rs
@@ -3,9 +3,10 @@
use uuid::Uuid;
use super::models::{
- AccessToken, ApiOperationKind, EmailApiError, RateLimitOrigin, TokenError, TokenFreshness,
+ AccessToken, ApiOperationKind, EmailApiError, RateLimitOrigin, RequestPermit, TokenError,
+ TokenFreshness,
};
-use super::ports::{ProviderRateLimiter, ProviderTokenSource};
+use super::ports::{ProviderRequestGate, ProviderTokenSource};
mod attachments;
mod blocklist;
@@ -28,16 +29,16 @@ mod test_support;
pub struct EmailApiClientServiceImpl<R, T, L> {
repository: R,
token_source: T,
- rate_limiter: L,
+ gate: L,
}
impl<R, T, L> EmailApiClientServiceImpl<R, T, L> {
/// Creates an email API service from its provider and infrastructure ports.
- pub fn new(repository: R, token_source: T, rate_limiter: L) -> Self {
+ pub fn new(repository: R, token_source: T, gate: L) -> Self {
Self {
repository,
token_source,
- rate_limiter,
+ gate,
}
}
}
@@ -45,7 +46,7 @@ impl<R, T, L> EmailApiClientServiceImpl<R, T, L> {
impl<R, T, L> EmailApiClientServiceImpl<R, T, L>
where
T: ProviderTokenSource,
- L: ProviderRateLimiter,
+ L: ProviderRequestGate,
{
/// Acquires an access token for health probes or provider-adjacent work.
///
@@ -64,71 +65,135 @@ where
.map_err(map_token_error)
}
- /// Admits an operation and acquires its access token, logging a failure in
- /// the calling operation's span.
- async fn prepare(
+ /// Runs one provider call with the cached token, without retrying.
+ ///
+ /// Admission and token failures are logged in the calling operation's
+ /// span; the call's own errors are left to the caller.
+ async fn call_with_token<O, F, Fut>(
&self,
link_id: Uuid,
operation: ApiOperationKind,
- ) -> Result<AccessToken, EmailApiError> {
- // Check the cheap local limiter first: under throttling this avoids
- // paying the full token dance (SELECT + Redis + possible auth-service
- // refresh + health write) per refused attempt. The limiter does not
- // consume provider quota on denial.
- self.check_rate_limit(link_id, operation).await?;
-
- self.get_access_token(link_id, TokenFreshness::Cached)
- .await
- .inspect_err(log_operation_error)
+ call: F,
+ ) -> Result<O, EmailApiError>
+ where
+ F: FnOnce(AccessToken) -> Fut,
+ Fut: Future<Output = Result<O, EmailApiError>>,
+ {
+ self.attempt(
+ link_id,
+ operation,
+ self.get_access_token(link_id, TokenFreshness::Cached),
+ call,
+ )
+ .await?
}
/// Runs a provider read with the cached token, retrying once with a fresh
/// token if the provider rejects it, and logs a failure in the calling
/// operation's span.
///
+ /// Callers declare `retry_count = 0` on their span; a retry sets it to 1.
+ /// Like every provider call, they also declare the empty fields that
+ /// [`Self::admit`] records.
+ async fn read_with_token<O, F, Fut>(
+ &self,
+ link_id: Uuid,
+ operation: ApiOperationKind,
+ read: F,
+ ) -> Result<O, EmailApiError>
+ where
+ F: Fn(AccessToken) -> Fut,
+ Fut: Future<Output = Result<O, EmailApiError>>,
+ {
+ self.retry_with_fresh_token(link_id, operation, read)
+ .await?
+ .inspect_err(log_operation_error)
+ }
+
+ /// Runs a provider read with the cached token, retrying once with a fresh
+ /// token if the provider rejects it.
+ ///
/// A cached token can outlive its grant. Only a fresh fetch reaches the
/// authentication service, which either renews the token or reports the
/// revoked grant so the token source marks the link for reauthorization.
///
- /// Callers declare `retry_count = 0` on their span; a retry sets it to 1.
- async fn read_with_token<O, F, Fut>(
+ /// Like [`Self::attempt`], admission and token failures are logged and
+ /// returned as the outer error; the provider's final answer is left to the
+ /// caller.
+ async fn retry_with_fresh_token<O, F, Fut>(
&self,
link_id: Uuid,
operation: ApiOperationKind,
read: F,
- ) -> Result<O, EmailApiError>
+ ) -> Result<Result<O, EmailApiError>, EmailApiError>
where
F: Fn(AccessToken) -> Fut,
Fut: Future<Output = Result<O, EmailApiError>>,
{
- let access_token = self.prepare(link_id, operation).await?;
- let result = match read(access_token).await {
+ let cached = self.get_access_token(link_id, TokenFreshness::Cached);
+ match self.attempt(link_id, operation, cached, &read).await? {
Err(EmailApiError::AuthRequired) => {
tracing::Span::current().record("retry_count", 1);
- self.check_rate_limit(link_id, operation).await?;
- let fresh = self
- .get_access_token(link_id, TokenFreshness::Fresh)
- .await
- .inspect_err(log_operation_error)?;
- read(fresh).await
+ let fresh = self.get_access_token(link_id, TokenFreshness::Fresh);
+ self.attempt(link_id, operation, fresh, &read).await
}
- result => result,
+ result => Ok(result),
+ }
+ }
+
+ /// Makes one provider call under an admission permit.
+ ///
+ /// The gate is checked before `token` runs: under throttling this avoids
+ /// paying the token dance (SELECT + Redis + possible auth-service refresh +
+ /// health write) per refused attempt. An admission or token failure is
+ /// logged and returned as the outer error; the provider's answer is the
+ /// inner result, which also decides whether the mailbox cools down. If
+ /// the caller is cancelled first, the dropped permit releases itself.
+ async fn attempt<O, F, Fut>(
+ &self,
+ link_id: Uuid,
+ operation: ApiOperationKind,
+ token: impl Future<Output = Result<AccessToken, EmailApiError>>,
+ call: F,
+ ) -> Result<Result<O, EmailApiError>, EmailApiError>
+ where
+ F: FnOnce(AccessToken) -> Fut,
+ Fut: Future<Output = Result<O, EmailApiError>>,
+ {
+ let permit = self.admit(link_id, operation).await?;
+ let result = match token.await.inspect_err(log_operation_error) {
+ Ok(token) => Ok(call(token).await),
+ Err(error) => Err(error),
};
- result.inspect_err(log_operation_error)
+ let provider_error = result.as_ref().ok().and_then(|call| call.as_ref().err());
+ self.gate.finish(permit, provider_error).await;
+ result
}
- /// Applies local quota policy, logging a refusal in the calling
+ /// Takes an admission permit, logging a refusal in the calling
/// operation's span.
- async fn check_rate_limit(
+ ///
+ /// Calling operations declare `gate_in_flight` and `gate_cap` as empty
+ /// span fields. They record the mailbox's load at admission next to the
+ /// request's outcome, which is how Gmail's real concurrency limit can be
+ /// measured.
+ async fn admit(
&self,
link_id: Uuid,
operation: ApiOperationKind,
- ) -> Result<(), EmailApiError> {
- self.rate_limiter
- .check_rate_limit(link_id, operation)
+ ) -> Result<RequestPermit, EmailApiError> {
+ let permit = self
+ .gate
+ .acquire(link_id, operation)
.await
.map_err(EmailApiError::from)
- .inspect_err(log_operation_error)
+ .inspect_err(log_operation_error)?;
+ if let Some(load) = permit.load() {
+ tracing::Span::current()
+ .record("gate_in_flight", load.in_flight)
+ .record("gate_cap", load.cap);
+ }
+ Ok(permit)
}
}
@@ -136,7 +201,7 @@ where
///
/// Unlike `#[instrument(err)]`, whose level is fixed, this distinguishes a
/// local rate-limit refusal: expected backpressure where no provider request
-/// was made and the caller decides whether to retry or drop. It is logged at
+/// was made and the caller decides when to retry. It is logged at
/// DEBUG; every other failure, including provider throttling, stays at ERROR.
fn log_operation_error(error: &EmailApiError) {
match error {
@@ -144,6 +209,9 @@ fn log_operation_error(error: &EmailApiError) {
origin: RateLimitOrigin::Local,
..
} => tracing::debug!(error = %error),
+ EmailApiError::RateLimited { origin, .. } => {
+ tracing::error!(error = %error, origin = ?origin)
+ }
_ => tracing::error!(error = %error),
}
}
diff --git a/crates/email_api_client/src/domain/service/attachments.rs b/crates/email_api_client/src/domain/service/attachments.rs
index f138c3b256..5a8fb234f6 100644
--- a/crates/email_api_client/src/domain/service/attachments.rs
+++ b/crates/email_api_client/src/domain/service/attachments.rs
@@ -1,17 +1,17 @@
use uuid::Uuid;
use super::super::models::EmailApiError;
-use super::super::ports::{MailboxAttachmentClient, ProviderRateLimiter, ProviderTokenSource};
+use super::super::ports::{MailboxAttachmentClient, ProviderRequestGate, ProviderTokenSource};
use super::{ApiOperationKind, EmailApiClientServiceImpl};
impl<R, T, L> EmailApiClientServiceImpl<R, T, L>
where
R: MailboxAttachmentClient,
T: ProviderTokenSource,
- L: ProviderRateLimiter,
+ L: ProviderRequestGate,
{
/// Downloads an attachment from a provider message.
- #[tracing::instrument(skip(self), fields(retry_count = 0))]
+ #[tracing::instrument(skip(self), fields(retry_count = 0, gate_in_flight = tracing::field::Empty, gate_cap = tracing::field::Empty))]
pub async fn get_attachment(
&self,
link_id: Uuid,
diff --git a/crates/email_api_client/src/domain/service/attachments/test.rs b/crates/email_api_client/src/domain/service/attachments/test.rs
index 3bfe8f65dc..42f598c1eb 100644
--- a/crates/email_api_client/src/domain/service/attachments/test.rs
+++ b/crates/email_api_client/src/domain/service/attachments/test.rs
@@ -2,7 +2,7 @@ use std::sync::{Arc, Mutex};
use uuid::Uuid;
-use super::super::test_support::{Call, FakeRateLimiter, FakeTokenSource, call_log};
+use super::super::test_support::{Call, FakeGate, FakeTokenSource, call_log};
use super::*;
use crate::domain::models::{AccessToken, TokenFreshness};
@@ -36,7 +36,7 @@ async fn get_attachment_uses_attachment_quota_and_forwards_identifiers() {
let service = EmailApiClientServiceImpl::new(
repository,
FakeTokenSource::new(calls.clone(), Ok(AccessToken::new("token"))),
- FakeRateLimiter::new(calls.clone(), Ok(())),
+ FakeGate::new(calls.clone(), Ok(())),
);
let result = service
@@ -51,8 +51,9 @@ async fn get_attachment_uses_attachment_quota_and_forwards_identifiers() {
assert_eq!(
*calls.lock().unwrap(),
vec![
- Call::RateLimit(link_id, ApiOperationKind::GetAttachment),
+ Call::Acquire(link_id, ApiOperationKind::GetAttachment),
Call::Token(link_id, TokenFreshness::Cached),
+ Call::Finish(link_id, None),
]
);
}
diff --git a/crates/email_api_client/src/domain/service/blocklist.rs b/crates/email_api_client/src/domain/service/blocklist.rs
index 4c2046c167..b548b26f4d 100644
--- a/crates/email_api_client/src/domain/service/blocklist.rs
+++ b/crates/email_api_client/src/domain/service/blocklist.rs
@@ -1,58 +1,60 @@
use uuid::Uuid;
use super::super::models::EmailApiError;
-use super::super::ports::{MailboxBlocklistClient, ProviderRateLimiter, ProviderTokenSource};
+use super::super::ports::{MailboxBlocklistClient, ProviderRequestGate, ProviderTokenSource};
use super::{ApiOperationKind, EmailApiClientServiceImpl, log_operation_error};
impl<R, T, L> EmailApiClientServiceImpl<R, T, L>
where
R: MailboxBlocklistClient,
T: ProviderTokenSource,
- L: ProviderRateLimiter,
+ L: ProviderRequestGate,
{
/// Ensures messages from an email address are sent to trash.
- #[tracing::instrument(skip(self, email_address))]
+ #[tracing::instrument(skip(self, email_address), fields(gate_in_flight = tracing::field::Empty, gate_cap = tracing::field::Empty))]
pub async fn block_sender(
&self,
link_id: Uuid,
email_address: &str,
) -> Result<(), EmailApiError> {
- let access_token = self.prepare(link_id, ApiOperationKind::BlockSender).await?;
-
- self.repository
- .block_sender(&access_token, email_address)
- .await
- .inspect_err(log_operation_error)
+ self.call_with_token(link_id, ApiOperationKind::BlockSender, |token| async move {
+ self.repository
+ .block_sender(&token, email_address)
+ .await
+ .inspect_err(log_operation_error)
+ })
+ .await
}
/// Removes the blocked-sender rule for an email address, when present.
- #[tracing::instrument(skip(self, email_address))]
+ #[tracing::instrument(skip(self, email_address), fields(gate_in_flight = tracing::field::Empty, gate_cap = tracing::field::Empty))]
pub async fn unblock_sender(
&self,
link_id: Uuid,
email_address: &str,
) -> Result<(), EmailApiError> {
- let access_token = self
- .prepare(link_id, ApiOperationKind::UnblockSender)
- .await?;
-
- self.repository
- .unblock_sender(&access_token, email_address)
- .await
- .inspect_err(log_operation_error)
+ self.call_with_token(
+ link_id,
+ ApiOperationKind::UnblockSender,
+ |token| async move {
+ self.repository
+ .unblock_sender(&token, email_address)
+ .await
+ .inspect_err(log_operation_error)
+ },
+ )
+ .await
}
/// Lists email addresses covered by blocked-sender rules.
- #[tracing::instrument(skip(self))]
+ #[tracing::instrument(skip(self), fields(retry_count = 0, gate_in_flight = tracing::field::Empty, gate_cap = tracing::field::Empty))]
pub async fn list_blocked_senders(&self, link_id: Uuid) -> Result<Vec<String>, EmailApiError> {
- let access_token = self
- .prepare(link_id, ApiOperationKind::ListBlockedSenders)
- .await?;
-
- self.repository
- .list_blocked_senders(&access_token)
- .await
- .inspect_err(log_operation_error)
+ self.read_with_token(
+ link_id,
+ ApiOperationKind::ListBlockedSenders,
+ |token| async move { self.repository.list_blocked_senders(&token).await },
+ )
+ .await
}
}
diff --git a/crates/email_api_client/src/domain/service/blocklist/test.rs b/crates/email_api_client/src/domain/service/blocklist/test.rs
index d2da5a5f7e..3bf2a1d2b5 100644
--- a/crates/email_api_client/src/domain/service/blocklist/test.rs
+++ b/crates/email_api_client/src/domain/service/blocklist/test.rs
@@ -2,7 +2,7 @@ use uuid::Uuid;
use super::super::super::models::{AccessToken, EmailApiError, TokenFreshness};
use super::super::super::ports::MailboxBlocklistClient;
-use super::super::test_support::{Call, FakeRateLimiter, FakeTokenSource, call_log};
+use super::super::test_support::{Call, FakeGate, FakeTokenSource, call_log};
use super::EmailApiClientServiceImpl;
#[derive(Clone)]
@@ -77,9 +77,10 @@ async fn blocklist_methods_use_their_matching_costs_and_operations() {
assert_eq!(
*calls.lock().unwrap(),
vec![
- Call::RateLimit(Uuid::nil(), operation),
+ Call::Acquire(Uuid::nil(), operation),
Call::Token(Uuid::nil(), TokenFreshness::Cached),
Call::Repository(repository_call),
+ Call::Finish(Uuid::nil(), None),
]
);
}
@@ -87,12 +88,12 @@ async fn blocklist_methods_use_their_matching_costs_and_operations() {
fn service(
calls: super::super::test_support::CallLog,
-) -> EmailApiClientServiceImpl<BlocklistClient, FakeTokenSource, FakeRateLimiter> {
+) -> EmailApiClientServiceImpl<BlocklistClient, FakeTokenSource, FakeGate> {
EmailApiClientServiceImpl::new(
BlocklistClient {
calls: calls.clone(),
},
FakeTokenSource::new(calls.clone(), Ok(AccessToken::new("access-token"))),
- FakeRateLimiter::new(calls, Ok(())),
+ FakeGate::new(calls, Ok(())),
)
}
diff --git a/crates/email_api_client/src/domain/service/contacts.rs b/crates/email_api_client/src/domain/service/contacts.rs
index 20538678aa..ddd3ae3182 100644
--- a/crates/email_api_client/src/domain/service/contacts.rs
+++ b/crates/email_api_client/src/domain/service/contacts.rs
@@ -2,63 +2,67 @@ use models_email::service::contact::{Contact, ContactList};
use uuid::Uuid;
use super::super::models::EmailApiError;
-use super::super::ports::{MailboxContactsClient, ProviderRateLimiter, ProviderTokenSource};
-use super::{ApiOperationKind, EmailApiClientServiceImpl, log_operation_error};
+use super::super::ports::{MailboxContactsClient, ProviderRequestGate, ProviderTokenSource};
+use super::{ApiOperationKind, EmailApiClientServiceImpl};
impl<R, T, L> EmailApiClientServiceImpl<R, T, L>
where
R: MailboxContactsClient,
T: ProviderTokenSource,
- L: ProviderRateLimiter,
+ L: ProviderRequestGate,
{
/// Fetches the linked mailbox owner's contact record.
- #[tracing::instrument(skip(self))]
+ #[tracing::instrument(skip(self), fields(retry_count = 0, gate_in_flight = tracing::field::Empty, gate_cap = tracing::field::Empty))]
pub async fn get_self_contact(&self, link_id: Uuid) -> Result<Contact, EmailApiError> {
- let access_token = self
- .prepare(link_id, ApiOperationKind::ListContacts)
- .await?;
-
- self.repository
- .get_self_contact(&access_token, link_id)
- .await
- .inspect_err(log_operation_error)
+ self.read_with_token(
+ link_id,
+ ApiOperationKind::ListContacts,
+ |token| async move { self.repository.get_self_contact(&token, link_id).await },
+ )
+ .await
}
/// Lists primary contacts, optionally continuing an incremental synchronization.
///
/// Failures log at WARN: an expired sync token ([`EmailApiError::OutdatedCursor`])
/// or a grant without the contacts scopes is routine, and callers recover or skip.
- #[tracing::instrument(skip(self, sync_token), err(level = "warn"))]
+ #[tracing::instrument(skip(self, sync_token), fields(retry_count = 0, gate_in_flight = tracing::field::Empty, gate_cap = tracing::field::Empty), err(level = "warn"))]
pub async fn list_contacts(
&self,
link_id: Uuid,
sync_token: Option<&str>,
) -> Result<ContactList, EmailApiError> {
- let access_token = self
- .prepare(link_id, ApiOperationKind::ListContacts)
- .await?;
-
- self.repository
- .list_contacts(&access_token, link_id, sync_token)
- .await
+ self.retry_with_fresh_token(
+ link_id,
+ ApiOperationKind::ListContacts,
+ |token| async move {
+ self.repository
+ .list_contacts(&token, link_id, sync_token)
+ .await
+ },
+ )
+ .await?
}
/// Lists automatically collected contacts, optionally continuing a synchronization.
///
/// Failures log at WARN for the same reasons as [`Self::list_contacts`].
- #[tracing::instrument(skip(self, sync_token), err(level = "warn"))]
+ #[tracing::instrument(skip(self, sync_token), fields(retry_count = 0, gate_in_flight = tracing::field::Empty, gate_cap = tracing::field::Empty), err(level = "warn"))]
pub async fn list_other_contacts(
&self,
link_id: Uuid,
sync_token: Option<&str>,
) -> Result<ContactList, EmailApiError> {
- let access_token = self
- .prepare(link_id, ApiOperationKind::ListContacts)
- .await?;
-
- self.repository
- .list_other_contacts(&access_token, link_id, sync_token)
- .await
+ self.retry_with_fresh_token(
+ link_id,
+ ApiOperationKind::ListContacts,
+ |token| async move {
+ self.repository
+ .list_other_contacts(&token, link_id, sync_token)
+ .await
+ },
+ )
+ .await?
}
}
diff --git a/crates/email_api_client/src/domain/service/contacts/test.rs b/crates/email_api_client/src/domain/service/contacts/test.rs
index 50a184308b..54df24189f 100644
--- a/crates/email_api_client/src/domain/service/contacts/test.rs
+++ b/crates/email_api_client/src/domain/service/contacts/test.rs
@@ -3,7 +3,7 @@ use std::sync::{Arc, Mutex};
use models_email::service::contact::{Contact, ContactList};
use uuid::Uuid;
-use super::super::test_support::{Call, FakeRateLimiter, FakeTokenSource, call_log};
+use super::super::test_support::{Call, FakeGate, FakeTokenSource, call_log};
use super::*;
use crate::domain::models::{AccessToken, TokenFreshness};
@@ -69,7 +69,7 @@ async fn assert_call<F, E>(operation: ApiOperationKind, expected_repository_call
where
E: FnOnce(Uuid) -> ContactCall,
F: AsyncFnOnce(
- &EmailApiClientServiceImpl<ContactsClient, FakeTokenSource, FakeRateLimiter>,
+ &EmailApiClientServiceImpl<ContactsClient, FakeTokenSource, FakeGate>,
Uuid,
) -> Result<(), EmailApiError>,
{
@@ -80,7 +80,7 @@ where
let service = EmailApiClientServiceImpl::new(
repository,
FakeTokenSource::new(calls.clone(), Ok(AccessToken::new("token"))),
- FakeRateLimiter::new(calls.clone(), Ok(())),
+ FakeGate::new(calls.clone(), Ok(())),
);
assert_eq!(invoke(&service, link_id).await, Err(expected_error()));
@@ -91,8 +91,66 @@ where
assert_eq!(
*calls.lock().unwrap(),
vec![
- Call::RateLimit(link_id, operation),
+ Call::Acquire(link_id, operation),
Call::Token(link_id, TokenFreshness::Cached),
+ Call::Finish(link_id, Some(expected_error())),
+ ]
+ );
+}
+
+/// Answers a cached token the way Google answers a revoked or expired one.
+struct RejectsCachedToken;
+
+impl MailboxContactsClient for RejectsCachedToken {
+ async fn get_self_contact(&self, _: &AccessToken, _: Uuid) -> Result<Contact, EmailApiError> {
+ unreachable!("not used by this test")
+ }
+
+ async fn list_contacts(
+ &self,
+ access_token: &AccessToken,
+ _: Uuid,
+ _: Option<&str>,
+ ) -> Result<ContactList, EmailApiError> {
+ match access_token.expose_secret() {
+ "cached" => Err(EmailApiError::AuthRequired),
+ _ => Err(expected_error()),
+ }
+ }
+
+ async fn list_other_contacts(
+ &self,
+ _: &AccessToken,
+ _: Uuid,
+ _: Option<&str>,
+ ) -> Result<ContactList, EmailApiError> {
+ unreachable!("not used by this test")
+ }
+}
+
+#[tokio::test]
+async fn a_rejected_cached_token_is_retried_fresh_for_contact_sync() {
+ let calls = call_log();
+ let link_id = Uuid::new_v4();
+ let service = EmailApiClientServiceImpl::new(
+ RejectsCachedToken,
+ FakeTokenSource::new(calls.clone(), Ok(AccessToken::new("cached")))
+ .with_fresh(Ok(AccessToken::new("fresh"))),
+ FakeGate::new(calls.clone(), Ok(())),
+ );
+
+ let result = service.list_contacts(link_id, None).await;
+
+ assert_eq!(result.err(), Some(expected_error()));
+ assert_eq!(
+ *calls.lock().unwrap(),
+ vec![
+ Call::Acquire(link_id, ApiOperationKind::ListContacts),
+ Call::Token(link_id, TokenFreshness::Cached),
+ Call::Finish(link_id, Some(EmailApiError::AuthRequired)),
+ Call::Acquire(link_id, ApiOperationKind::ListContacts),
+ Call::Token(link_id, TokenFreshness::Fresh),
+ Call::Finish(link_id, Some(expected_error())),
]
);
}
diff --git a/crates/email_api_client/src/domain/service/labels.rs b/crates/email_api_client/src/domain/service/labels.rs
index d1e9dc2aa5..0e454a292f 100644
--- a/crates/email_api_client/src/domain/service/labels.rs
+++ b/crates/email_api_client/src/domain/service/labels.rs
@@ -2,17 +2,17 @@ use models_email::email::service::label::Label;
use uuid::Uuid;
use super::super::models::EmailApiError;
-use super::super::ports::{MailboxLabelClient, ProviderRateLimiter, ProviderTokenSource};
+use super::super::ports::{MailboxLabelClient, ProviderRequestGate, ProviderTokenSource};
use super::{ApiOperationKind, EmailApiClientServiceImpl, log_operation_error};
impl<R, T, L> EmailApiClientServiceImpl<R, T, L>
where
R: MailboxLabelClient,
T: ProviderTokenSource,
- L: ProviderRateLimiter,
+ L: ProviderRequestGate,
{
/// Lists all labels for a linked mailbox.
- #[tracing::instrument(skip(self), fields(retry_count = 0))]
+ #[tracing::instrument(skip(self), fields(retry_count = 0, gate_in_flight = tracing::field::Empty, gate_cap = tracing::field::Empty))]
pub async fn list_labels(&self, link_id: Uuid) -> Result<Vec<Label>, EmailApiError> {
self.read_with_token(link_id, ApiOperationKind::ListLabels, |token| async move {
self.repository.list_labels(&token, link_id).await
@@ -21,33 +21,35 @@ where
}
/// Creates a user label.
- #[tracing::instrument(skip(self))]
+ #[tracing::instrument(skip(self), fields(gate_in_flight = tracing::field::Empty, gate_cap = tracing::field::Empty))]
pub async fn create_label(
&self,
link_id: Uuid,
label_name: &str,
) -> Result<Label, EmailApiError> {
- let access_token = self.prepare(link_id, ApiOperationKind::CreateLabel).await?;
-
- self.repository
- .create_label(&access_token, link_id, label_name)
- .await
- .inspect_err(log_operation_error)
+ self.call_with_token(link_id, ApiOperationKind::CreateLabel, |token| async move {
+ self.repository
+ .create_label(&token, link_id, label_name)
+ .await
+ .inspect_err(log_operation_error)
+ })
+ .await
}
/// Deletes a provider label.
- #[tracing::instrument(skip(self))]
+ #[tracing::instrument(skip(self), fields(gate_in_flight = tracing::field::Empty, gate_cap = tracing::field::Empty))]
pub async fn delete_label(
&self,
link_id: Uuid,
provider_label_id: &str,
) -> Result<(), EmailApiError> {
- let access_token = self.prepare(link_id, ApiOperationKind::DeleteLabel).await?;
-
- self.repository
- .delete_label(&access_token, provider_label_id)
- .await
- .inspect_err(log_operation_error)
+ self.call_with_token(link_id, ApiOperationKind::DeleteLabel, |token| async move {
+ self.repository
+ .delete_label(&token, provider_label_id)
+ .await
+ .inspect_err(log_operation_error)
+ })
+ .await
}
}
diff --git a/crates/email_api_client/src/domain/service/labels/test.rs b/crates/email_api_client/src/domain/service/labels/test.rs
index 5a56a12b8a..e379360cf9 100644
--- a/crates/email_api_client/src/domain/service/labels/test.rs
+++ b/crates/email_api_client/src/domain/service/labels/test.rs
@@ -3,7 +3,7 @@ use std::sync::{Arc, Mutex};
use models_email::email::service::label::Label;
use uuid::Uuid;
-use super::super::test_support::{Call, FakeRateLimiter, FakeTokenSource, call_log};
+use super::super::test_support::{Call, FakeGate, FakeTokenSource, call_log};
use super::*;
use crate::domain::models::{AccessToken, TokenFreshness};
@@ -59,7 +59,7 @@ async fn list_labels_uses_list_labels_quota_and_forwards_link_id() {
let service = EmailApiClientServiceImpl::new(
repository,
FakeTokenSource::new(calls.clone(), Ok(AccessToken::new("token"))),
- FakeRateLimiter::new(calls.clone(), Ok(())),
+ FakeGate::new(calls.clone(), Ok(())),
);
assert!(matches!(
@@ -70,8 +70,9 @@ async fn list_labels_uses_list_labels_quota_and_forwards_link_id() {
assert_eq!(
*calls.lock().unwrap(),
vec![
- Call::RateLimit(link_id, ApiOperationKind::ListLabels),
+ Call::Acquire(link_id, ApiOperationKind::ListLabels),
Call::Token(link_id, TokenFreshness::Cached),
+ Call::Finish(link_id, Some(expected_error())),
]
);
}
@@ -85,7 +86,7 @@ async fn create_label_uses_create_quota_and_forwards_name() {
let service = EmailApiClientServiceImpl::new(
repository,
FakeTokenSource::new(calls.clone(), Ok(AccessToken::new("token"))),
- FakeRateLimiter::new(calls.clone(), Ok(())),
+ FakeGate::new(calls.clone(), Ok(())),
);
assert!(matches!(
@@ -96,8 +97,9 @@ async fn create_label_uses_create_quota_and_forwards_name() {
assert_eq!(
*calls.lock().unwrap(),
vec![
- Call::RateLimit(link_id, ApiOperationKind::CreateLabel),
+ Call::Acquire(link_id, ApiOperationKind::CreateLabel),
Call::Token(link_id, TokenFreshness::Cached),
+ Call::Finish(link_id, Some(expected_error())),
]
);
}
@@ -111,7 +113,7 @@ async fn delete_label_uses_delete_quota_and_forwards_provider_id() {
let service = EmailApiClientServiceImpl::new(
repository,
FakeTokenSource::new(calls.clone(), Ok(AccessToken::new("token"))),
- FakeRateLimiter::new(calls.clone(), Ok(())),
+ FakeGate::new(calls.clone(), Ok(())),
);
assert_eq!(
@@ -122,8 +124,9 @@ async fn delete_label_uses_delete_quota_and_forwards_provider_id() {
assert_eq!(
*calls.lock().unwrap(),
vec![
- Call::RateLimit(link_id, ApiOperationKind::DeleteLabel),
+ Call::Acquire(link_id, ApiOperationKind::DeleteLabel),
Call::Token(link_id, TokenFreshness::Cached),
+ Call::Finish(link_id, Some(expected_error())),
]
);
}
diff --git a/crates/email_api_client/src/domain/service/messages.rs b/crates/email_api_client/src/domain/service/messages.rs
index 0623e60382..a9fc94702f 100644
--- a/crates/email_api_client/src/domain/service/messages.rs
+++ b/crates/email_api_client/src/domain/service/messages.rs
@@ -3,7 +3,7 @@ use uuid::Uuid;
use super::super::models::{CalendarPart, EmailApiError, MessageWithCalendarParts, ThreadListPage};
use super::super::ports::{
- MailboxCalendarClient, MailboxMessageClient, ProviderRateLimiter, ProviderTokenSource,
+ MailboxCalendarClient, MailboxMessageClient, ProviderRequestGate, ProviderTokenSource,
};
use super::{ApiOperationKind, EmailApiClientServiceImpl, log_operation_error};
@@ -11,11 +11,11 @@ impl<R, T, L> EmailApiClientServiceImpl<R, T, L>
where
R: MailboxMessageClient,
T: ProviderTokenSource,
- L: ProviderRateLimiter,
+ L: ProviderRequestGate,
{
/// Fetches and normalizes one provider message, including any calendar
/// invitation parts, in a single provider read.
- #[tracing::instrument(skip(self), fields(retry_count = 0))]
+ #[tracing::instrument(skip(self), fields(retry_count = 0, gate_in_flight = tracing::field::Empty, gate_cap = tracing::field::Empty))]
pub async fn get_message(
&self,
link_id: Uuid,
@@ -30,7 +30,7 @@ where
}
/// Fetches the provider label identifiers attached to one message.
- #[tracing::instrument(skip(self), fields(retry_count = 0))]
+ #[tracing::instrument(skip(self), fields(retry_count = 0, gate_in_flight = tracing::field::Empty, gate_cap = tracing::field::Empty))]
pub async fn get_message_label_ids(
&self,
link_id: Uuid,
@@ -45,7 +45,7 @@ where
}
/// Lists message identifiers carrying all requested provider labels.
- #[tracing::instrument(skip(self), fields(retry_count = 0))]
+ #[tracing::instrument(skip(self), fields(retry_count = 0, gate_in_flight = tracing::field::Empty, gate_cap = tracing::field::Empty))]
pub async fn list_messages(
&self,
link_id: Uuid,
@@ -65,7 +65,7 @@ where
}
/// Lists message identifiers belonging to one provider thread.
- #[tracing::instrument(skip(self), fields(retry_count = 0))]
+ #[tracing::instrument(skip(self), fields(retry_count = 0, gate_in_flight = tracing::field::Empty, gate_cap = tracing::field::Empty))]
pub async fn get_message_ids_for_thread(
&self,
link_id: Uuid,
@@ -80,7 +80,7 @@ where
}
/// Fetches and normalizes every message in one provider thread.
- #[tracing::instrument(skip(self), fields(retry_count = 0))]
+ #[tracing::instrument(skip(self), fields(retry_count = 0, gate_in_flight = tracing::field::Empty, gate_cap = tracing::field::Empty))]
pub async fn get_thread(
&self,
link_id: Uuid,
@@ -95,7 +95,7 @@ where
}
/// Lists one page of provider threads.
- #[tracing::instrument(skip(self), fields(retry_count = 0))]
+ #[tracing::instrument(skip(self), fields(retry_count = 0, gate_in_flight = tracing::field::Empty, gate_cap = tracing::field::Empty))]
pub async fn list_threads(
&self,
link_id: Uuid,
@@ -112,7 +112,7 @@ where
}
/// Applies provider label additions and removals to one message.
- #[tracing::instrument(skip(self))]
+ #[tracing::instrument(skip(self), fields(gate_in_flight = tracing::field::Empty, gate_cap = tracing::field::Empty))]
pub async fn modify_message_labels(
&self,
link_id: Uuid,
@@ -120,19 +120,22 @@ where
labels_to_add: &[String],
labels_to_remove: &[String],
) -> Result<(), EmailApiError> {
- let access_token = self
- .prepare(link_id, ApiOperationKind::ModifyMessageLabels)
- .await?;
-
- self.repository
- .modify_message_labels(
- &access_token,
- provider_message_id,
- labels_to_add,
- labels_to_remove,
- )
- .await
- .inspect_err(log_operation_error)
+ self.call_with_token(
+ link_id,
+ ApiOperationKind::ModifyMessageLabels,
+ |token| async move {
+ self.repository
+ .modify_message_labels(
+ &token,
+ provider_message_id,
+ labels_to_add,
+ labels_to_remove,
+ )
+ .await
+ .inspect_err(log_operation_error)
+ },
+ )
+ .await
}
}
@@ -140,7 +143,7 @@ impl<R, T, L> EmailApiClientServiceImpl<R, T, L>
where
R: MailboxCalendarClient,
T: ProviderTokenSource,
- L: ProviderRateLimiter,
+ L: ProviderRequestGate,
{
/// Finds calendar invitation parts in one provider message via a fresh
/// provider fetch.
@@ -148,17 +151,18 @@ where
/// This charges a full message read; ingest paths that already fetched the
/// message should consume [`MessageWithCalendarParts::calendar_parts`]
/// instead. This lookup exists for durable re-extraction jobs.
- #[tracing::instrument(skip(self))]
+ #[tracing::instrument(skip(self), fields(retry_count = 0, gate_in_flight = tracing::field::Empty, gate_cap = tracing::field::Empty))]
pub async fn get_calendar_parts(
&self,
link_id: Uuid,
provider_message_id: &str,
) -> Result<Vec<CalendarPart>, EmailApiError> {
- let access_token = self.prepare(link_id, ApiOperationKind::GetMessage).await?;
- self.repository
- .get_calendar_parts(&access_token, provider_message_id)
- .await
- .inspect_err(log_operation_error)
+ self.read_with_token(link_id, ApiOperationKind::GetMessage, |token| async move {
+ self.repository
+ .get_calendar_parts(&token, provider_message_id)
+ .await
+ })
+ .await
}
}
diff --git a/crates/email_api_client/src/domain/service/messages/test.rs b/crates/email_api_client/src/domain/service/messages/test.rs
index 1efc48a741..882c2d0988 100644
--- a/crates/email_api_client/src/domain/service/messages/test.rs
+++ b/crates/email_api_client/src/domain/service/messages/test.rs
@@ -3,7 +3,7 @@ use std::sync::{Arc, Mutex};
use models_email::email::service::message::Message;
use uuid::Uuid;
-use super::super::test_support::{Call, FakeRateLimiter, FakeTokenSource, call_log};
+use super::super::test_support::{Call, FakeGate, FakeTokenSource, call_log};
use super::*;
use crate::domain::models::{AccessToken, TokenFreshness};
@@ -138,7 +138,7 @@ async fn assert_call<F, E>(operation: ApiOperationKind, expected_repository_call
where
E: FnOnce(Uuid) -> MessageCall,
F: AsyncFnOnce(
- &EmailApiClientServiceImpl<MessageClient, FakeTokenSource, FakeRateLimiter>,
+ &EmailApiClientServiceImpl<MessageClient, FakeTokenSource, FakeGate>,
Uuid,
) -> Result<(), EmailApiError>,
{
@@ -149,7 +149,7 @@ where
let service = EmailApiClientServiceImpl::new(
repository,
FakeTokenSource::new(calls.clone(), Ok(AccessToken::new("token"))),
- FakeRateLimiter::new(calls.clone(), Ok(())),
+ FakeGate::new(calls.clone(), Ok(())),
);
assert_eq!(invoke(&service, link_id).await, Err(expected_error()));
@@ -160,8 +160,9 @@ where
assert_eq!(
*calls.lock().unwrap(),
vec![
- Call::RateLimit(link_id, operation),
+ Call::Acquire(link_id, operation),
Call::Token(link_id, TokenFreshness::Cached),
+ Call::Finish(link_id, Some(expected_error())),
]
);
}
diff --git a/crates/email_api_client/src/domain/service/send.rs b/crates/email_api_client/src/domain/service/send.rs
index ca4a3f32e9..a38c928ced 100644
--- a/crates/email_api_client/src/domain/service/send.rs
+++ b/crates/email_api_client/src/domain/service/send.rs
@@ -1,10 +1,10 @@
use uuid::Uuid;
use super::super::models::{
- EmailApiError, PreparedSendMessage, SendRequest, SentIds, validate_message_id,
+ EmailApiError, PreparedSendMessage, SendRequest, SentIds, TokenFreshness, validate_message_id,
};
use super::super::ports::{
- MailboxSendClient, MailboxSendRecoveryClient, ProviderRateLimiter, ProviderTokenSource,
+ MailboxSendClient, MailboxSendRecoveryClient, ProviderRequestGate, ProviderTokenSource,
};
use super::{ApiOperationKind, EmailApiClientServiceImpl, log_operation_error};
@@ -12,22 +12,23 @@ impl<R, T, L> EmailApiClientServiceImpl<R, T, L>
where
R: MailboxSendClient,
T: ProviderTokenSource,
- L: ProviderRateLimiter,
+ L: ProviderRequestGate,
{
/// Sends a message through the linked mailbox.
- #[tracing::instrument(skip(self, request))]
+ #[tracing::instrument(skip(self, request), fields(gate_in_flight = tracing::field::Empty, gate_cap = tracing::field::Empty))]
pub async fn send_message(
&self,
link_id: Uuid,
request: &SendRequest,
provider_thread_id: Option<&str>,
) -> Result<SentIds, EmailApiError> {
- let access_token = self.prepare(link_id, ApiOperationKind::SendMessage).await?;
-
- self.repository
- .send_message(&access_token, request, provider_thread_id)
- .await
- .inspect_err(log_operation_error)
+ self.call_with_token(link_id, ApiOperationKind::SendMessage, |token| async move {
+ self.repository
+ .send_message(&token, request, provider_thread_id)
+ .await
+ .inspect_err(log_operation_error)
+ })
+ .await
}
}
@@ -35,11 +36,11 @@ impl<R, T, L> EmailApiClientServiceImpl<R, T, L>
where
R: MailboxSendRecoveryClient,
T: ProviderTokenSource,
- L: ProviderRateLimiter,
+ L: ProviderRequestGate,
{
- /// Completes quota, token and MIME preparation without dispatching a send.
- /// Persist delivery-started state only after this method succeeds.
- #[tracing::instrument(skip(self, request))]
+ /// Completes admission, token and MIME preparation without dispatching a
+ /// send. Persist delivery-started state only after this method succeeds.
+ #[tracing::instrument(skip(self, request), fields(gate_in_flight = tracing::field::Empty, gate_cap = tracing::field::Empty))]
pub async fn prepare_send(
&self,
link_id: Uuid,
@@ -47,8 +48,20 @@ where
provider_thread_id: Option<&str>,
) -> Result<PreparedSendMessage, EmailApiError> {
let mime = request.build_mime().inspect_err(log_operation_error)?;
- let access_token = self.prepare(link_id, ApiOperationKind::SendMessage).await?;
+ let permit = self.admit(link_id, ApiOperationKind::SendMessage).await?;
+ let access_token = match self
+ .get_access_token(link_id, TokenFreshness::Cached)
+ .await
+ .inspect_err(log_operation_error)
+ {
+ Ok(access_token) => access_token,
+ Err(error) => {
+ self.gate.finish(permit, None).await;
+ return Err(error);
+ }
+ };
Ok(PreparedSendMessage {
+ permit,
access_token,
mime,
provider_thread_id: provider_thread_id.map(str::to_owned),
@@ -60,33 +73,36 @@ where
#[tracing::instrument(skip_all)]
pub async fn send_prepared(
&self,
- prepared: &PreparedSendMessage,
+ prepared: PreparedSendMessage,
) -> Result<SentIds, EmailApiError> {
- self.repository
+ let result = self
+ .repository
.send_prepared(
&prepared.access_token,
&prepared.mime,
prepared.provider_thread_id.as_deref(),
)
- .await
- .inspect_err(log_operation_error)
+ .await;
+ self.gate
+ .finish(prepared.permit, result.as_ref().err())
+ .await;
+ result.inspect_err(log_operation_error)
}
/// Checks sent mail for a stable delivery identifier without sending again.
- #[tracing::instrument(skip(self, message_id))]
+ #[tracing::instrument(skip(self, message_id), fields(retry_count = 0, gate_in_flight = tracing::field::Empty, gate_cap = tracing::field::Empty))]
pub async fn find_sent_message(
&self,
link_id: Uuid,
message_id: &str,
) -> Result<Option<SentIds>, EmailApiError> {
validate_message_id(message_id).inspect_err(log_operation_error)?;
- let access_token = self
- .prepare(link_id, ApiOperationKind::ListMessages)
- .await?;
- self.repository
- .find_sent_message(&access_token, message_id)
- .await
- .inspect_err(log_operation_error)
+ self.read_with_token(
+ link_id,
+ ApiOperationKind::ListMessages,
+ |token| async move { self.repository.find_sent_message(&token, message_id).await },
+ )
+ .await
}
}
diff --git a/crates/email_api_client/src/domain/service/send/test.rs b/crates/email_api_client/src/domain/service/send/test.rs
index fc31102260..e8d0a137bc 100644
--- a/crates/email_api_client/src/domain/service/send/test.rs
+++ b/crates/email_api_client/src/domain/service/send/test.rs
@@ -3,10 +3,10 @@ use models_email::email::service::message::MessageToSend;
use uuid::Uuid;
use super::super::super::models::{
- AccessToken, EmailApiError, RateLimitRefusal, SendRequest, SentIds, TokenFreshness,
+ AccessToken, EmailApiError, RateLimitRefusal, SendRequest, SentIds, TokenError, TokenFreshness,
};
use super::super::super::ports::{MailboxSendClient, MailboxSendRecoveryClient};
-use super::super::test_support::{Call, FakeRateLimiter, FakeTokenSource, call_log};
+use super::super::test_support::{Call, FakeGate, FakeTokenSource, call_log};
use super::EmailApiClientServiceImpl;
#[derive(Clone)]
@@ -74,14 +74,14 @@ impl MailboxSendRecoveryClient for SendClient {
}
#[tokio::test]
-async fn preparation_never_dispatches_and_dispatch_does_not_repeat_preparation() {
+async fn preparation_holds_admission_until_the_single_dispatch_releases_it() {
let calls = call_log();
let service = EmailApiClientServiceImpl::new(
SendClient {
calls: calls.clone(),
},
FakeTokenSource::new(calls.clone(), Ok(AccessToken::new("access-token"))),
- FakeRateLimiter::new(calls.clone(), Ok(())),
+ FakeGate::new(calls.clone(), Ok(())),
);
let mut request = send_request();
request.message_id = Some("[email protected]".into());
@@ -92,13 +92,72 @@ async fn preparation_never_dispatches_and_dispatch_does_not_repeat_preparation()
assert_eq!(
*calls.lock().unwrap(),
[
- Call::RateLimit(Uuid::nil(), super::ApiOperationKind::SendMessage),
+ Call::Acquire(Uuid::nil(), super::ApiOperationKind::SendMessage),
Call::Token(Uuid::nil(), TokenFreshness::Cached),
]
);
- service.send_prepared(&prepared).await.unwrap();
- assert_eq!(calls.lock().unwrap().len(), 3);
- assert_eq!(calls.lock().unwrap()[2], Call::Repository("send_prepared"));
+ service.send_prepared(prepared).await.unwrap();
+ assert_eq!(
+ calls.lock().unwrap()[2..],
+ [
+ Call::Repository("send_prepared"),
+ Call::Finish(Uuid::nil(), None),
+ ]
+ );
+}
+
+#[tokio::test]
+async fn a_prepared_send_dropped_before_dispatch_releases_its_admission() {
+ let calls = call_log();
+ let service = EmailApiClientServiceImpl::new(
+ SendClient {
+ calls: calls.clone(),
+ },
+ FakeTokenSource::new(calls.clone(), Ok(AccessToken::new("access-token"))),
+ FakeGate::new(calls.clone(), Ok(())),
+ );
+ let prepared = service
+ .prepare_send(Uuid::nil(), &send_request(), Some("thread-1"))
+ .await
+ .unwrap();
+
+ // The delivery coordinator lost its claim before submission.
+ drop(prepared);
+
+ assert_eq!(
+ *calls.lock().unwrap(),
+ [
+ Call::Acquire(Uuid::nil(), super::ApiOperationKind::SendMessage),
+ Call::Token(Uuid::nil(), TokenFreshness::Cached),
+ Call::Abandon(Uuid::nil()),
+ ]
+ );
+}
+
+#[tokio::test]
+async fn token_failure_during_preparation_releases_admission() {
+ let calls = call_log();
+ let service = EmailApiClientServiceImpl::new(
+ SendClient {
+ calls: calls.clone(),
+ },
+ FakeTokenSource::new(calls.clone(), Err(TokenError::ReauthRequired)),
+ FakeGate::new(calls.clone(), Ok(())),
+ );
+ let error = service
+ .prepare_send(Uuid::nil(), &send_request(), None)
+ .await
+ .err()
+ .unwrap();
+ assert_eq!(error, EmailApiError::AuthRequired);
+ assert_eq!(
+ *calls.lock().unwrap(),
+ [
+ Call::Acquire(Uuid::nil(), super::ApiOperationKind::SendMessage),
+ Call::Token(Uuid::nil(), TokenFreshness::Cached),
+ Call::Finish(Uuid::nil(), None),
+ ]
+ );
}
#[tokio::test]
@@ -109,7 +168,7 @@ async fn quota_refusal_is_a_preparation_error_before_token_or_dispatch() {
calls: calls.clone(),
},
FakeTokenSource::new(calls.clone(), Ok(AccessToken::new("access-token"))),
- FakeRateLimiter::new(calls.clone(), Err(RateLimitRefusal { retry_after: None })),
+ FakeGate::new(calls.clone(), Err(RateLimitRefusal { retry_after: None })),
);
let error = service
.prepare_send(Uuid::nil(), &send_request(), None)
@@ -119,7 +178,7 @@ async fn quota_refusal_is_a_preparation_error_before_token_or_dispatch() {
assert!(matches!(error, EmailApiError::RateLimited { .. }));
assert_eq!(
*calls.lock().unwrap(),
- [Call::RateLimit(
+ [Call::Acquire(
Uuid::nil(),
super::ApiOperationKind::SendMessage
)]
@@ -134,7 +193,7 @@ async fn recovery_uses_read_quota_and_does_not_send() {
calls: calls.clone(),
},
FakeTokenSource::new(calls.clone(), Ok(AccessToken::new("access-token"))),
- FakeRateLimiter::new(calls.clone(), Ok(())),
+ FakeGate::new(calls.clone(), Ok(())),
);
let sent = service
.find_sent_message(Uuid::nil(), "[email protected]")
@@ -145,9 +204,10 @@ async fn recovery_uses_read_quota_and_does_not_send() {
assert_eq!(
*calls.lock().unwrap(),
[
- Call::RateLimit(Uuid::nil(), super::ApiOperationKind::ListMessages),
+ Call::Acquire(Uuid::nil(), super::ApiOperationKind::ListMessages),
Call::Token(Uuid::nil(), TokenFreshness::Cached),
Call::Repository("find_sent_message"),
+ Call::Finish(Uuid::nil(), None),
]
);
}
@@ -160,7 +220,7 @@ async fn send_uses_send_cost_and_returns_ids_without_mutating_message() {
calls: calls.clone(),
},
FakeTokenSource::new(calls.clone(), Ok(AccessToken::new("access-token"))),
- FakeRateLimiter::new(calls.clone(), Ok(())),
+ FakeGate::new(calls.clone(), Ok(())),
);
let request = send_request();
@@ -177,9 +237,10 @@ async fn send_uses_send_cost_and_returns_ids_without_mutating_message() {
assert_eq!(
*calls.lock().unwrap(),
vec![
- Call::RateLimit(Uuid::nil(), super::ApiOperationKind::SendMessage),
+ Call::Acquire(Uuid::nil(), super::ApiOperationKind::SendMessage),
Call::Token(Uuid::nil(), TokenFreshness::Cached),
Call::Repository("send_message"),
+ Call::Finish(Uuid::nil(), None),
]
);
}
diff --git a/crates/email_api_client/src/domain/service/subscription.rs b/crates/email_api_client/src/domain/service/subscription.rs
index ec95ee3c78..d55bf2e4a1 100644
--- a/crates/email_api_client/src/domain/service/subscription.rs
+++ b/crates/email_api_client/src/domain/service/subscription.rs
@@ -2,60 +2,69 @@ use models_email::service::link::Link;
use uuid::Uuid;
use super::super::models::{EmailApiError, ProviderSubscription, TokenFreshness};
-use super::super::ports::{MailboxSubscriptionClient, ProviderRateLimiter, ProviderTokenSource};
+use super::super::ports::{MailboxSubscriptionClient, ProviderRequestGate, ProviderTokenSource};
use super::{ApiOperationKind, EmailApiClientServiceImpl, log_operation_error};
impl<R, T, L> EmailApiClientServiceImpl<R, T, L>
where
R: MailboxSubscriptionClient,
T: ProviderTokenSource,
- L: ProviderRateLimiter,
+ L: ProviderRequestGate,
{
/// Registers or renews the linked mailbox's notification subscription.
- #[tracing::instrument(skip(self))]
+ #[tracing::instrument(skip(self), fields(gate_in_flight = tracing::field::Empty, gate_cap = tracing::field::Empty))]
pub async fn register_subscription(
&self,
link_id: Uuid,
) -> Result<ProviderSubscription, EmailApiError> {
- let access_token = self.prepare(link_id, ApiOperationKind::Subscribe).await?;
- self.repository
- .subscribe(&access_token)
- .await
- .inspect_err(log_operation_error)
+ self.call_with_token(link_id, ApiOperationKind::Subscribe, |token| async move {
+ self.repository
+ .subscribe(&token)
+ .await
+ .inspect_err(log_operation_error)
+ })
+ .await
}
/// Registers a subscription using a freshly fetched token.
///
/// This is intended for mailbox initialization, where a cached access token
/// could outlive a revoked provider grant.
- #[tracing::instrument(skip(self, link), fields(link_id = %link.id))]
+ #[tracing::instrument(skip(self, link), fields(link_id = %link.id, gate_in_flight = tracing::field::Empty, gate_cap = tracing::field::Empty))]
pub async fn register_subscription_without_cache(
&self,
link: &Link,
) -> Result<ProviderSubscription, EmailApiError> {
- self.check_rate_limit(link.id, ApiOperationKind::Subscribe)
- .await?;
- let access_token = self
- .token_source
- .get_access_token_for_link(link, TokenFreshness::Fresh)
- .await
- .map_err(super::map_token_error)
- .inspect_err(log_operation_error)?;
-
- self.repository
- .subscribe(&access_token)
- .await
- .inspect_err(log_operation_error)
+ let token = async {
+ self.token_source
+ .get_access_token_for_link(link, TokenFreshness::Fresh)
+ .await
+ .map_err(super::map_token_error)
+ };
+ self.attempt(
+ link.id,
+ ApiOperationKind::Subscribe,
+ token,
+ |token| async move {
+ self.repository
+ .subscribe(&token)
+ .await
+ .inspect_err(log_operation_error)
+ },
+ )
+ .await?
}
/// Stops the linked mailbox's notification subscription.
- #[tracing::instrument(skip(self))]
+ #[tracing::instrument(skip(self), fields(gate_in_flight = tracing::field::Empty, gate_cap = tracing::field::Empty))]
pub async fn stop_subscription(&self, link_id: Uuid) -> Result<(), EmailApiError> {
- let access_token = self.prepare(link_id, ApiOperationKind::Unsubscribe).await?;
- self.repository
- .unsubscribe(&access_token)
- .await
- .inspect_err(log_operation_error)
+ self.call_with_token(link_id, ApiOperationKind::Unsubscribe, |token| async move {
+ self.repository
+ .unsubscribe(&token)
+ .await
+ .inspect_err(log_operation_error)
+ })
+ .await
}
/// Stops the mailbox's subscription without token-health side effects.
@@ -63,21 +72,26 @@ where
/// This is intended for link teardown: a grant that died since the last
/// probe must not set `needs_reauth` or fan out a reauthorization
/// notification for an inbox that is being intentionally removed.
- #[tracing::instrument(skip(self, link), fields(link_id = %link.id))]
+ #[tracing::instrument(skip(self, link), fields(link_id = %link.id, gate_in_flight = tracing::field::Empty, gate_cap = tracing::field::Empty))]
pub async fn stop_subscription_for_link(&self, link: &Link) -> Result<(), EmailApiError> {
- self.check_rate_limit(link.id, ApiOperationKind::Unsubscribe)
- .await?;
- let access_token = self
- .token_source
- .get_access_token_health_neutral(link, TokenFreshness::Cached)
- .await
- .map_err(super::map_token_error)
- .inspect_err(log_operation_error)?;
-
- self.repository
- .unsubscribe(&access_token)
- .await
- .inspect_err(log_operation_error)
+ let token = async {
+ self.token_source
+ .get_access_token_health_neutral(link, TokenFreshness::Cached)
+ .await
+ .map_err(super::map_token_error)
+ };
+ self.attempt(
+ link.id,
+ ApiOperationKind::Unsubscribe,
+ token,
+ |token| async move {
+ self.repository
+ .unsubscribe(&token)
+ .await
+ .inspect_err(log_operation_error)
+ },
+ )
+ .await?
}
}
diff --git a/crates/email_api_client/src/domain/service/subscription/test.rs b/crates/email_api_client/src/domain/service/subscription/test.rs
index 037669385d..a47818c2b4 100644
--- a/crates/email_api_client/src/domain/service/subscription/test.rs
+++ b/crates/email_api_client/src/domain/service/subscription/test.rs
@@ -6,7 +6,7 @@ use super::super::super::models::{
AccessToken, EmailApiError, ProviderSubscription, SyncCursor, TokenFreshness,
};
use super::super::super::ports::MailboxSubscriptionClient;
-use super::super::test_support::{Call, FakeRateLimiter, FakeTokenSource, call_log};
+use super::super::test_support::{Call, FakeGate, FakeTokenSource, call_log};
use super::EmailApiClientServiceImpl;
#[derive(Clone)]
@@ -54,9 +54,10 @@ async fn registration_and_stop_use_matching_costs_and_operations() {
assert_eq!(
*calls.lock().unwrap(),
vec![
- Call::RateLimit(Uuid::nil(), operation),
+ Call::Acquire(Uuid::nil(), operation),
Call::Token(Uuid::nil(), TokenFreshness::Cached),
Call::Repository(repository_call),
+ Call::Finish(Uuid::nil(), None),
]
);
}
@@ -75,9 +76,10 @@ async fn registration_can_bypass_the_token_cache_for_initialization() {
assert_eq!(
*calls.lock().unwrap(),
vec![
- Call::RateLimit(Uuid::nil(), super::ApiOperationKind::Subscribe),
+ Call::Acquire(Uuid::nil(), super::ApiOperationKind::Subscribe),
Call::Token(Uuid::nil(), TokenFreshness::Fresh),
Call::Repository("subscribe"),
+ Call::Finish(Uuid::nil(), None),
]
);
}
@@ -92,9 +94,10 @@ async fn teardown_stop_uses_the_health_neutral_token_path() {
assert_eq!(
*calls.lock().unwrap(),
vec![
- Call::RateLimit(Uuid::nil(), super::ApiOperationKind::Unsubscribe),
+ Call::Acquire(Uuid::nil(), super::ApiOperationKind::Unsubscribe),
Call::HealthNeutralToken(Uuid::nil(), TokenFreshness::Cached),
Call::Repository("unsubscribe"),
+ Call::Finish(Uuid::nil(), None),
]
);
}
@@ -117,12 +120,12 @@ fn link() -> Link {
fn service(
calls: super::super::test_support::CallLog,
-) -> EmailApiClientServiceImpl<SubscriptionClient, FakeTokenSource, FakeRateLimiter> {
+) -> EmailApiClientServiceImpl<SubscriptionClient, FakeTokenSource, FakeGate> {
EmailApiClientServiceImpl::new(
SubscriptionClient {
calls: calls.clone(),
},
FakeTokenSource::new(calls.clone(), Ok(AccessToken::new("access-token"))),
- FakeRateLimiter::new(calls, Ok(())),
+ FakeGate::new(calls, Ok(())),
)
}
diff --git a/crates/email_api_client/src/domain/service/sync.rs b/crates/email_api_client/src/domain/service/sync.rs
index 51ebf593bb..b5e5984a81 100644
--- a/crates/email_api_client/src/domain/service/sync.rs
+++ b/crates/email_api_client/src/domain/service/sync.rs
@@ -1,28 +1,26 @@
use uuid::Uuid;
use super::super::models::{ChangeBatch, EmailApiError, SyncCursor};
-use super::super::ports::{MailboxSyncClient, ProviderRateLimiter, ProviderTokenSource};
-use super::{ApiOperationKind, EmailApiClientServiceImpl, log_operation_error};
+use super::super::ports::{MailboxSyncClient, ProviderRequestGate, ProviderTokenSource};
+use super::{ApiOperationKind, EmailApiClientServiceImpl};
impl<R, T, L> EmailApiClientServiceImpl<R, T, L>
where
R: MailboxSyncClient,
T: ProviderTokenSource,
- L: ProviderRateLimiter,
+ L: ProviderRequestGate,
{
/// Returns the number of threads reported by a linked mailbox.
- #[tracing::instrument(skip(self))]
+ #[tracing::instrument(skip(self), fields(retry_count = 0, gate_in_flight = tracing::field::Empty, gate_cap = tracing::field::Empty))]
pub async fn get_thread_count(&self, link_id: Uuid) -> Result<u64, EmailApiError> {
- let access_token = self.prepare(link_id, ApiOperationKind::GetProfile).await?;
-
- self.repository
- .get_thread_count(&access_token)
- .await
- .inspect_err(log_operation_error)
+ self.read_with_token(link_id, ApiOperationKind::GetProfile, |token| async move {
+ self.repository.get_thread_count(&token).await
+ })
+ .await
}
/// Lists mailbox changes following `cursor`.
- #[tracing::instrument(skip(self), fields(retry_count = 0))]
+ #[tracing::instrument(skip(self), fields(retry_count = 0, gate_in_flight = tracing::field::Empty, gate_cap = tracing::field::Empty))]
pub async fn list_changes(
&self,
link_id: Uuid,
diff --git a/crates/email_api_client/src/domain/service/sync/test.rs b/crates/email_api_client/src/domain/service/sync/test.rs
index 57feb42077..55523a4a5c 100644
--- a/crates/email_api_client/src/domain/service/sync/test.rs
+++ b/crates/email_api_client/src/domain/service/sync/test.rs
@@ -2,7 +2,7 @@ use std::sync::{Arc, Mutex};
use uuid::Uuid;
-use super::super::test_support::{Call, FakeRateLimiter, FakeTokenSource, call_log};
+use super::super::test_support::{Call, FakeGate, FakeTokenSource, call_log};
use super::*;
use crate::domain::models::{AccessToken, InboxChanges, TokenFreshness};
@@ -34,11 +34,11 @@ impl MailboxSyncClient for SyncClient {
fn service(
repository: SyncClient,
calls: &super::super::test_support::CallLog,
-) -> EmailApiClientServiceImpl<SyncClient, FakeTokenSource, FakeRateLimiter> {
+) -> EmailApiClientServiceImpl<SyncClient, FakeTokenSource, FakeGate> {
EmailApiClientServiceImpl::new(
repository,
FakeTokenSource::new(calls.clone(), Ok(AccessToken::new("token"))),
- FakeRateLimiter::new(calls.clone(), Ok(())),
+ FakeGate::new(calls.clone(), Ok(())),
)
}
@@ -56,8 +56,9 @@ async fn get_thread_count_uses_profile_quota_and_forwards_link_id() {
assert_eq!(
*calls.lock().unwrap(),
vec![
- Call::RateLimit(link_id, ApiOperationKind::GetProfile),
+ Call::Acquire(link_id, ApiOperationKind::GetProfile),
Call::Token(link_id, TokenFreshness::Cached),
+ Call::Finish(link_id, None),
]
);
}
@@ -79,8 +80,9 @@ async fn list_changes_uses_changes_quota_and_forwards_cursor() {
assert_eq!(
*calls.lock().unwrap(),
vec![
- Call::RateLimit(link_id, ApiOperationKind::ListChanges),
+ Call::Acquire(link_id, ApiOperationKind::ListChanges),
Call::Token(link_id, TokenFreshness::Cached),
+ Call::Finish(link_id, None),
]
);
}
diff --git a/crates/email_api_client/src/domain/service/test.rs b/crates/email_api_client/src/domain/service/test.rs
index df81300114..3a2f937652 100644
--- a/crates/email_api_client/src/domain/service/test.rs
+++ b/crates/email_api_client/src/domain/service/test.rs
@@ -1,15 +1,17 @@
use std::sync::{Arc, Mutex};
+use std::task::{Context, Waker};
use std::time::Duration;
use models_email::email::service::label::Label;
use uuid::Uuid;
use super::super::models::{
- AccessToken, ApiOperationKind, EmailApiError, RateLimitRefusal, TokenError, TokenFreshness,
+ AccessToken, ApiOperationKind, EmailApiError, RateLimitOrigin, RateLimitRefusal, TokenError,
+ TokenFreshness,
};
use super::super::ports::MailboxLabelClient;
use super::EmailApiClientServiceImpl;
-use super::test_support::{Call, FakeRateLimiter, FakeRepository, FakeTokenSource, call_log};
+use super::test_support::{Call, FakeGate, FakeRepository, FakeTokenSource, call_log};
/// Rejects the listed tokens the way Gmail answers a 401.
#[derive(Clone, Default)]
@@ -29,6 +31,31 @@ impl MailboxLabelClient for TokenCheckingLabels {
}
}
+ async fn create_label(
+ &self,
+ _: &AccessToken,
+ _: Uuid,
+ _: &str,
+ ) -> Result<Label, EmailApiError> {
+ Err(EmailApiError::RateLimited {
+ retry_after: None,
+ origin: RateLimitOrigin::Provider { concurrency: false },
+ })
+ }
+
+ async fn delete_label(&self, _: &AccessToken, _: &str) -> Result<(), EmailApiError> {
+ unreachable!("not used by these tests")
+ }
+}
+
+/// Never answers, like a Gmail request still in flight.
+struct UnansweredLabels;
+
+impl MailboxLabelClient for UnansweredLabels {
+ async fn list_labels(&self, _: &AccessToken, _: Uuid) -> Result<Vec<Label>, EmailApiError> {
+ std::future::pending().await
+ }
+
async fn create_label(
&self,
_: &AccessToken,
@@ -43,6 +70,31 @@ impl MailboxLabelClient for TokenCheckingLabels {
}
}
+#[tokio::test]
+async fn a_caller_cancelled_mid_request_releases_its_admission() {
+ let calls = call_log();
+ let service = EmailApiClientServiceImpl::new(
+ UnansweredLabels,
+ FakeTokenSource::new(calls.clone(), Ok(AccessToken::new("access-token"))),
+ FakeGate::new(calls.clone(), Ok(())),
+ );
+ let mut read = Box::pin(service.list_labels(Uuid::nil()));
+ let poll = read.as_mut().poll(&mut Context::from_waker(Waker::noop()));
+ assert!(poll.is_pending());
+
+ // An HTTP client disconnected, or a lease-keeping select! gave up.
+ drop(read);
+
+ assert_eq!(
+ *calls.lock().unwrap(),
+ vec![
+ Call::Acquire(Uuid::nil(), ApiOperationKind::ListLabels),
+ Call::Token(Uuid::nil(), TokenFreshness::Cached),
+ Call::Abandon(Uuid::nil()),
+ ]
+ );
+}
+
async fn list_labels_with(
rejected: Vec<&'static str>,
fresh: Result<AccessToken, TokenError>,
@@ -56,7 +108,7 @@ async fn list_labels_with(
let service = EmailApiClientServiceImpl::new(
repository,
FakeTokenSource::new(calls.clone(), Ok(AccessToken::new("cached"))).with_fresh(fresh),
- FakeRateLimiter::new(calls.clone(), Ok(())),
+ FakeGate::new(calls.clone(), Ok(())),
);
let result = service
@@ -68,22 +120,23 @@ async fn list_labels_with(
(result, calls, seen)
}
-fn retried_calls() -> Vec<Call> {
- vec![
- Call::RateLimit(Uuid::nil(), ApiOperationKind::ListLabels),
- Call::Token(Uuid::nil(), TokenFreshness::Cached),
- Call::RateLimit(Uuid::nil(), ApiOperationKind::ListLabels),
- Call::Token(Uuid::nil(), TokenFreshness::Fresh),
- ]
-}
-
#[tokio::test]
async fn a_rejected_cached_token_is_retried_with_a_fresh_one() {
let (result, calls, seen) =
list_labels_with(vec!["cached"], Ok(AccessToken::new("fresh"))).await;
assert_eq!(result, Ok(0));
- assert_eq!(calls, retried_calls());
+ assert_eq!(
+ calls,
+ vec![
+ Call::Acquire(Uuid::nil(), ApiOperationKind::ListLabels),
+ Call::Token(Uuid::nil(), TokenFreshness::Cached),
+ Call::Finish(Uuid::nil(), Some(EmailApiError::AuthRequired)),
+ Call::Acquire(Uuid::nil(), ApiOperationKind::ListLabels),
+ Call::Token(Uuid::nil(), TokenFreshness::Fresh),
+ Call::Finish(Uuid::nil(), None),
+ ]
+ );
assert_eq!(seen, ["cached", "fresh"]);
}
@@ -93,7 +146,17 @@ async fn a_revoked_grant_found_by_the_fresh_fetch_stops_the_read() {
list_labels_with(vec!["cached"], Err(TokenError::ReauthRequired)).await;
assert_eq!(result, Err(EmailApiError::AuthRequired));
- assert_eq!(calls, retried_calls());
+ assert_eq!(
+ calls,
+ vec![
+ Call::Acquire(Uuid::nil(), ApiOperationKind::ListLabels),
+ Call::Token(Uuid::nil(), TokenFreshness::Cached),
+ Call::Finish(Uuid::nil(), Some(EmailApiError::AuthRequired)),
+ Call::Acquire(Uuid::nil(), ApiOperationKind::ListLabels),
+ Call::Token(Uuid::nil(), TokenFreshness::Fresh),
+ Call::Finish(Uuid::nil(), None),
+ ]
+ );
assert_eq!(seen, ["cached"]);
}
@@ -103,69 +166,104 @@ async fn a_rejected_fresh_token_is_not_retried_again() {
list_labels_with(vec!["cached", "fresh"], Ok(AccessToken::new("fresh"))).await;
assert_eq!(result, Err(EmailApiError::AuthRequired));
- assert_eq!(calls, retried_calls());
+ assert_eq!(
+ calls,
+ vec![
+ Call::Acquire(Uuid::nil(), ApiOperationKind::ListLabels),
+ Call::Token(Uuid::nil(), TokenFreshness::Cached),
+ Call::Finish(Uuid::nil(), Some(EmailApiError::AuthRequired)),
+ Call::Acquire(Uuid::nil(), ApiOperationKind::ListLabels),
+ Call::Token(Uuid::nil(), TokenFreshness::Fresh),
+ Call::Finish(Uuid::nil(), Some(EmailApiError::AuthRequired)),
+ ]
+ );
assert_eq!(seen, ["cached", "fresh"]);
}
#[tokio::test]
-async fn token_failure_stops_before_repository_after_the_quota_check() {
+async fn token_failure_releases_admission_without_calling_the_provider() {
let calls = call_log();
let service = EmailApiClientServiceImpl::new(
FakeRepository::new(calls.clone()),
FakeTokenSource::new(calls.clone(), Err(TokenError::ReauthRequired)),
- FakeRateLimiter::new(calls.clone(), Ok(())),
+ FakeGate::new(calls.clone(), Ok(())),
);
let result = service
- .prepare(Uuid::nil(), ApiOperationKind::GetMessage)
+ .modify_message_labels(Uuid::nil(), "message-1", &[], &["INBOX".to_string()])
.await;
assert_eq!(result, Err(EmailApiError::AuthRequired));
assert_eq!(
*calls.lock().unwrap(),
vec![
- Call::RateLimit(Uuid::nil(), ApiOperationKind::GetMessage),
+ Call::Acquire(Uuid::nil(), ApiOperationKind::ModifyMessageLabels),
Call::Token(Uuid::nil(), TokenFreshness::Cached),
+ Call::Finish(Uuid::nil(), None),
]
);
}
#[tokio::test]
-async fn rate_limit_refusal_stops_before_the_token_dance_and_repository() {
+async fn admission_refusal_stops_before_the_token_dance_and_repository() {
let calls = call_log();
let retry_after = Duration::from_secs(17);
let service = EmailApiClientServiceImpl::new(
FakeRepository::new(calls.clone()),
FakeTokenSource::new(calls.clone(), Ok(AccessToken::new("access-token"))),
- FakeRateLimiter::new(calls.clone(), Err(RateLimitRefusal::new(Some(retry_after)))),
+ FakeGate::new(calls.clone(), Err(RateLimitRefusal::new(Some(retry_after)))),
);
- let result = service
- .prepare(Uuid::nil(), ApiOperationKind::SendMessage)
- .await;
+ let result = service.get_thread_count(Uuid::nil()).await;
assert_eq!(
result,
Err(EmailApiError::RateLimited {
retry_after: Some(retry_after),
- origin: crate::domain::models::RateLimitOrigin::Local,
+ origin: RateLimitOrigin::Local,
})
);
- // A refused attempt must not pay the token acquisition (SELECT + Redis +
- // possible auth-service refresh + health write).
+ // A refused attempt holds no permit and must not pay the token acquisition
+ // (SELECT + Redis + possible auth-service refresh + health write).
assert_eq!(
*calls.lock().unwrap(),
- vec![Call::RateLimit(Uuid::nil(), ApiOperationKind::SendMessage)]
+ vec![Call::Acquire(Uuid::nil(), ApiOperationKind::GetProfile)]
+ );
+}
+
+#[tokio::test]
+async fn a_provider_rate_limit_is_reported_to_the_gate_when_releasing_admission() {
+ let calls = call_log();
+ let service = EmailApiClientServiceImpl::new(
+ TokenCheckingLabels::default(),
+ FakeTokenSource::new(calls.clone(), Ok(AccessToken::new("access-token"))),
+ FakeGate::new(calls.clone(), Ok(())),
+ );
+ let throttled = EmailApiError::RateLimited {
+ retry_after: None,
+ origin: RateLimitOrigin::Provider { concurrency: false },
+ };
+
+ let result = service.create_label(Uuid::nil(), "Receipts").await;
+
+ assert_eq!(result.err(), Some(throttled.clone()));
+ assert_eq!(
+ *calls.lock().unwrap(),
+ vec![
+ Call::Acquire(Uuid::nil(), ApiOperationKind::CreateLabel),
+ Call::Token(Uuid::nil(), TokenFreshness::Cached),
+ Call::Finish(Uuid::nil(), Some(throttled)),
+ ]
);
}
#[tokio::test]
-async fn explicit_token_probe_honors_requested_freshness_without_quota_check() {
+async fn explicit_token_probe_honors_requested_freshness_without_admission() {
let calls = call_log();
let service = EmailApiClientServiceImpl::new(
FakeRepository::new(calls.clone()),
FakeTokenSource::new(calls.clone(), Ok(AccessToken::new("access-token"))),
- FakeRateLimiter::new(calls.clone(), Ok(())),
+ FakeGate::new(calls.clone(), Ok(())),
);
let result = service
@@ -203,7 +301,7 @@ async fn token_errors_preserve_transient_and_permanent_classification() {
let service = EmailApiClientServiceImpl::new(
FakeRepository::new(calls.clone()),
FakeTokenSource::new(calls.clone(), Err(token_error)),
- FakeRateLimiter::new(calls, Ok(())),
+ FakeGate::new(calls, Ok(())),
);
assert_eq!(
diff --git a/crates/email_api_client/src/domain/service/test_support.rs b/crates/email_api_client/src/domain/service/test_support.rs
index ad12f44305..d7666b3582 100644
--- a/crates/email_api_client/src/domain/service/test_support.rs
+++ b/crates/email_api_client/src/domain/service/test_support.rs
@@ -7,19 +7,22 @@ use uuid::Uuid;
use super::super::models::{
AccessToken, ApiOperationKind, ChangeBatch, EmailApiError, ProviderSubscription,
- RateLimitRefusal, SendRequest, SentIds, SyncCursor, ThreadListPage, TokenError, TokenFreshness,
+ RateLimitRefusal, RequestPermit, SendRequest, SentIds, SyncCursor, ThreadListPage, TokenError,
+ TokenFreshness,
};
use super::super::ports::{
MailboxAttachmentClient, MailboxBlocklistClient, MailboxContactsClient, MailboxLabelClient,
MailboxMessageClient, MailboxSendClient, MailboxSubscriptionClient, MailboxSyncClient,
- ProviderRateLimiter, ProviderTokenSource,
+ ProviderRequestGate, ProviderTokenSource,
};
#[derive(Debug, Clone, PartialEq, Eq)]
pub(super) enum Call {
Token(Uuid, TokenFreshness),
HealthNeutralToken(Uuid, TokenFreshness),
- RateLimit(Uuid, ApiOperationKind),
+ Acquire(Uuid, ApiOperationKind),
+ Finish(Uuid, Option<EmailApiError>),
+ Abandon(Uuid),
Repository(&'static str),
}
@@ -94,28 +97,41 @@ impl ProviderTokenSource for FakeTokenSource {
}
#[derive(Clone)]
-pub(super) struct FakeRateLimiter {
+pub(super) struct FakeGate {
calls: CallLog,
result: Result<(), RateLimitRefusal>,
}
-impl FakeRateLimiter {
+impl FakeGate {
pub(super) fn new(calls: CallLog, result: Result<(), RateLimitRefusal>) -> Self {
Self { calls, result }
}
}
-impl ProviderRateLimiter for FakeRateLimiter {
- async fn check_rate_limit(
+impl ProviderRequestGate for FakeGate {
+ async fn acquire(
&self,
link_id: Uuid,
operation: ApiOperationKind,
- ) -> Result<(), RateLimitRefusal> {
+ ) -> Result<RequestPermit, RateLimitRefusal> {
self.calls
.lock()
.unwrap()
- .push(Call::RateLimit(link_id, operation));
- self.result.clone()
+ .push(Call::Acquire(link_id, operation));
+ let calls = self.calls.clone();
+ self.result.clone().map(|()| {
+ RequestPermit::new(link_id, Uuid::nil(), move |link_id, _| {
+ calls.lock().unwrap().push(Call::Abandon(link_id));
+ })
+ })
+ }
+
+ async fn finish(&self, permit: RequestPermit, error: Option<&EmailApiError>) {
+ self.calls
+ .lock()
+ .unwrap()
+ .push(Call::Finish(permit.link_id(), error.cloned()));
+ permit.disarm();
}
}
diff --git a/crates/email_api_client/src/outbound/gmail.rs b/crates/email_api_client/src/outbound/gmail.rs
index 2e53e51fff..3a0cd55315 100644
--- a/crates/email_api_client/src/outbound/gmail.rs
+++ b/crates/email_api_client/src/outbound/gmail.rs
@@ -81,6 +81,10 @@ fn is_quota_403_body(body: &str) -> bool {
QUOTA_403_REASONS.iter().any(|reason| body.contains(reason))
}
+/// Gmail's per-second 429 shares this one's `rateLimitExceeded` reason; only
+/// the message tells the concurrency limit apart.
+const CONCURRENCY_429_MESSAGE: &str = "Too many concurrent requests for user";
+
pub(crate) fn map_gmail_error(error: GmailApiHttpError) -> EmailApiError {
match error {
GmailApiHttpError::Http {
@@ -91,14 +95,16 @@ pub(crate) fn map_gmail_error(error: GmailApiHttpError) -> EmailApiError {
401 => EmailApiError::AuthRequired,
403 if is_quota_403_body(&body) => EmailApiError::RateLimited {
retry_after,
- origin: RateLimitOrigin::Provider,
+ origin: RateLimitOrigin::Provider { concurrency: false },
},
403 => EmailApiError::Forbidden,
404 => EmailApiError::NotFound,
409 => EmailApiError::Conflict,
429 => EmailApiError::RateLimited {
retry_after,
- origin: RateLimitOrigin::Provider,
+ origin: RateLimitOrigin::Provider {
+ concurrency: body.contains(CONCURRENCY_429_MESSAGE),
+ },
},
500..=599 => EmailApiError::Transient {
message: format!("Gmail API returned {status}: {body}"),
diff --git a/crates/email_api_client/src/outbound/gmail/test/error.rs b/crates/email_api_client/src/outbound/gmail/test/error.rs
index 3b26196c4e..c3e8acc5f9 100644
--- a/crates/email_api_client/src/outbound/gmail/test/error.rs
+++ b/crates/email_api_client/src/outbound/gmail/test/error.rs
@@ -49,7 +49,33 @@ fn maps_provider_statuses_centrally() {
)),
EmailApiError::RateLimited {
retry_after: Some(retry_after),
- origin: RateLimitOrigin::Provider,
+ origin: RateLimitOrigin::Provider { concurrency: false },
+ }
+ );
+}
+
+#[test]
+fn only_the_concurrent_requests_429_is_a_concurrency_limit() {
+ assert_eq!(
+ map_gmail_error(http_error(
+ StatusCode::TOO_MANY_REQUESTS,
+ r#"{"error":{"code":429,"message":"Too many concurrent requests for user.","errors":[{"reason":"rateLimitExceeded"}]}}"#,
+ None,
+ )),
+ EmailApiError::RateLimited {
+ retry_after: None,
+ origin: RateLimitOrigin::Provider { concurrency: true },
+ }
+ );
+ assert_eq!(
+ map_gmail_error(http_error(
+ StatusCode::TOO_MANY_REQUESTS,
+ r#"{"error":{"code":429,"message":"User-rate limit exceeded. Retry after 2026-10-25T10:00:00Z","errors":[{"reason":"rateLimitExceeded"}]}}"#,
+ None,
+ )),
+ EmailApiError::RateLimited {
+ retry_after: None,
+ origin: RateLimitOrigin::Provider { concurrency: false },
}
);
}
@@ -64,7 +90,7 @@ fn quota_403s_are_rate_limited_and_plain_403s_stay_forbidden() {
)),
EmailApiError::RateLimited {
retry_after: None,
- origin: RateLimitOrigin::Provider,
+ origin: RateLimitOrigin::Provider { concurrency: false },
}
);
@@ -77,7 +103,7 @@ fn quota_403s_are_rate_limited_and_plain_403s_stay_forbidden() {
)),
EmailApiError::RateLimited {
retry_after: Some(retry_after),
- origin: RateLimitOrigin::Provider,
+ origin: RateLimitOrigin::Provider { concurrency: false },
}
);
@@ -90,7 +116,7 @@ fn quota_403s_are_rate_limited_and_plain_403s_stay_forbidden() {
)),
EmailApiError::RateLimited {
retry_after: None,
- origin: RateLimitOrigin::Provider,
+ origin: RateLimitOrigin::Provider { concurrency: false },
},
"reason {reason} should map to RateLimited",
);
diff --git a/crates/email_api_client/src/outbound/outlook/test.rs b/crates/email_api_client/src/outbound/outlook/test.rs
index a492bfc112..e7ec18fe5d 100644
--- a/crates/email_api_client/src/outbound/outlook/test.rs
+++ b/crates/email_api_client/src/outbound/outlook/test.rs
@@ -163,7 +163,7 @@ async fn throttling_respects_provider_retry_after() {
error,
EmailApiError::RateLimited {
retry_after: Some(Duration::from_secs(17)),
- origin: RateLimitOrigin::Provider
+ origin: RateLimitOrigin::Provider { concurrency: false }
}
);
}
diff --git a/crates/email_api_client/src/outbound/outlook/transport.rs b/crates/email_api_client/src/outbound/outlook/transport.rs
index 74fe6a153f..664c01083d 100644
--- a/crates/email_api_client/src/outbound/outlook/transport.rs
+++ b/crates/email_api_client/src/outbound/outlook/transport.rs
@@ -197,7 +197,7 @@ impl OutlookApiClientRepository {
StatusCode::CONFLICT | StatusCode::PRECONDITION_FAILED => EmailApiError::Conflict,
StatusCode::TOO_MANY_REQUESTS => EmailApiError::RateLimited {
retry_after,
- origin: RateLimitOrigin::Provider,
+ origin: RateLimitOrigin::Provider { concurrency: false },
},
status if status.is_server_error() || status == StatusCode::REQUEST_TIMEOUT => {
EmailApiError::Transient {
diff --git a/crates/email_db_client/src/attachments/provider/upload.rs b/crates/email_db_client/src/attachments/provider/upload.rs
index 15314ce5c6..225a187691 100644
--- a/crates/email_db_client/src/attachments/provider/upload.rs
+++ b/crates/email_db_client/src/attachments/provider/upload.rs
@@ -572,6 +572,19 @@ pub async fn new_email_media_atts(
Ok(attachments)
}
+/// Releases an attachment's upload claim so the next claiming query picks it up again.
+#[tracing::instrument(skip(db), err)]
+pub async fn release_upload_claim(db: &Pool<Postgres>, attachment_id: Uuid) -> anyhow::Result<()> {
+ sqlx::query!(
+ "UPDATE email_attachments SET upload_claimed_at = NULL WHERE id = $1",
+ attachment_id
+ )
+ .execute(db)
+ .await?;
+
+ Ok(())
+}
+
pub async fn fetch_attachment_upload_metadata_by_id(
db: &Pool<Postgres>,
attachment_id: Uuid,
diff --git a/crates/email_db_client/src/threads/insert.rs b/crates/email_db_client/src/threads/insert.rs
index 8a15d13cdb..4030d64236 100644
--- a/crates/email_db_client/src/threads/insert.rs
+++ b/crates/email_db_client/src/threads/insert.rs
@@ -156,13 +156,14 @@ pub async fn insert_thread(
Ok(existing_id)
}
-/// inserts a thread object into the database that has no metadata or rizz
+/// Inserts a thread with no metadata, returning its id, or `None` when the
+/// mailbox already has the thread, including when a concurrent insert won.
#[tracing::instrument(skip(pool), err)]
pub async fn insert_blank_thread(
pool: &PgPool,
thread_provider_id: &str,
link_id: Uuid,
-) -> anyhow::Result<Uuid> {
+) -> anyhow::Result<Option<Uuid>> {
let thread = thread::Thread {
db_id: macro_uuid::generate_uuid_v7(),
provider_id: Some(thread_provider_id.to_string()),
@@ -180,5 +181,5 @@ pub async fn insert_blank_thread(
let mut tx = pool.begin().await?;
let id = insert_thread(&mut tx, &thread, link_id).await?;
tx.commit().await?;
- Ok(id)
+ Ok((id == thread.db_id).then_some(id))
}
diff --git a/crates/email_db_client/src/threads/insert/test.rs b/crates/email_db_client/src/threads/insert/test.rs
index 2746eaadd6..42c6582ad5 100644
--- a/crates/email_db_client/src/threads/insert/test.rs
+++ b/crates/email_db_client/src/threads/insert/test.rs
@@ -112,9 +112,9 @@ async fn blank_upsert_preserves_populated_timestamp(pool: Pool<Postgres>) -> any
// A redelivered backfill blank insert must not wipe the timestamp or
// rewrite the row.
- let upserted_thread_id = insert_blank_thread(&pool, "prov1", link_id).await?;
+ let inserted = insert_blank_thread(&pool, "prov1", link_id).await?;
- assert_eq!(upserted_thread_id, existing_thread_id);
+ assert_eq!(inserted, None, "the mailbox already had the thread");
let after = get_thread_by_id_and_link_id(&pool, existing_thread_id, link_id)
.await?
.expect("provider thread should still exist");
@@ -137,7 +137,9 @@ async fn blank_upsert_preserves_populated_timestamp(pool: Pool<Postgres>) -> any
async fn blank_insert_creates_new_thread(pool: Pool<Postgres>) -> anyhow::Result<()> {
let link_id = Uuid::parse_str("00000000-0000-0000-0000-000000000001")?;
- let thread_id = insert_blank_thread(&pool, "prov-brand-new", link_id).await?;
+ let thread_id = insert_blank_thread(&pool, "prov-brand-new", link_id)
+ .await?
+ .expect("the thread is new");
let thread = get_thread_by_id_and_link_id(&pool, thread_id, link_id)
.await?
@@ -148,6 +150,28 @@ async fn blank_insert_creates_new_thread(pool: Pool<Postgres>) -> anyhow::Result
Ok(())
}
+#[sqlx::test(
+ migrator = "MACRO_DB_MIGRATIONS",
+ fixtures(path = "../../../fixtures", scripts("links"))
+)]
+async fn only_one_of_two_concurrent_blank_inserts_creates_the_thread(
+ pool: Pool<Postgres>,
+) -> anyhow::Result<()> {
+ let link_id = Uuid::parse_str("00000000-0000-0000-0000-000000000001")?;
+
+ let (first, second) = tokio::join!(
+ insert_blank_thread(&pool, "prov-raced", link_id),
+ insert_blank_thread(&pool, "prov-raced", link_id)
+ );
+
+ assert_eq!(
+ [first?, second?].iter().flatten().count(),
+ 1,
+ "only the winning insert backfills the thread's messages"
+ );
+ Ok(())
+}
+
#[sqlx::test(
migrator = "MACRO_DB_MIGRATIONS",
fixtures(path = "../../../fixtures", scripts("links"))
diff --git a/crates/gmail_client/AGENTS.md b/crates/gmail_client/AGENTS.md
index 6edc3a93b9..137f5bad3b 100644
--- a/crates/gmail_client/AGENTS.md
+++ b/crates/gmail_client/AGENTS.md
@@ -42,6 +42,9 @@ contacts, watch) behind a single public facade; most modules are
- Any new error path should go through `sanitize_error_body` (or an
equivalent) before logging — Gmail error bodies can contain email addresses.
+- Requests time out after 60 s (`REQUEST_TIMEOUT`). The email service's
+ Gmail request gate holds a permit for each call and sizes its lease above
+ this (`PERMIT_MS` in `gmail_gate.rs`); keep the two in step.
## Related
diff --git a/crates/gmail_client/src/lib.rs b/crates/gmail_client/src/lib.rs
index 25efce93fd..685da92cb8 100644
--- a/crates/gmail_client/src/lib.rs
+++ b/crates/gmail_client/src/lib.rs
@@ -16,8 +16,13 @@ mod test;
use regex::Regex;
use std::sync::LazyLock;
+use std::time::Duration;
const MAX_ERROR_BODY_LEN: usize = 1024;
+// A stalled request fails instead of holding its worker, and its mailbox's
+// request-gate permit, indefinitely. Callers size permit leases above this.
+const REQUEST_TIMEOUT: Duration = Duration::from_secs(60);
+const CONNECT_TIMEOUT: Duration = Duration::from_secs(10);
static EMAIL_REGEX: LazyLock<Regex> =
LazyLock::new(|| Regex::new(r"[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\.[a-zA-Z]{2,}").unwrap());
@@ -108,8 +113,14 @@ impl GmailClient {
audience: String,
) -> Self {
Self {
- inner: reqwest::Client::new(),
+ inner: reqwest::Client::builder()
+ .timeout(REQUEST_TIMEOUT)
+ .connect_timeout(CONNECT_TIMEOUT)
+ .build()
+ .expect("Gmail HTTP client should initialize"),
send_client: reqwest::Client::builder()
+ .timeout(REQUEST_TIMEOUT)
+ .connect_timeout(CONNECT_TIMEOUT)
.retry(reqwest::retry::never())
.redirect(reqwest::redirect::Policy::none())
.build()
diff --git a/crates/sqs_client/src/email.rs b/crates/sqs_client/src/email.rs
index 2b9df57aa2..1306e11e70 100644
--- a/crates/sqs_client/src/email.rs
+++ b/crates/sqs_client/src/email.rs
@@ -8,6 +8,10 @@ use models_email::service::pubsub::SFSUploaderMessage;
use models_email::service::pubsub::ScheduledPubsubMessage;
use uuid::Uuid;
+/// Message attribute counting how often a backfill operation has been sent
+/// back to its queue by [`SQS::defer_email_backfill_message`].
+pub const BACKFILL_DEFERRALS_ATTRIBUTE: &str = "deferrals";
+
impl SQS {
pub fn email_link_manager_queue(mut self, link_manager_queue: &str) -> Self {
self.link_manager_queue = Some(link_manager_queue.to_string());
@@ -66,6 +70,35 @@ impl SQS {
Err(anyhow::anyhow!("email_backfill_queue is not configured"))
}
+ /// Sends a backfill operation back to the backfill queue as a new message
+ /// that becomes visible after `delay_seconds`. Unlike waiting out the
+ /// visibility timeout, the wait uses none of the operation's deliveries.
+ /// `deferrals` is recorded in [`BACKFILL_DEFERRALS_ATTRIBUTE`].
+ #[tracing::instrument(skip(self), err)]
+ pub async fn defer_email_backfill_message(
+ &self,
+ message: &BackfillPubsubMessage,
+ delay_seconds: i32,
+ deferrals: u32,
+ ) -> anyhow::Result<()> {
+ let Some(email_backfill_queue) = &self.email_backfill_queue else {
+ anyhow::bail!("email_backfill_queue is not configured");
+ };
+ let deferrals = aws_sdk_sqs::types::MessageAttributeValue::builder()
+ .data_type("Number")
+ .string_value(deferrals.to_string())
+ .build()?;
+ self.inner
+ .send_message()
+ .queue_url(email_backfill_queue)
+ .message_body(serde_json::to_string(&message)?)
+ .delay_seconds(delay_seconds)
+ .message_attributes(BACKFILL_DEFERRALS_ATTRIBUTE, deferrals)
+ .send()
+ .await?;
+ Ok(())
+ }
+
/// Sends a message to the email crm cleanup queue
#[tracing::instrument(skip(self), err)]
pub async fn enqueue_email_crm_cleanup_message(
diff --git a/crates/workspace-hack/Cargo.toml b/crates/workspace-hack/Cargo.toml
index da66ea6e9a..6e869b1514 100644
--- a/crates/workspace-hack/Cargo.toml
+++ b/crates/workspace-hack/Cargo.toml
@@ -120,7 +120,7 @@ prost = { version = "0.12", features = ["prost-derive"] }
rand-274715c4dabd11b0 = { package = "rand", version = "0.9" }
rand-c38e5c1d305a1b54 = { package = "rand", version = "0.8", features = ["small_rng"] }
rand_core = { version = "0.6", default-features = false, features = ["std"] }
-redis = { version = "1", features = ["cluster-async", "tokio-rustls-comp"] }
+redis = { version = "1", features = ["cluster-async", "connection-manager", "tokio-rustls-comp"] }
regex = { version = "1" }
regex-automata = { version = "0.4" }
regex-syntax = { version = "0.8" }
@@ -284,7 +284,7 @@ quote = { version = "1" }
rand-274715c4dabd11b0 = { package = "rand", version = "0.9" }
rand-c38e5c1d305a1b54 = { package = "rand", version = "0.8", features = ["small_rng"] }
rand_core = { version = "0.6", default-features = false, features = ["std"] }
-redis = { version = "1", features = ["cluster-async", "tokio-rustls-comp"] }
+redis = { version = "1", features = ["cluster-async", "connection-manager", "tokio-rustls-comp"] }
regex = { version = "1" }
regex-automata = { version = "0.4" }
regex-syntax = { version = "0.8" }
diff --git a/services/email_service/AGENTS.md b/services/email_service/AGENTS.md
index 1dd25a9eb8..1eeb13a2c5 100644
--- a/services/email_service/AGENTS.md
+++ b/services/email_service/AGENTS.md
@@ -67,6 +67,20 @@ the gateway `/email` prefix.
calendar refresh so unrelated invitation updates retain teammates' OOO rows.
- Spawn `run_transfers` regardless of the Outlook flags. Gmail inbox moves
finish there; with writes off it pauses only the Outlook retire step.
+- Every `GmailApi` call takes a permit from `RedisGmailRequestGate`
+ (`outbound/email_api/gmail_gate.rs`): per mailbox it caps requests in
+ flight and quota units per window, and pauses after a Gmail 429 (live work
+ for a second, backfill for at least 10 s). A "Too many concurrent requests"
+ 429 also lowers that mailbox's in-flight cap, which then rises by one a
+ minute back to five; backfill gets what the cap leaves above two. It fails
+ closed, so a Redis outage surfaces as local rate limits. Consumers must
+ requeue or redeliver a rate-limited message, never drop it. Up to twelve
+ refused backfill requests per mailbox wait up to 20 s for room so backfill's
+ slots don't idle; the rest are sent back as new, delayed messages
+ (`pubsub/backfill/deferral.rs`), because waiting out the visibility timeout
+ would spend the queue's 20 deliveries.
+ Inbox sync releases the upload claims of a refused attachment fetch and
+ requeues the upsert, because a claimed attachment is never offered again.
- Header addresses that Gmail accepts can fail `MacroUserIdStr::try_from_email`.
Skip the bad address with a WARN that omits it (the parse error echoes it)
instead of failing the step; see `pubsub::util::build_contact_connections`.
diff --git a/services/email_service/Cargo.toml b/services/email_service/Cargo.toml
index 8eff8db927..edbc2cb25d 100644
--- a/services/email_service/Cargo.toml
+++ b/services/email_service/Cargo.toml
@@ -144,8 +144,10 @@ models_pagination = { path = "../../crates/models_pagination", features = ["axum
models_permissions = { path = "../../crates/models_permissions" }
once_cell = { workspace = true }
models_properties = { path = "../../crates/models_properties" }
+rand = { workspace = true }
redis = { workspace = true, features = [
"cluster-async",
+ "connection-manager",
"tokio-rustls-comp",
] }
regex = { workspace = true }
diff --git a/services/email_service/src/api/email/init.rs b/services/email_service/src/api/email/init.rs
index fb982885d1..eff3d7ae5a 100644
--- a/services/email_service/src/api/email/init.rs
+++ b/services/email_service/src/api/email/init.rs
@@ -111,6 +111,10 @@ impl InitError {
impl IntoResponse for InitError {
fn into_response(self) -> Response {
let status_code = self.status_code();
+ let headers = match &self {
+ InitError::ProviderError(error) => super::provider_error::provider_error_headers(error),
+ _ => axum::http::HeaderMap::new(),
+ };
match self {
InitError::SharedInboxConflict {
@@ -142,7 +146,7 @@ impl IntoResponse for InitError {
}),
)
.into_response(),
- other => (status_code, other.to_string()).into_response(),
+ other => (status_code, headers, other.to_string()).into_response(),
}
}
}
diff --git a/services/email_service/src/api/email/init/test.rs b/services/email_service/src/api/email/init/test.rs
index 4d52361ea1..d16e2078c7 100644
--- a/services/email_service/src/api/email/init/test.rs
+++ b/services/email_service/src/api/email/init/test.rs
@@ -99,11 +99,29 @@ fn unusable_provider_grants_classify_as_no_gmail_grant() {
fn rate_limiting_classifies_as_http_429() {
let error = classify_provider_init_error(EmailApiError::RateLimited {
retry_after: None,
- origin: email_api_client::domain::models::RateLimitOrigin::Provider,
+ origin: email_api_client::domain::models::RateLimitOrigin::Provider { concurrency: false },
});
assert_eq!(error.status_code(), StatusCode::TOO_MANY_REQUESTS);
}
+#[test]
+fn a_rate_limited_init_tells_the_client_when_to_retry() {
+ let response = InitError::ProviderError(EmailApiError::RateLimited {
+ retry_after: Some(std::time::Duration::from_millis(1_200)),
+ origin: email_api_client::domain::models::RateLimitOrigin::Local,
+ })
+ .into_response();
+
+ assert_eq!(response.status(), StatusCode::TOO_MANY_REQUESTS);
+ assert_eq!(
+ response
+ .headers()
+ .get(axum::http::header::RETRY_AFTER)
+ .and_then(|value| value.to_str().ok()),
+ Some("2")
+ );
+}
+
#[tokio::test]
async fn provider_diagnostics_are_not_returned_to_clients() {
let response = InitError::ProviderError(EmailApiError::Permanent {
diff --git a/services/email_service/src/api/email/provider_error.rs b/services/email_service/src/api/email/provider_error.rs
index 4cbdb6b2f5..8c756b6d39 100644
--- a/services/email_service/src/api/email/provider_error.rs
+++ b/services/email_service/src/api/email/provider_error.rs
@@ -5,15 +5,16 @@ use email_api_client::domain::models::EmailApiError;
#[cfg(test)]
mod test;
-/// Response headers implied by a provider error: rate-limited requests carry
-/// the provider's suggested delay as a `Retry-After` header when known.
+/// Response headers implied by a rate limit: its suggested delay, when known,
+/// as a `Retry-After` header rounded up to whole seconds, so a sub-second wait
+/// is not sent as `0`.
pub(crate) fn provider_error_headers(error: &EmailApiError) -> HeaderMap {
let mut headers = HeaderMap::new();
if let EmailApiError::RateLimited {
retry_after: Some(delay),
..
} = error
- && let Ok(value) = HeaderValue::from_str(&delay.as_secs().to_string())
+ && let Ok(value) = HeaderValue::from_str(&delay.as_millis().div_ceil(1_000).to_string())
{
headers.insert(RETRY_AFTER, value);
}
diff --git a/services/email_service/src/api/email/provider_error/test.rs b/services/email_service/src/api/email/provider_error/test.rs
index eef819cfd6..a9a3a9661e 100644
--- a/services/email_service/src/api/email/provider_error/test.rs
+++ b/services/email_service/src/api/email/provider_error/test.rs
@@ -10,7 +10,7 @@ fn maps_provider_errors_to_api_statuses() {
(
EmailApiError::RateLimited {
retry_after: Some(Duration::from_secs(10)),
- origin: RateLimitOrigin::Provider,
+ origin: RateLimitOrigin::Provider { concurrency: false },
},
StatusCode::TOO_MANY_REQUESTS,
),
@@ -55,17 +55,27 @@ fn maps_outdated_cursor_to_conflict() {
fn rate_limited_responses_carry_a_retry_after_header_when_known() {
let headers = provider_error_headers(&EmailApiError::RateLimited {
retry_after: Some(Duration::from_secs(42)),
- origin: RateLimitOrigin::Provider,
+ origin: RateLimitOrigin::Provider { concurrency: false },
});
assert_eq!(
headers.get(RETRY_AFTER).and_then(|v| v.to_str().ok()),
Some("42")
);
+ let headers = provider_error_headers(&EmailApiError::RateLimited {
+ retry_after: Some(Duration::from_millis(950)),
+ origin: RateLimitOrigin::Local,
+ });
+ assert_eq!(
+ headers.get(RETRY_AFTER).and_then(|v| v.to_str().ok()),
+ Some("1"),
+ "a sub-second wait must not tell clients to retry immediately"
+ );
+
assert!(
provider_error_headers(&EmailApiError::RateLimited {
retry_after: None,
- origin: RateLimitOrigin::Provider,
+ origin: RateLimitOrigin::Provider { concurrency: false },
})
.is_empty()
);
diff --git a/services/email_service/src/bin/pubsub_workers/pubsub_workers.rs b/services/email_service/src/bin/pubsub_workers/pubsub_workers.rs
index 4499d79e24..f73e0d865a 100644
--- a/services/email_service/src/bin/pubsub_workers/pubsub_workers.rs
+++ b/services/email_service/src/bin/pubsub_workers/pubsub_workers.rs
@@ -4,7 +4,7 @@ use document_storage_service_client::DocumentStorageServiceClient;
use email_api_client::GmailApiClientRepository;
use email_service::config::Config;
use email_service::outbound::email_api::{
- EmailServiceTokenSource, GmailApi, RateBudget, RedisProviderRateLimiter,
+ EmailServiceTokenSource, GmailApi, RateBudget, RedisGmailRequestGate,
};
use email_service::pubsub::CrmMetadataResolver;
use email_service::util::redis::RedisClient;
@@ -30,15 +30,15 @@ fn compose_email_api(
db: PgPool,
subscription_topic: String,
auth_service_client: authentication_service_client::AuthServiceClient,
- redis_client: RedisClient,
redis_conn: redis::aio::MultiplexedConnection,
+ gate_conn: redis::aio::ConnectionManager,
sqs_client: sqs_client::SQS,
rate_budget: RateBudget,
) -> GmailApi {
GmailApi::new(
GmailApiClientRepository::from_subscription_topic(subscription_topic),
EmailServiceTokenSource::new(db, redis_conn, auth_service_client, sqs_client),
- RedisProviderRateLimiter::new(redis_client, rate_budget),
+ RedisGmailRequestGate::new(gate_conn, rate_budget),
)
}
@@ -268,12 +268,7 @@ async fn main() -> anyhow::Result<()> {
queue: ingress_queue,
});
- let redis_client = RedisClient::new(
- redis_inner_client,
- config.redis_rate_limit_reqs,
- config.redis_rate_limit_reqs_backfill,
- config.redis_rate_limit_window_secs,
- );
+ let redis_client = RedisClient::new(redis_inner_client, config.redis_rate_limit_window_secs);
// One long-lived multiplexed connection shared by both token sources;
// cloning it is an Arc bump, not a new TCP dial per provider call.
@@ -282,13 +277,18 @@ async fn main() -> anyhow::Result<()> {
.get_multiplexed_async_connection()
.await
.context("failed to get multiplexed redis connection for token sources")?;
+ let gate_conn = redis_client
+ .inner
+ .get_connection_manager()
+ .await
+ .context("failed to get reconnecting redis connection for the gmail request gates")?;
let email_api_live = compose_email_api(
db.clone(),
config.gmail_gcp_queue.to_string(),
auth_service_client.clone(),
- redis_client.clone(),
redis_conn.clone(),
+ gate_conn.clone(),
sqs_client.clone(),
RateBudget::Live,
);
@@ -321,8 +321,8 @@ async fn main() -> anyhow::Result<()> {
db_backfill.clone(),
config.gmail_gcp_queue.to_string(),
auth_service_client.clone(),
- redis_client.clone(),
redis_conn,
+ gate_conn,
sqs_client.clone(),
RateBudget::Backfill,
);
diff --git a/services/email_service/src/config.rs b/services/email_service/src/config.rs
index 072c198d47..27bf31e0eb 100644
--- a/services/email_service/src/config.rs
+++ b/services/email_service/src/config.rs
@@ -125,16 +125,7 @@ pub struct Config {
#[macro_config_default(3)]
pub sfs_uploader_workers: i32,
- /// The number of requests we allow per window for backfilling. Less than redis_rate_limit_reqs
- /// so we have room for normal gmail operations while backfilling is occurring
- #[macro_config_default(13000)]
- pub redis_rate_limit_reqs_backfill: u32,
-
- /// The number of requests we allow per window.
- #[macro_config_default(14000)]
- pub redis_rate_limit_reqs: u32,
-
- /// The size of the sliding window we use for rate limit.
+ /// The size of the sliding window for the Google Calendar rate limit.
#[macro_config_default(60)]
pub redis_rate_limit_window_secs: u32,
diff --git a/services/email_service/src/outbound/email_api/gmail_gate.rs b/services/email_service/src/outbound/email_api/gmail_gate.rs
new file mode 100644
index 0000000000..fcd9562419
--- /dev/null
+++ b/services/email_service/src/outbound/email_api/gmail_gate.rs
@@ -0,0 +1,445 @@
+//! Redis admission shared by every Gmail request for a mailbox.
+
+#[cfg(test)]
+mod test;
+
+use std::time::Duration;
+
+use email_api_client::domain::models::{
+ ApiOperationKind, EmailApiError, MailboxLoad, RateLimitOrigin, RateLimitRefusal, RequestPermit,
+};
+use email_api_client::domain::ports::ProviderRequestGate;
+use models_email::gmail::operations::GmailApiOperation;
+use redis::aio::ConnectionManager;
+use uuid::Uuid;
+
+// Gmail answers 429 "Too many concurrent requests for user" above an
+// unpublished per-user limit that every client of the mailbox shares; it was
+// 91% of our Gmail rate limits. Backfill leaves two of the mailbox's slots to
+// sync and user writes.
+const CONCURRENT_REQUESTS: u32 = 5;
+const LIVE_RESERVED_REQUESTS: u32 = 2;
+// Gmail's real limit may be lower, and other clients of the mailbox use it
+// too. After that 429, the mailbox's cap drops to three quarters, at most once
+// a second so requests already in flight don't count twice, then rises by one
+// a minute back to five.
+const LEARNED_CAP_HOLD_MS: u32 = 1_000;
+const LEARNED_CAP_INCREASE_MS: u32 = 60_000;
+// Gmail allows 250 quota units per user per second (moving average) and, under
+// this project's grandfathered quota, 15,000 per minute. A ten-second window
+// enforces both: 2,300 units averages 230 a second and caps a minute at 13,800.
+const WINDOW_SECONDS: u32 = 10;
+const UNITS_PER_WINDOW: u32 = 2_300;
+const BACKFILL_UNITS_PER_WINDOW: u32 = 1_800;
+// Longer than the Gmail client's 60-second request timeout. Finished and
+// dropped permits release themselves; the lease frees the slots of a crashed
+// process.
+const PERMIT_MS: u32 = 90_000;
+// Google asks clients to wait at least a second before retrying a 429.
+const BUSY_RETRY_MS: u32 = 1_000;
+// A refused backfill request defers its SQS message, and deferred work comes
+// back in waves that leave backfill's slots idle. Up to twelve refused backfill
+// requests per mailbox, about a second of work for its three slots, wait for
+// room instead, so a freed slot is taken at once. The rest still defer, which
+// keeps one mailbox from tying up the workers other mailboxes need.
+const BACKFILL_WAITERS: u32 = 12;
+// Well inside the backfill queue's 60-second visibility timeout.
+const BACKFILL_WAIT: Duration = Duration::from_secs(20);
+// How often a waiting request checks for a free slot. Reads take about 200 ms,
+// and with several waiters polling a freed slot idles a few milliseconds.
+const WAIT_POLL_MS: u32 = 50;
+// A waiter's place outlives its sleep by this much, so a crashed waiter frees it.
+const WAITER_LEASE_MARGIN_MS: u32 = 1_000;
+// Gmail sends no Retry-After with its rate limits. The concurrency and
+// per-second limits behind 97% of them clear within Google's one-second
+// minimum, and a longer pause stalls every caller for the mailbox.
+const DEFAULT_COOLDOWN: Duration = Duration::from_secs(1);
+// Backfill stays out longer after Gmail rate-limits the mailbox, so sync and
+// user writes get the room Gmail has left before backfill fills it again.
+const BACKFILL_COOLDOWN: Duration = Duration::from_secs(10);
+// Per-second and per-minute quotas recover within a minute. Longer Retry-After
+// values come from daily sending or bandwidth limits, which must not pause the
+// mailbox's other work for hours.
+const MAX_COOLDOWN: Duration = Duration::from_secs(60);
+const UNAVAILABLE_RETRY_AFTER: Duration = Duration::from_secs(5);
+
+// Returns {delay, waiting, in_flight, cap}: {0, 0, in_flight, cap} admits the
+// request, reporting the mailbox's requests in flight counting this one and the
+// cap it was admitted under; {delay, 1, 0, 0} keeps it waiting for room and
+// {delay, 0, 0, 0} refuses it.
+const ACQUIRE: &str = r#"
+local clock = redis.call('TIME')
+local now = tonumber(clock[1]) * 1000 + math.floor(tonumber(clock[2]) / 1000)
+local waiters = tonumber(ARGV[8])
+local function refuse(delay, busy)
+ local wait = delay
+ if busy then wait = tonumber(ARGV[9]) end
+ if waiters > 0 then
+ redis.call('ZREMRANGEBYSCORE', KEYS[4], '-inf', now)
+ if wait <= tonumber(ARGV[10]) and
+ (redis.call('ZSCORE', KEYS[4], ARGV[1]) or redis.call('ZCARD', KEYS[4]) < waiters) then
+ local lease = wait + tonumber(ARGV[11])
+ redis.call('ZADD', KEYS[4], now + lease, ARGV[1])
+ if redis.call('PTTL', KEYS[4]) < lease then redis.call('PEXPIRE', KEYS[4], lease) end
+ return {wait, 1, 0, 0}
+ end
+ redis.call('ZREM', KEYS[4], ARGV[1])
+ end
+ return {delay, 0, 0, 0}
+end
+local cooldown = redis.call('PTTL', KEYS[3])
+if ARGV[12] == '1' then cooldown = math.max(cooldown, redis.call('PTTL', KEYS[5])) end
+if cooldown > 0 then return refuse(cooldown, false) end
+local cap = tonumber(ARGV[2])
+local learned = redis.call('HMGET', KEYS[6], 'cap', 'at')
+if learned[1] then
+ cap = math.min(cap, tonumber(learned[1]) + math.floor((now - tonumber(learned[2])) / tonumber(ARGV[13])))
+end
+redis.call('ZREMRANGEBYSCORE', KEYS[1], '-inf', now)
+local in_flight = redis.call('ZCARD', KEYS[1])
+if in_flight >= cap - tonumber(ARGV[14]) then return refuse(tonumber(ARGV[7]), true) end
+local second = tonumber(clock[1])
+local window = tonumber(ARGV[5])
+local limit = tonumber(ARGV[3])
+local cost = tonumber(ARGV[4])
+local buckets = redis.call('HGETALL', KEYS[2])
+local live = {}
+local used = 0
+for i = 1, #buckets, 2 do
+ local bucket = tonumber(buckets[i])
+ if bucket <= second - window then
+ redis.call('HDEL', KEYS[2], buckets[i])
+ else
+ local units = tonumber(buckets[i + 1])
+ used = used + units
+ table.insert(live, {bucket, units})
+ end
+end
+if used + cost > limit then
+ table.sort(live, function(a, b) return a[1] < b[1] end)
+ for _, entry in ipairs(live) do
+ used = used - entry[2]
+ if used + cost <= limit then
+ return refuse(math.max(1, (entry[1] + window) * 1000 - now), false)
+ end
+ end
+ return refuse(window * 1000, false)
+end
+redis.call('ZADD', KEYS[1], now + tonumber(ARGV[6]), ARGV[1])
+redis.call('PEXPIRE', KEYS[1], ARGV[6])
+redis.call('HINCRBY', KEYS[2], clock[1], cost)
+redis.call('EXPIRE', KEYS[2], window + 1)
+if waiters > 0 then redis.call('ZREM', KEYS[4], ARGV[1]) end
+return {0, 0, in_flight + 1, cap}
+"#;
+
+const FINISH: &str = r#"
+redis.call('ZREM', KEYS[1], ARGV[1])
+for i = 2, 3 do
+ local delay = tonumber(ARGV[i])
+ if delay > 0 and redis.call('PTTL', KEYS[i]) < delay then
+ redis.call('SET', KEYS[i], '1', 'PX', delay)
+ end
+end
+if ARGV[4] == '1' then
+ local clock = redis.call('TIME')
+ local now = tonumber(clock[1]) * 1000 + math.floor(tonumber(clock[2]) / 1000)
+ local default = tonumber(ARGV[5])
+ local increase = tonumber(ARGV[7])
+ local learned = redis.call('HMGET', KEYS[4], 'cap', 'at')
+ if not learned[2] or now - tonumber(learned[2]) >= tonumber(ARGV[6]) then
+ local cap = default
+ if learned[1] then
+ cap = math.min(cap, tonumber(learned[1]) + math.floor((now - tonumber(learned[2])) / increase))
+ end
+ cap = math.max(1, math.floor(cap * 3 / 4))
+ redis.call('HSET', KEYS[4], 'cap', cap, 'at', now)
+ redis.call('PEXPIRE', KEYS[4], (default - cap) * increase)
+ end
+end
+return 1
+"#;
+
+/// Selects which share of a mailbox's Gmail allowance a service instance uses.
+///
+/// `Live` and `Backfill` are two thresholds over one shared Redis state per
+/// mailbox, not two allowances: every request occupies the same in-flight set
+/// and charges the same window, and the budget only decides the limit it is
+/// compared against. Splitting the keys per budget would let live and backfill
+/// together exceed Gmail's per-user limits. Don't "fix" the key sharing.
+#[derive(Debug, Clone, Copy, PartialEq, Eq)]
+pub enum RateBudget {
+ /// Interactive, incremental, and user-initiated work, which may use the
+ /// mailbox's whole allowance.
+ Live,
+ /// Mailbox backfills, held below the full allowance so live work keeps
+ /// room while a backfill runs.
+ Backfill,
+}
+
+impl RateBudget {
+ /// The in-flight slots left to others, and the quota units per window.
+ fn limits(self) -> (u32, u32) {
+ match self {
+ Self::Live => (0, UNITS_PER_WINDOW),
+ Self::Backfill => (LIVE_RESERVED_REQUESTS, BACKFILL_UNITS_PER_WINDOW),
+ }
+ }
+
+ /// Live callers handle their own retries, so only backfill waits.
+ fn wait(self) -> Duration {
+ match self {
+ Self::Live => Duration::ZERO,
+ Self::Backfill => BACKFILL_WAIT,
+ }
+ }
+}
+
+#[derive(Debug, PartialEq, Eq)]
+enum Admission {
+ Admitted(MailboxLoad),
+ Wait(Duration),
+ Refused(Duration),
+}
+
+/// Caps in-flight Gmail requests and quota units per mailbox, and pauses the
+/// mailbox after Gmail rate-limits it.
+///
+/// It fails closed: when Redis cannot decide, the request is refused as a
+/// local rate limit, so callers defer or requeue the work rather than send it
+/// unmetered. Because of that it needs a reconnecting connection; a dropped
+/// multiplexed connection would stop all Gmail work until the process restarts.
+#[derive(Clone)]
+pub struct RedisGmailRequestGate {
+ connection: ConnectionManager,
+ budget: RateBudget,
+ wait: Duration,
+}
+
+impl RedisGmailRequestGate {
+ /// Creates a gate that admits requests within the selected budget.
+ pub fn new(connection: ConnectionManager, budget: RateBudget) -> Self {
+ Self {
+ connection,
+ budget,
+ wait: budget.wait(),
+ }
+ }
+}
+
+fn keys(link_id: Uuid) -> [String; 6] {
+ // Hash tags colocate the Lua keys in Redis Cluster.
+ let scope = format!("gmail:{{{link_id}}}");
+ [
+ format!("{scope}:inflight"),
+ format!("{scope}:window"),
+ format!("{scope}:cooldown"),
+ format!("{scope}:waiting"),
+ format!("{scope}:backfill_cooldown"),
+ format!("{scope}:limit"),
+ ]
+}
+
+/// What a finished request tells the gate about the mailbox.
+#[derive(Default)]
+struct Outcome {
+ /// Pause for every caller.
+ cooldown: Duration,
+ /// Pause for backfill alone.
+ backfill_cooldown: Duration,
+ /// Gmail refused it for too many requests in flight.
+ concurrency_limited: bool,
+}
+
+fn outcome(error: Option<&EmailApiError>) -> Outcome {
+ match error {
+ Some(EmailApiError::RateLimited {
+ origin: RateLimitOrigin::Provider { concurrency },
+ retry_after,
+ }) => {
+ let cooldown = retry_after.unwrap_or(DEFAULT_COOLDOWN).min(MAX_COOLDOWN);
+ Outcome {
+ cooldown,
+ backfill_cooldown: cooldown.max(BACKFILL_COOLDOWN),
+ concurrency_limited: *concurrency,
+ }
+ }
+ _ => Outcome::default(),
+ }
+}
+
+impl ProviderRequestGate for RedisGmailRequestGate {
+ async fn acquire(
+ &self,
+ link_id: Uuid,
+ operation: ApiOperationKind,
+ ) -> Result<RequestPermit, RateLimitRefusal> {
+ let permit = Uuid::now_v7();
+ let cost = gmail_operation(operation).cost();
+ let admission = match self.admit(link_id, permit, cost, self.wait).await {
+ Ok(Admission::Wait(delay)) => self.wait_for_room(link_id, permit, cost, delay).await,
+ admission => admission,
+ };
+ match admission {
+ Ok(Admission::Admitted(load)) => {
+ let gate = self.clone();
+ Ok(RequestPermit::new(link_id, permit, move |link_id, permit| {
+ // Drop cannot wait. Without a runtime, the lease frees the slot.
+ if let Ok(runtime) = tokio::runtime::Handle::try_current() {
+ runtime.spawn(async move {
+ gate.release(link_id, permit, Outcome::default()).await
+ });
+ }
+ })
+ .with_load(load))
+ }
+ Ok(Admission::Wait(delay) | Admission::Refused(delay)) => {
+ Err(RateLimitRefusal::new(Some(delay)))
+ }
+ Err(error) => {
+ tracing::warn!(%link_id, %error, "Gmail request admission is unavailable");
+ Err(RateLimitRefusal::new(Some(UNAVAILABLE_RETRY_AFTER)))
+ }
+ }
+ }
+
+ async fn finish(&self, permit: RequestPermit, error: Option<&EmailApiError>) {
+ self.release(permit.link_id(), permit.id(), outcome(error))
+ .await;
+ permit.disarm();
+ }
+}
+
+impl RedisGmailRequestGate {
+ /// Runs one admission check. `wait` is how much longer the request may wait for room.
+ async fn admit(
+ &self,
+ link_id: Uuid,
+ permit: Uuid,
+ cost: u32,
+ wait: Duration,
+ ) -> redis::RedisResult<Admission> {
+ let [
+ inflight,
+ window,
+ cooldown,
+ waiting,
+ backfill_cooldown,
+ limit,
+ ] = keys(link_id);
+ let (reserved_requests, units_per_window) = self.budget.limits();
+ let waiters = if self.wait.is_zero() {
+ 0
+ } else {
+ BACKFILL_WAITERS
+ };
+ let (delay, waits, in_flight, cap): (u64, u8, u32, u32) = redis::Script::new(ACQUIRE)
+ .key(inflight)
+ .key(window)
+ .key(cooldown)
+ .key(waiting)
+ .key(backfill_cooldown)
+ .key(limit)
+ .arg(permit.to_string())
+ .arg(CONCURRENT_REQUESTS)
+ .arg(units_per_window)
+ .arg(cost)
+ .arg(WINDOW_SECONDS)
+ .arg(PERMIT_MS)
+ .arg(BUSY_RETRY_MS)
+ .arg(waiters)
+ .arg(WAIT_POLL_MS)
+ .arg(wait.as_millis().to_string())
+ .arg(WAITER_LEASE_MARGIN_MS)
+ .arg(u8::from(self.budget == RateBudget::Backfill))
+ .arg(LEARNED_CAP_INCREASE_MS)
+ .arg(reserved_requests)
+ .invoke_async(&mut self.connection.clone())
+ .await?;
+ let delay = Duration::from_millis(delay);
+ Ok(match (delay.is_zero(), waits == 1) {
+ (true, _) => Admission::Admitted(MailboxLoad { in_flight, cap }),
+ (false, true) => Admission::Wait(delay),
+ (false, false) => Admission::Refused(delay),
+ })
+ }
+
+ /// Holds a refused request until it is admitted or its wait runs out.
+ #[tracing::instrument(
+ name = "gmail.request_gate.wait",
+ skip(self, cost),
+ fields(admitted),
+ err
+ )]
+ async fn wait_for_room(
+ &self,
+ link_id: Uuid,
+ permit: Uuid,
+ cost: u32,
+ mut delay: Duration,
+ ) -> redis::RedisResult<Admission> {
+ let deadline = tokio::time::Instant::now() + self.wait;
+ loop {
+ tokio::time::sleep(delay).await;
+ let wait = deadline.saturating_duration_since(tokio::time::Instant::now());
+ match self.admit(link_id, permit, cost, wait).await? {
+ Admission::Wait(next) => delay = next,
+ admission => {
+ tracing::Span::current()
+ .record("admitted", matches!(admission, Admission::Admitted(_)));
+ return Ok(admission);
+ }
+ }
+ }
+ }
+
+ async fn release(&self, link_id: Uuid, permit: Uuid, outcome: Outcome) {
+ let [inflight, _, cooldown_key, _, backfill_cooldown_key, limit] = keys(link_id);
+ let result: redis::RedisResult<i32> = redis::Script::new(FINISH)
+ .key(inflight)
+ .key(cooldown_key)
+ .key(backfill_cooldown_key)
+ .key(limit)
+ .arg(permit.to_string())
+ .arg(outcome.cooldown.as_millis().to_string())
+ .arg(outcome.backfill_cooldown.as_millis().to_string())
+ .arg(u8::from(outcome.concurrency_limited))
+ .arg(CONCURRENT_REQUESTS)
+ .arg(LEARNED_CAP_HOLD_MS)
+ .arg(LEARNED_CAP_INCREASE_MS)
+ .invoke_async(&mut self.connection.clone())
+ .await;
+ if result.is_err() {
+ // The provider call already happened and its outcome must reach the
+ // caller even when releasing admission fails. The lease expires.
+ tracing::warn!(%link_id, "failed to finish Gmail request admission");
+ }
+ }
+}
+
+fn gmail_operation(operation: ApiOperationKind) -> GmailApiOperation {
+ match operation {
+ ApiOperationKind::GetProfile => GmailApiOperation::UsersGetProfile,
+ ApiOperationKind::ListChanges => GmailApiOperation::HistoryList,
+ ApiOperationKind::ListLabels => GmailApiOperation::LabelsList,
+ ApiOperationKind::CreateLabel => GmailApiOperation::LabelsCreate,
+ ApiOperationKind::DeleteLabel => GmailApiOperation::LabelsDelete,
+ ApiOperationKind::GetMessage => GmailApiOperation::MessagesGet,
+ ApiOperationKind::ListMessages => GmailApiOperation::MessagesList,
+ ApiOperationKind::ModifyMessageLabels => GmailApiOperation::MessagesModify,
+ ApiOperationKind::GetAttachment => GmailApiOperation::MessagesAttachmentsGet,
+ ApiOperationKind::SendMessage => GmailApiOperation::MessagesSend,
+ ApiOperationKind::GetThread => GmailApiOperation::ThreadsGet,
+ ApiOperationKind::ListThreads => GmailApiOperation::ThreadsList,
+ // People API calls share this budget. Gmail has no corresponding quota
+ // operation, so use a one-unit operation as the accounting proxy.
+ ApiOperationKind::ListContacts => GmailApiOperation::UsersGetProfile,
+ // Block and unblock are composite filter operations. One admission
+ // charges the dominant write operation rather than every request.
+ ApiOperationKind::BlockSender => GmailApiOperation::SettingsFiltersCreate,
+ ApiOperationKind::UnblockSender => GmailApiOperation::SettingsFiltersDelete,
+ ApiOperationKind::ListBlockedSenders => GmailApiOperation::SettingsFiltersList,
+ ApiOperationKind::Subscribe => GmailApiOperation::Watch,
+ ApiOperationKind::Unsubscribe => GmailApiOperation::Stop,
+ }
+}
diff --git a/services/email_service/src/outbound/email_api/gmail_gate/test.rs b/services/email_service/src/outbound/email_api/gmail_gate/test.rs
new file mode 100644
index 0000000000..72e3fca17c
--- /dev/null
+++ b/services/email_service/src/outbound/email_api/gmail_gate/test.rs
@@ -0,0 +1,552 @@
+use super::*;
+
+async fn connection() -> ConnectionManager {
+ redis::Client::open("redis://127.0.0.1:6379/")
+ .unwrap()
+ .get_connection_manager()
+ .await
+ .expect("start local Redis before running this integration test")
+}
+
+async fn cleanup(connection: &ConnectionManager, link_id: Uuid) {
+ let _: i32 = redis::cmd("DEL")
+ .arg(&keys(link_id))
+ .query_async(&mut connection.clone())
+ .await
+ .unwrap();
+}
+
+fn refusal_delay(result: Result<RequestPermit, RateLimitRefusal>) -> Duration {
+ result
+ .err()
+ .expect("the request should be refused")
+ .retry_after
+ .expect("a refusal reports when to retry")
+}
+
+#[tokio::test]
+#[ignore = "requires local Redis on port 6379; uses only unique mailbox keys"]
+async fn backfill_leaves_two_of_five_in_flight_slots_to_live_work() {
+ let connection = connection().await;
+ let live = RedisGmailRequestGate::new(connection.clone(), RateBudget::Live);
+ let mut backfill = RedisGmailRequestGate::new(connection.clone(), RateBudget::Backfill);
+ backfill.wait = Duration::ZERO;
+ let link_id = Uuid::now_v7();
+
+ let mut backfill_permits = Vec::new();
+ for _ in 0..3 {
+ backfill_permits.push(
+ backfill
+ .acquire(link_id, ApiOperationKind::GetMessage)
+ .await
+ .unwrap(),
+ );
+ }
+ assert_eq!(
+ refusal_delay(
+ backfill
+ .acquire(link_id, ApiOperationKind::GetMessage)
+ .await
+ ),
+ Duration::from_secs(1)
+ );
+ let first_live = live
+ .acquire(link_id, ApiOperationKind::ModifyMessageLabels)
+ .await
+ .unwrap();
+ let _second_live = live
+ .acquire(link_id, ApiOperationKind::ModifyMessageLabels)
+ .await
+ .unwrap();
+ assert_eq!(
+ refusal_delay(
+ live.acquire(link_id, ApiOperationKind::ModifyMessageLabels)
+ .await
+ ),
+ Duration::from_secs(1)
+ );
+
+ live.finish(first_live, None).await;
+ let _third_live = live
+ .acquire(link_id, ApiOperationKind::ModifyMessageLabels)
+ .await
+ .unwrap();
+ backfill.finish(backfill_permits.remove(0), None).await;
+ assert!(
+ backfill
+ .acquire(link_id, ApiOperationKind::GetMessage)
+ .await
+ .is_err(),
+ "a released backfill slot must not let backfill exceed its share while live holds the rest"
+ );
+ cleanup(&connection, link_id).await;
+}
+
+#[tokio::test]
+#[ignore = "requires local Redis on port 6379; uses only unique mailbox keys"]
+async fn twelve_refused_backfill_requests_wait_for_room_and_the_rest_defer() {
+ let connection = connection().await;
+ let backfill = RedisGmailRequestGate::new(connection.clone(), RateBudget::Backfill);
+ let link_id = Uuid::now_v7();
+ let mut permits = Vec::new();
+ for _ in 0..3 {
+ permits.push(
+ backfill
+ .acquire(link_id, ApiOperationKind::GetMessage)
+ .await
+ .unwrap(),
+ );
+ }
+
+ let waiters: Vec<Uuid> = (0..12).map(|_| Uuid::now_v7()).collect();
+ for &waiter in &waiters {
+ assert_eq!(
+ backfill
+ .admit(link_id, waiter, 5, BACKFILL_WAIT)
+ .await
+ .unwrap(),
+ Admission::Wait(Duration::from_millis(50))
+ );
+ }
+ let deferred = Uuid::now_v7();
+ assert_eq!(
+ backfill
+ .admit(link_id, deferred, 5, BACKFILL_WAIT)
+ .await
+ .unwrap(),
+ Admission::Refused(Duration::from_secs(1))
+ );
+
+ backfill.finish(permits.remove(0), None).await;
+ assert_eq!(
+ backfill
+ .wait_for_room(link_id, waiters[0], 5, Duration::from_millis(50))
+ .await
+ .unwrap(),
+ Admission::Admitted(MailboxLoad {
+ in_flight: 3,
+ cap: 5
+ })
+ );
+ // The admitted waiter gave up its place.
+ assert_eq!(
+ backfill
+ .admit(link_id, deferred, 5, BACKFILL_WAIT)
+ .await
+ .unwrap(),
+ Admission::Wait(Duration::from_millis(50))
+ );
+ cleanup(&connection, link_id).await;
+}
+
+#[tokio::test]
+#[ignore = "requires local Redis on port 6379; uses only unique mailbox keys"]
+async fn a_backfill_request_takes_a_slot_freed_while_it_waits() {
+ let connection = connection().await;
+ let backfill = RedisGmailRequestGate::new(connection.clone(), RateBudget::Backfill);
+ let link_id = Uuid::now_v7();
+ let mut permits = Vec::new();
+ for _ in 0..3 {
+ permits.push(
+ backfill
+ .acquire(link_id, ApiOperationKind::GetMessage)
+ .await
+ .unwrap(),
+ );
+ }
+
+ let mut waiting = tokio::spawn({
+ let backfill = backfill.clone();
+ async move {
+ backfill
+ .acquire(link_id, ApiOperationKind::GetMessage)
+ .await
+ }
+ });
+ assert!(
+ tokio::time::timeout(Duration::from_millis(300), &mut waiting)
+ .await
+ .is_err(),
+ "a full mailbox holds the backfill request instead of refusing it"
+ );
+ backfill.finish(permits.remove(0), None).await;
+ let admitted = tokio::time::timeout(Duration::from_secs(5), waiting)
+ .await
+ .unwrap()
+ .unwrap();
+ assert!(admitted.is_ok());
+ cleanup(&connection, link_id).await;
+}
+
+#[tokio::test]
+#[ignore = "requires local Redis on port 6379; uses only unique mailbox keys"]
+async fn a_backfill_request_defers_once_its_wait_runs_out() {
+ let connection = connection().await;
+ let mut backfill = RedisGmailRequestGate::new(connection.clone(), RateBudget::Backfill);
+ backfill.wait = Duration::from_millis(200);
+ let link_id = Uuid::now_v7();
+ let mut permits = Vec::new();
+ for _ in 0..3 {
+ permits.push(
+ backfill
+ .acquire(link_id, ApiOperationKind::GetMessage)
+ .await
+ .unwrap(),
+ );
+ }
+
+ assert_eq!(
+ refusal_delay(
+ backfill
+ .acquire(link_id, ApiOperationKind::GetMessage)
+ .await
+ ),
+ Duration::from_secs(1)
+ );
+ let waiting: u32 = redis::cmd("ZCARD")
+ .arg(&keys(link_id)[3])
+ .query_async(&mut connection.clone())
+ .await
+ .unwrap();
+ assert_eq!(waiting, 0, "a request that stops waiting leaves its place");
+ cleanup(&connection, link_id).await;
+}
+
+#[tokio::test]
+#[ignore = "requires local Redis on port 6379; uses only unique mailbox keys"]
+async fn a_gmail_rate_limit_pauses_live_work_briefly_and_backfill_longer() {
+ let connection = connection().await;
+ let live = RedisGmailRequestGate::new(connection.clone(), RateBudget::Live);
+ let mut backfill = RedisGmailRequestGate::new(connection.clone(), RateBudget::Backfill);
+ backfill.wait = Duration::ZERO;
+ let link_id = Uuid::now_v7();
+
+ let throttled = live
+ .acquire(link_id, ApiOperationKind::GetMessage)
+ .await
+ .unwrap();
+ let local_refusal = live
+ .acquire(link_id, ApiOperationKind::GetMessage)
+ .await
+ .unwrap();
+ live.finish(
+ local_refusal,
+ Some(&EmailApiError::RateLimited {
+ retry_after: None,
+ origin: RateLimitOrigin::Local,
+ }),
+ )
+ .await;
+ let _admitted = live
+ .acquire(link_id, ApiOperationKind::GetMessage)
+ .await
+ .unwrap();
+
+ live.finish(
+ throttled,
+ Some(&EmailApiError::RateLimited {
+ retry_after: None,
+ origin: RateLimitOrigin::Provider { concurrency: false },
+ }),
+ )
+ .await;
+ let live_delay = refusal_delay(live.acquire(link_id, ApiOperationKind::SendMessage).await);
+ assert!(live_delay > Duration::from_millis(500) && live_delay <= DEFAULT_COOLDOWN);
+ let backfill_delay =
+ refusal_delay(backfill.acquire(link_id, ApiOperationKind::GetThread).await);
+ assert!(
+ backfill_delay > Duration::from_secs(9) && backfill_delay <= BACKFILL_COOLDOWN,
+ "backfill leaves the room Gmail has left to live work for longer"
+ );
+ cleanup(&connection, link_id).await;
+
+ let permit = live
+ .acquire(link_id, ApiOperationKind::SendMessage)
+ .await
+ .unwrap();
+ live.finish(
+ permit,
+ Some(&EmailApiError::RateLimited {
+ retry_after: Some(Duration::from_secs(3_600)),
+ origin: RateLimitOrigin::Provider { concurrency: false },
+ }),
+ )
+ .await;
+ let delay = refusal_delay(live.acquire(link_id, ApiOperationKind::GetMessage).await);
+ assert!(delay > Duration::from_secs(59) && delay <= MAX_COOLDOWN);
+ let delay = refusal_delay(
+ backfill
+ .acquire(link_id, ApiOperationKind::GetMessage)
+ .await,
+ );
+ assert!(delay > Duration::from_secs(59) && delay <= MAX_COOLDOWN);
+ cleanup(&connection, link_id).await;
+}
+
+#[tokio::test]
+#[ignore = "requires local Redis on port 6379; uses only unique mailbox keys"]
+async fn a_concurrency_429_lowers_the_mailbox_cap_until_it_recovers() {
+ let connection = connection().await;
+ let live = RedisGmailRequestGate::new(connection.clone(), RateBudget::Live);
+ let mut backfill = RedisGmailRequestGate::new(connection.clone(), RateBudget::Backfill);
+ backfill.wait = Duration::ZERO;
+ let link_id = Uuid::now_v7();
+ let [_, _, cooldown, _, backfill_cooldown, limit] = keys(link_id);
+ let concurrency_429 = EmailApiError::RateLimited {
+ retry_after: None,
+ origin: RateLimitOrigin::Provider { concurrency: true },
+ };
+
+ let first = live
+ .acquire(link_id, ApiOperationKind::GetMessage)
+ .await
+ .unwrap();
+ let second = live
+ .acquire(link_id, ApiOperationKind::GetMessage)
+ .await
+ .unwrap();
+ live.finish(first, Some(&concurrency_429)).await;
+ // Requests already in flight under the old cap don't lower it again.
+ live.finish(second, Some(&concurrency_429)).await;
+ let _: () = redis::cmd("DEL")
+ .arg(&cooldown)
+ .arg(&backfill_cooldown)
+ .query_async(&mut connection.clone())
+ .await
+ .unwrap();
+
+ let backfill_permit = backfill
+ .acquire(link_id, ApiOperationKind::GetMessage)
+ .await
+ .unwrap();
+ assert!(
+ backfill
+ .acquire(link_id, ApiOperationKind::GetMessage)
+ .await
+ .is_err(),
+ "backfill keeps one of the three slots Gmail allows"
+ );
+ let mut permits = vec![backfill_permit];
+ for _ in 0..2 {
+ permits.push(
+ live.acquire(link_id, ApiOperationKind::GetMessage)
+ .await
+ .unwrap(),
+ );
+ }
+ assert_eq!(
+ refusal_delay(live.acquire(link_id, ApiOperationKind::GetMessage).await),
+ Duration::from_secs(1),
+ "the mailbox's cap fell from five to three"
+ );
+
+ // Two minutes later the cap is back at five.
+ let rewind = r#"
+ redis.call('HINCRBY', KEYS[1], 'at', -120000)
+ return 1
+ "#;
+ let _: i32 = redis::Script::new(rewind)
+ .key(&limit)
+ .invoke_async(&mut connection.clone())
+ .await
+ .unwrap();
+ for _ in 0..2 {
+ permits.push(
+ live.acquire(link_id, ApiOperationKind::GetMessage)
+ .await
+ .unwrap(),
+ );
+ }
+ cleanup(&connection, link_id).await;
+
+ let permit = live
+ .acquire(link_id, ApiOperationKind::GetMessage)
+ .await
+ .unwrap();
+ live.finish(
+ permit,
+ Some(&EmailApiError::RateLimited {
+ retry_after: None,
+ origin: RateLimitOrigin::Provider { concurrency: false },
+ }),
+ )
+ .await;
+ let learned: Option<u32> = redis::cmd("HGET")
+ .arg(&limit)
+ .arg("cap")
+ .query_async(&mut connection.clone())
+ .await
+ .unwrap();
+ assert_eq!(learned, None, "a quota 429 leaves the cap alone");
+ cleanup(&connection, link_id).await;
+}
+
+#[tokio::test]
+#[ignore = "requires local Redis on port 6379; uses only unique mailbox keys"]
+async fn an_admitted_request_reports_the_mailbox_load_it_joined() {
+ let connection = connection().await;
+ let live = RedisGmailRequestGate::new(connection.clone(), RateBudget::Live);
+ let link_id = Uuid::now_v7();
+
+ let first = live
+ .acquire(link_id, ApiOperationKind::GetMessage)
+ .await
+ .unwrap();
+ let second = live
+ .acquire(link_id, ApiOperationKind::GetMessage)
+ .await
+ .unwrap();
+ assert_eq!(
+ first.load(),
+ Some(MailboxLoad {
+ in_flight: 1,
+ cap: 5
+ })
+ );
+ assert_eq!(
+ second.load(),
+ Some(MailboxLoad {
+ in_flight: 2,
+ cap: 5
+ })
+ );
+
+ live.finish(
+ first,
+ Some(&EmailApiError::RateLimited {
+ retry_after: None,
+ origin: RateLimitOrigin::Provider { concurrency: true },
+ }),
+ )
+ .await;
+ let _: () = redis::cmd("DEL")
+ .arg(&keys(link_id)[2])
+ .query_async(&mut connection.clone())
+ .await
+ .unwrap();
+ let third = live
+ .acquire(link_id, ApiOperationKind::GetMessage)
+ .await
+ .unwrap();
+ assert_eq!(
+ third.load(),
+ Some(MailboxLoad {
+ in_flight: 2,
+ cap: 3
+ }),
+ "the load reports the learned cap"
+ );
+ cleanup(&connection, link_id).await;
+}
+
+#[tokio::test]
+#[ignore = "requires local Redis on port 6379; uses only unique mailbox keys"]
+async fn quota_units_are_refused_until_the_oldest_second_leaves_the_window() {
+ let connection = connection().await;
+ let live = RedisGmailRequestGate::new(connection.clone(), RateBudget::Live);
+ let mut backfill = RedisGmailRequestGate::new(connection.clone(), RateBudget::Backfill);
+ backfill.wait = Duration::ZERO;
+ let link_id = Uuid::now_v7();
+ let fill = r#"
+ local clock = redis.call('TIME')
+ redis.call('HSET', KEYS[1], tonumber(clock[1]) - 2, ARGV[1])
+ redis.call('HSET', KEYS[1], tonumber(clock[1]) - 1, ARGV[2])
+ redis.call('HSET', KEYS[1], tonumber(clock[1]) - 60, 100000)
+ return 1
+ "#;
+ let _: i32 = redis::Script::new(fill)
+ .key(&keys(link_id)[1])
+ .arg(200)
+ .arg(2_000)
+ .invoke_async(&mut connection.clone())
+ .await
+ .unwrap();
+
+ // 2,200 units are in the window; the expired bucket no longer counts.
+ let permit = live
+ .acquire(link_id, ApiOperationKind::ListThreads)
+ .await
+ .unwrap();
+ live.finish(permit, None).await;
+ // A send fits once the oldest second leaves the window; backfill's lower
+ // threshold also has to wait for the next one.
+ let live_delay = refusal_delay(live.acquire(link_id, ApiOperationKind::SendMessage).await);
+ let backfill_delay = refusal_delay(
+ backfill
+ .acquire(link_id, ApiOperationKind::GetMessage)
+ .await,
+ );
+ assert!(live_delay > Duration::from_secs(6) && live_delay <= Duration::from_secs(8));
+ assert!(backfill_delay > Duration::from_secs(7) && backfill_delay <= Duration::from_secs(9));
+ assert!(backfill_delay > live_delay);
+ cleanup(&connection, link_id).await;
+}
+
+#[tokio::test]
+#[ignore = "requires local Redis on port 6379; uses only unique mailbox keys"]
+async fn a_redis_failure_defers_the_request_instead_of_admitting_it() {
+ let connection = connection().await;
+ let live = RedisGmailRequestGate::new(connection.clone(), RateBudget::Live);
+ let link_id = Uuid::now_v7();
+ let _: () = redis::cmd("SET")
+ .arg(&keys(link_id)[0])
+ .arg("not a sorted set")
+ .query_async(&mut connection.clone())
+ .await
+ .unwrap();
+
+ assert_eq!(
+ live.acquire(link_id, ApiOperationKind::ModifyMessageLabels)
+ .await
+ .err(),
+ Some(RateLimitRefusal::new(Some(UNAVAILABLE_RETRY_AFTER)))
+ );
+ cleanup(&connection, link_id).await;
+}
+
+#[tokio::test]
+#[ignore = "requires local Redis on port 6379; kills only its own connection"]
+async fn admission_resumes_after_redis_drops_the_connection() {
+ let connection = connection().await;
+ let live = RedisGmailRequestGate::new(connection.clone(), RateBudget::Live);
+ let link_id = Uuid::now_v7();
+ let id: i64 = redis::cmd("CLIENT")
+ .arg("ID")
+ .query_async(&mut connection.clone())
+ .await
+ .unwrap();
+ let _: redis::RedisResult<i32> = redis::cmd("CLIENT")
+ .arg("KILL")
+ .arg("ID")
+ .arg(id)
+ .query_async(&mut connection.clone())
+ .await;
+
+ // The first request after the drop may still meet the dead connection.
+ let first = live
+ .acquire(link_id, ApiOperationKind::GetMessage)
+ .await
+ .err();
+ assert!(
+ first.is_none() || first == Some(RateLimitRefusal::new(Some(UNAVAILABLE_RETRY_AFTER))),
+ "{first:?}"
+ );
+ assert!(
+ live.acquire(link_id, ApiOperationKind::GetMessage)
+ .await
+ .is_ok(),
+ "the gate reconnects instead of refusing until a restart"
+ );
+ cleanup(&connection, link_id).await;
+}
+
+#[test]
+fn contacts_use_a_one_unit_shared_budget_proxy() {
+ assert_eq!(gmail_operation(ApiOperationKind::ListContacts).cost(), 1);
+}
+
+#[test]
+fn composite_blocklist_operations_use_the_dominant_filter_write_cost() {
+ assert_eq!(gmail_operation(ApiOperationKind::BlockSender).cost(), 5);
+ assert_eq!(gmail_operation(ApiOperationKind::UnblockSender).cost(), 5);
+}
diff --git a/services/email_service/src/outbound/email_api/mod.rs b/services/email_service/src/outbound/email_api/mod.rs
index f9f39d6f04..892439240e 100644
--- a/services/email_service/src/outbound/email_api/mod.rs
+++ b/services/email_service/src/outbound/email_api/mod.rs
@@ -1,12 +1,12 @@
//! Email API infrastructure adapters.
+mod gmail_gate;
mod microsoft;
-mod rate_limiter;
mod token_source;
pub use email_api_client::outbound::microsoft_gate::RedisMicrosoftRequestGate;
+pub use gmail_gate::{RateBudget, RedisGmailRequestGate};
pub use microsoft::{MicrosoftCredentialsClient, ProviderMailboxGateway};
-pub use rate_limiter::{RateBudget, RedisProviderRateLimiter};
pub use token_source::{EmailServiceTokenSource, StaticTokenSource};
use email_api_client::GmailApiClientRepository;
@@ -16,5 +16,5 @@ use email_api_client::domain::service::EmailApiClientServiceImpl;
pub type GmailApi = EmailApiClientServiceImpl<
GmailApiClientRepository,
EmailServiceTokenSource,
- RedisProviderRateLimiter,
+ RedisGmailRequestGate,
>;
diff --git a/services/email_service/src/outbound/email_api/rate_limiter.rs b/services/email_service/src/outbound/email_api/rate_limiter.rs
deleted file mode 100644
index a07f7bf62b..0000000000
--- a/services/email_service/src/outbound/email_api/rate_limiter.rs
+++ /dev/null
@@ -1,114 +0,0 @@
-//! Redis-backed provider rate-limiter adapter.
-
-#[cfg(test)]
-mod test;
-
-use email_api_client::domain::models::{ApiOperationKind, RateLimitRefusal};
-use email_api_client::domain::ports::ProviderRateLimiter;
-use models_email::gmail::operations::GmailApiOperation;
-use uuid::Uuid;
-
-use crate::util::redis::RedisClient;
-use crate::util::redis::rate_limit::RateLimitArgs;
-
-/// Selects the shared provider quota budget used by a service instance.
-///
-/// `Live` and `Backfill` are two *thresholds over one shared Redis window*,
-/// not two counters: every operation charges the same per-user key, and the
-/// budget only decides the limit that charge is compared against. Splitting
-/// the Redis key per budget would let live + backfill together exceed Gmail's
-/// per-user quota — and would silently reset quota state already deployed
-/// under the shared key. Don't "fix" the key sharing.
-#[derive(Debug, Clone, Copy, PartialEq, Eq)]
-pub enum RateBudget {
- /// The normal threshold used by interactive and incremental operations.
- Live,
- /// The lower threshold reserved for mailbox backfills, leaving headroom
- /// for live operations while a backfill runs.
- Backfill,
-}
-
-impl RateBudget {
- fn is_backfill(self) -> bool {
- matches!(self, Self::Backfill)
- }
-}
-
-/// Applies Gmail quota policy through the email service's Redis rate limiter.
-///
-/// Redis failures remain fail-open because [`RedisClient::is_rate_limited`]
-/// reports them as an allowed request.
-#[derive(Clone)]
-pub struct RedisProviderRateLimiter {
- redis_client: RedisClient,
- budget: RateBudget,
-}
-
-impl RedisProviderRateLimiter {
- /// Creates a provider rate limiter using the selected quota budget.
- pub fn new(redis_client: RedisClient, budget: RateBudget) -> Self {
- Self {
- redis_client,
- budget,
- }
- }
-}
-
-impl ProviderRateLimiter for RedisProviderRateLimiter {
- async fn check_rate_limit(
- &self,
- link_id: Uuid,
- operation: ApiOperationKind,
- ) -> Result<(), RateLimitRefusal> {
- let args = rate_limit_args(link_id, operation, self.budget);
-
- rate_limit_result(self.redis_client.is_rate_limited(args).await)
- }
-}
-
-fn rate_limit_result(is_limited: bool) -> Result<(), RateLimitRefusal> {
- if is_limited {
- return Err(RateLimitRefusal::new(None));
- }
-
- Ok(())
-}
-
-fn rate_limit_args(
- link_id: Uuid,
- operation: ApiOperationKind,
- budget: RateBudget,
-) -> RateLimitArgs {
- RateLimitArgs {
- user_id: link_id,
- operation: gmail_operation(operation),
- is_backfill: budget.is_backfill(),
- }
-}
-
-fn gmail_operation(operation: ApiOperationKind) -> GmailApiOperation {
- match operation {
- ApiOperationKind::GetProfile => GmailApiOperation::UsersGetProfile,
- ApiOperationKind::ListChanges => GmailApiOperation::HistoryList,
- ApiOperationKind::ListLabels => GmailApiOperation::LabelsList,
- ApiOperationKind::CreateLabel => GmailApiOperation::LabelsCreate,
- ApiOperationKind::DeleteLabel => GmailApiOperation::LabelsDelete,
- ApiOperationKind::GetMessage => GmailApiOperation::MessagesGet,
- ApiOperationKind::ListMessages => GmailApiOperation::MessagesList,
- ApiOperationKind::ModifyMessageLabels => GmailApiOperation::MessagesModify,
- ApiOperationKind::GetAttachment => GmailApiOperation::MessagesAttachmentsGet,
- ApiOperationKind::SendMessage => GmailApiOperation::MessagesSend,
- ApiOperationKind::GetThread => GmailApiOperation::ThreadsGet,
- ApiOperationKind::ListThreads => GmailApiOperation::ThreadsList,
- // People API calls share this budget in v1. Gmail has no corresponding
- // quota operation, so use a one-unit operation as the accounting proxy.
- ApiOperationKind::ListContacts => GmailApiOperation::UsersGetProfile,
- // Block and unblock are composite filter operations. One preflight
- // check charges the dominant write operation rather than every request.
- ApiOperationKind::BlockSender => GmailApiOperation::SettingsFiltersCreate,
- ApiOperationKind::UnblockSender => GmailApiOperation::SettingsFiltersDelete,
- ApiOperationKind::ListBlockedSenders => GmailApiOperation::SettingsFiltersList,
- ApiOperationKind::Subscribe => GmailApiOperation::Watch,
- ApiOperationKind::Unsubscribe => GmailApiOperation::Stop,
- }
-}
diff --git a/services/email_service/src/outbound/email_api/rate_limiter/test.rs b/services/email_service/src/outbound/email_api/rate_limiter/test.rs
deleted file mode 100644
index 8c43bf81d6..0000000000
--- a/services/email_service/src/outbound/email_api/rate_limiter/test.rs
+++ /dev/null
@@ -1,66 +0,0 @@
-use email_api_client::domain::models::ApiOperationKind;
-use models_email::gmail::operations::GmailApiOperation;
-use uuid::Uuid;
-
-use super::{
- RateBudget, RedisProviderRateLimiter, gmail_operation, rate_limit_args, rate_limit_result,
-};
-
-/// Redis being unreachable must fail open: quota accounting degrades, mail
-/// flow does not.
-#[tokio::test]
-async fn unreachable_redis_fails_open() {
- let redis_client = crate::util::redis::RedisClient::new(
- redis::Client::open("redis://127.0.0.1:1/").expect("client construction is offline"),
- 100,
- 50,
- 60,
- );
- let limiter = RedisProviderRateLimiter::new(redis_client, RateBudget::Live);
-
- let result = email_api_client::domain::ports::ProviderRateLimiter::check_rate_limit(
- &limiter,
- Uuid::new_v4(),
- ApiOperationKind::GetMessage,
- )
- .await;
-
- assert!(
- result.is_ok(),
- "redis outage must not refuse provider calls"
- );
-}
-
-#[test]
-fn contacts_use_a_one_unit_shared_budget_proxy() {
- assert_eq!(gmail_operation(ApiOperationKind::ListContacts).cost(), 1);
-}
-
-#[test]
-fn composite_blocklist_operations_use_the_dominant_filter_write_cost() {
- assert_eq!(gmail_operation(ApiOperationKind::BlockSender).cost(), 5);
- assert_eq!(gmail_operation(ApiOperationKind::UnblockSender).cost(), 5);
-}
-
-#[test]
-fn refusal_is_returned_only_for_a_denied_preflight_check() {
- assert!(rate_limit_result(false).is_ok());
-
- let refusal = rate_limit_result(true).expect_err("denied request should be refused");
- assert_eq!(refusal.retry_after, None);
-}
-
-#[test]
-fn rate_budget_selects_live_or_backfill_limit() {
- let link_id = Uuid::new_v4();
-
- let live = rate_limit_args(link_id, ApiOperationKind::GetMessage, RateBudget::Live);
- assert_eq!(live.user_id, link_id);
- assert_eq!(live.operation, GmailApiOperation::MessagesGet);
- assert!(!live.is_backfill);
-
- let backfill = rate_limit_args(link_id, ApiOperationKind::GetMessage, RateBudget::Backfill);
- assert_eq!(backfill.user_id, link_id);
- assert_eq!(backfill.operation, GmailApiOperation::MessagesGet);
- assert!(backfill.is_backfill);
-}
diff --git a/services/email_service/src/outbound/scheduled_delivery.rs b/services/email_service/src/outbound/scheduled_delivery.rs
index 92b9d00351..fc28f9c0b4 100644
--- a/services/email_service/src/outbound/scheduled_delivery.rs
+++ b/services/email_service/src/outbound/scheduled_delivery.rs
@@ -282,7 +282,7 @@ impl ScheduledMessageSender<ScheduledClaim, SentDelivery> for ScheduledDeliveryA
) -> Result<SentDelivery, SubmissionError> {
let ids = self
.email_api
- .send_prepared(&prepared.provider)
+ .send_prepared(prepared.provider)
.await
.map_err(classify_submission_error)?;
apply_sent_ids(&mut prepared.delivery.message, ids);
diff --git a/services/email_service/src/outbound/scheduled_delivery/test.rs b/services/email_service/src/outbound/scheduled_delivery/test.rs
index 747f07210d..5658b38017 100644
--- a/services/email_service/src/outbound/scheduled_delivery/test.rs
+++ b/services/email_service/src/outbound/scheduled_delivery/test.rs
@@ -48,7 +48,7 @@ fn explicit_provider_refusal_is_safe_to_restore() {
},
EmailApiError::RateLimited {
retry_after: None,
- origin: RateLimitOrigin::Provider,
+ origin: RateLimitOrigin::Provider { concurrency: false },
},
] {
assert!(matches!(
diff --git a/services/email_service/src/pubsub/backfill/backfill_attachment.rs b/services/email_service/src/pubsub/backfill/backfill_attachment.rs
index ab5febe94e..6fffe8a61a 100644
--- a/services/email_service/src/pubsub/backfill/backfill_attachment.rs
+++ b/services/email_service/src/pubsub/backfill/backfill_attachment.rs
@@ -41,10 +41,12 @@ pub async fn backfill_attachment(
upload_attachment(ctx_upload, &attachment_args)
.await
.map_err(|e| match e {
- UploadAttachmentError::RateLimited => ProcessingError::Retryable(DetailedError {
- reason: FailureReason::GmailApiRateLimited,
- source: anyhow::Error::new(e).context("Failed to upload attachment"),
- }),
+ UploadAttachmentError::RateLimited(error) => {
+ ProcessingError::Retryable(DetailedError {
+ reason: FailureReason::GmailApiRateLimited,
+ source: anyhow::Error::new(error).context("Failed to upload attachment"),
+ })
+ }
_ => ProcessingError::NonRetryable(DetailedError {
reason: FailureReason::GmailApiFailed,
source: anyhow::Error::new(e).context("Failed to upload attachment"),
diff --git a/services/email_service/src/pubsub/backfill/backfill_thread.rs b/services/email_service/src/pubsub/backfill/backfill_thread.rs
index 6be59c5549..e051b36e09 100644
--- a/services/email_service/src/pubsub/backfill/backfill_thread.rs
+++ b/services/email_service/src/pubsub/backfill/backfill_thread.rs
@@ -53,8 +53,11 @@ pub async fn backfill_thread(
.await
.map_err(|error| map_email_api_error(error, "Failed to get provider thread message IDs"))?;
- // insert thread object
- let thread_db_id = email_db_client::threads::insert::insert_blank_thread(
+ // The priority pass and the normal sweep both enqueue a thread, and both
+ // copies can pass the check above while they wait for Gmail. Only the copy
+ // that inserts the thread backfills its messages; counting them twice
+ // would complete the thread early and strand the duplicates.
+ let Some(thread_db_id) = email_db_client::threads::insert::insert_blank_thread(
&ctx.db,
&thread_provider_id,
link.id,
@@ -65,7 +68,11 @@ pub async fn backfill_thread(
e,
format!("Failed to insert thread shell for {}", thread_provider_id),
)
- })?;
+ })?
+ else {
+ incr_completed_threads(ctx, link, scope.job_id).await?;
+ return Ok(());
+ };
// create backfill pubsub message for each email message
enqueue_backfill_messages(ctx, scope, link, thread_db_id, message_ids).await
diff --git a/services/email_service/src/pubsub/backfill/deferral.rs b/services/email_service/src/pubsub/backfill/deferral.rs
new file mode 100644
index 0000000000..0eae567272
--- /dev/null
+++ b/services/email_service/src/pubsub/backfill/deferral.rs
@@ -0,0 +1,76 @@
+//! When rate-limited backfill work goes back to Gmail.
+//!
+//! A rate-limited operation is sent back to the backfill queue as a new
+//! message instead of waiting out the visibility timeout. The request gate
+//! refuses most of a large backfill's operations many times before they get
+//! one of the mailbox's backfill slots, and each wait on the visibility
+//! timeout would spend one of the queue's 20 deliveries. A refusal must never
+//! cost a backfill its work.
+
+use std::ops::RangeInclusive;
+use std::time::Duration;
+
+use email_api_client::domain::models::EmailApiError;
+use models_email::email::service::pubsub::{DetailedError, FailureReason};
+use rand::Rng;
+use sqs_client::email::BACKFILL_DEFERRALS_ATTRIBUTE;
+
+#[cfg(test)]
+mod test;
+
+/// Ceiling of the first deferral's delay; each later deferral doubles it.
+const FIRST_DEFERRAL: Duration = Duration::from_secs(1);
+/// Longest wait between attempts. A mailbox's deferred backlog then comes
+/// back faster than its three backfill slots drain it, so slots don't sit
+/// idle, while long-waiting work retries about once a minute.
+const MAX_DEFERRAL: Duration = Duration::from_secs(60);
+/// SQS's longest message delay, which bounds a provider's longer Retry-After.
+const MAX_SQS_DELAY: Duration = Duration::from_secs(900);
+
+/// The wait a rate-limited operation was told to observe, when the failure is
+/// a rate limit: the request gate's or Gmail's Retry-After, if either gave one.
+pub(crate) fn rate_limit(error: &DetailedError) -> Option<Option<Duration>> {
+ if error.reason != FailureReason::GmailApiRateLimited {
+ return None;
+ }
+ Some(match error.source.downcast_ref::<EmailApiError>() {
+ Some(EmailApiError::RateLimited { retry_after, .. }) => *retry_after,
+ _ => None,
+ })
+}
+
+/// How often the received operation has already been deferred.
+pub(crate) fn deferrals(message: &aws_sdk_sqs::types::Message) -> u32 {
+ message
+ .message_attributes()
+ .and_then(|attributes| attributes.get(BACKFILL_DEFERRALS_ATTRIBUTE))
+ .and_then(|value| value.string_value())
+ .and_then(|value| value.parse().ok())
+ .unwrap_or(0)
+}
+
+/// Whole seconds to hold an operation deferred `deferrals` times before.
+pub(crate) fn delay_seconds(deferrals: u32, retry_after: Option<Duration>) -> i32 {
+ delay(deferrals, retry_after, |range| {
+ rand::rng().random_range(range)
+ })
+}
+
+/// Exponential backoff with equal jitter, so operations refused together come
+/// back spread out, and never sooner than the refusal's Retry-After.
+fn delay(
+ deferrals: u32,
+ retry_after: Option<Duration>,
+ jitter: impl FnOnce(RangeInclusive<Duration>) -> Duration,
+) -> i32 {
+ let ceiling = 2u32
+ .checked_pow(deferrals)
+ .and_then(|factor| FIRST_DEFERRAL.checked_mul(factor))
+ .map_or(MAX_DEFERRAL, |ceiling| ceiling.min(MAX_DEFERRAL));
+ let delay = jitter(ceiling / 2..=ceiling)
+ .max(retry_after.unwrap_or_default())
+ .min(MAX_SQS_DELAY);
+ // SQS counts whole seconds; rounding down would retry before Retry-After.
+ let seconds = delay.as_secs() + u64::from(delay.subsec_nanos() > 0);
+ i32::try_from(seconds.max(1)).unwrap_or(i32::MAX)
+}
diff --git a/services/email_service/src/pubsub/backfill/deferral/test.rs b/services/email_service/src/pubsub/backfill/deferral/test.rs
new file mode 100644
index 0000000000..ff001f258b
--- /dev/null
+++ b/services/email_service/src/pubsub/backfill/deferral/test.rs
@@ -0,0 +1,108 @@
+use std::time::Duration;
+
+use email_api_client::domain::models::{EmailApiError, RateLimitOrigin};
+use models_email::email::service::pubsub::ProcessingError;
+
+use super::*;
+use crate::pubsub::backfill::db_error::map_db_error;
+use crate::pubsub::backfill::email_api_error::map_email_api_error;
+
+fn retryable(error: ProcessingError) -> DetailedError {
+ match error {
+ ProcessingError::Retryable(detail) => detail,
+ ProcessingError::NonRetryable(detail) => {
+ panic!("expected a retryable error, got {:?}", detail.reason)
+ }
+ }
+}
+
+fn longest(range: RangeInclusive<Duration>) -> Duration {
+ *range.end()
+}
+
+fn shortest(range: RangeInclusive<Duration>) -> Duration {
+ *range.start()
+}
+
+#[test]
+fn a_refused_thread_listing_page_is_deferred_instead_of_spending_a_delivery() {
+ let refused = retryable(map_email_api_error(
+ EmailApiError::RateLimited {
+ retry_after: Some(Duration::from_millis(2_500)),
+ origin: RateLimitOrigin::Local,
+ },
+ "Failed to list provider threads",
+ ));
+
+ assert_eq!(
+ rate_limit(&refused),
+ Some(Some(Duration::from_millis(2_500)))
+ );
+}
+
+#[test]
+fn a_gmail_rate_limit_without_retry_after_is_deferred_too() {
+ let throttled = retryable(map_email_api_error(
+ EmailApiError::RateLimited {
+ retry_after: None,
+ origin: RateLimitOrigin::Provider { concurrency: false },
+ },
+ "Failed to fetch message",
+ ));
+
+ assert_eq!(rate_limit(&throttled), Some(None));
+}
+
+#[test]
+fn other_retryable_failures_keep_waiting_out_the_visibility_timeout() {
+ let database = retryable(map_db_error(
+ anyhow::anyhow!("connection reset"),
+ "Failed to insert thread",
+ ));
+ let transient = retryable(map_email_api_error(
+ EmailApiError::Transient {
+ message: "Gmail API returned 503".to_string(),
+ },
+ "Failed to fetch message",
+ ));
+
+ assert_eq!(rate_limit(&database), None);
+ assert_eq!(rate_limit(&transient), None);
+}
+
+#[test]
+fn deferrals_grow_from_a_second_to_a_minute() {
+ assert_eq!(delay(0, None, shortest), 1);
+ assert_eq!(delay(0, None, longest), 1);
+ assert_eq!(delay(1, None, longest), 2);
+ assert_eq!(delay(3, None, shortest), 4);
+ assert_eq!(delay(3, None, longest), 8);
+ assert_eq!(delay(6, None, longest), 60);
+ assert_eq!(delay(40, None, shortest), 30);
+ assert_eq!(delay(40, None, longest), 60);
+}
+
+#[test]
+fn a_deferral_waits_at_least_its_retry_after_rounded_up() {
+ assert_eq!(delay(0, Some(Duration::from_millis(4_200)), longest), 5);
+ assert_eq!(delay(3, Some(Duration::from_millis(500)), longest), 8);
+ assert_eq!(delay(0, Some(Duration::from_secs(3_600)), longest), 900);
+}
+
+#[test]
+fn the_deferral_count_is_read_from_the_message() {
+ let deferred = aws_sdk_sqs::types::Message::builder()
+ .message_attributes(
+ BACKFILL_DEFERRALS_ATTRIBUTE,
+ aws_sdk_sqs::types::MessageAttributeValue::builder()
+ .data_type("Number")
+ .string_value("4")
+ .build()
+ .unwrap(),
+ )
+ .build();
+ let fresh = aws_sdk_sqs::types::Message::builder().build();
+
+ assert_eq!(deferrals(&deferred), 4);
+ assert_eq!(deferrals(&fresh), 0);
+}
diff --git a/services/email_service/src/pubsub/backfill/email_api_error/test.rs b/services/email_service/src/pubsub/backfill/email_api_error/test.rs
index b0259a5037..6b22d6fcb7 100644
--- a/services/email_service/src/pubsub/backfill/email_api_error/test.rs
+++ b/services/email_service/src/pubsub/backfill/email_api_error/test.rs
@@ -24,7 +24,9 @@ fn maps_rate_limits_to_retryable_rate_limit_failures() {
assert_mapping(
EmailApiError::RateLimited {
retry_after: Some(Duration::from_secs(5)),
- origin: email_api_client::domain::models::RateLimitOrigin::Provider,
+ origin: email_api_client::domain::models::RateLimitOrigin::Provider {
+ concurrency: false,
+ },
},
true,
FailureReason::GmailApiRateLimited,
diff --git a/services/email_service/src/pubsub/backfill/error_handlers.rs b/services/email_service/src/pubsub/backfill/error_handlers.rs
index e6d40fb24d..8e15bf5cad 100644
--- a/services/email_service/src/pubsub/backfill/error_handlers.rs
+++ b/services/email_service/src/pubsub/backfill/error_handlers.rs
@@ -1,3 +1,4 @@
+use crate::pubsub::backfill::deferral;
use crate::pubsub::backfill::increment_counters::{
incr_completed_messages, incr_completed_threads,
};
@@ -9,6 +10,7 @@ use models_email::email::service::backfill::{
};
use models_email::email::service::pubsub::{DetailedError, FailureReason};
use sqs_worker::cleanup_message;
+use std::time::Duration;
use uuid::Uuid;
/// Handles non-retryable errors by updating the appropriate status in the database and cleaning up the SQS message
@@ -107,6 +109,40 @@ async fn notify_job_failed(ctx: &PubSubContext, job_id: Uuid) {
}
}
+/// Sends a rate-limited operation back to the queue after a growing delay
+/// (see [`deferral`]), keeping its delivery budget for real failures. If the
+/// send fails, the message is left for redelivery after its visibility timeout.
+#[tracing::instrument(
+ skip(ctx, message, data, e),
+ fields(link_id = ?data.backfill_operation.link_id()),
+ err
+)]
+pub async fn defer_rate_limited_operation(
+ ctx: &PubSubContext,
+ message: &aws_sdk_sqs::types::Message,
+ data: &BackfillPubsubMessage,
+ e: &DetailedError,
+ retry_after: Option<Duration>,
+) -> anyhow::Result<()> {
+ let deferrals = deferral::deferrals(message);
+ let delay_seconds = deferral::delay_seconds(deferrals, retry_after);
+ if let Err(error) = ctx
+ .sqs_client
+ .defer_email_backfill_message(data, delay_seconds, deferrals + 1)
+ .await
+ {
+ tracing::warn!(error = ?error, "Failed to defer a rate-limited backfill operation; it will be redelivered");
+ return handle_retryable_error(data, e).await;
+ }
+ tracing::debug!(
+ deferrals,
+ delay_seconds,
+ "Rate-limited backfill operation deferred"
+ );
+ cleanup_message(&ctx.sqs_worker, message).await?;
+ Ok(())
+}
+
/// Handles retryable errors by updating status to InProgress and adding the error message
#[tracing::instrument(
skip(data, _e),
diff --git a/services/email_service/src/pubsub/backfill/mod.rs b/services/email_service/src/pubsub/backfill/mod.rs
index b56880458c..459a9edad3 100644
--- a/services/email_service/src/pubsub/backfill/mod.rs
+++ b/services/email_service/src/pubsub/backfill/mod.rs
@@ -2,6 +2,7 @@ mod backfill_attachment;
mod backfill_message;
mod backfill_thread;
pub(crate) mod db_error;
+mod deferral;
mod depopulate_crm_contact;
mod depopulate_crm_for_user;
pub(crate) mod email_api_error;
diff --git a/services/email_service/src/pubsub/backfill/process.rs b/services/email_service/src/pubsub/backfill/process.rs
index afe3e41f0d..36c0684573 100644
--- a/services/email_service/src/pubsub/backfill/process.rs
+++ b/services/email_service/src/pubsub/backfill/process.rs
@@ -1,5 +1,5 @@
use crate::pubsub::backfill::{
- backfill_attachment, backfill_message, backfill_thread, depopulate_crm_contact,
+ backfill_attachment, backfill_message, backfill_thread, deferral, depopulate_crm_contact,
depopulate_crm_for_user, error_handlers, increment_counters, init, list_threads,
populate_crm_contact, populate_crm_for_user, seed_sent_contact, update_metadata,
};
@@ -45,9 +45,13 @@ pub async fn process_message(
}
// A temporary failure occurred. We log it and don't clean up the message, so it gets retried
- Err(ProcessingError::Retryable(e)) => {
- error_handlers::handle_retryable_error(&data, &e).await
- }
+ Err(ProcessingError::Retryable(e)) => match deferral::rate_limit(&e) {
+ Some(retry_after) => {
+ error_handlers::defer_rate_limited_operation(&ctx, message, &data, &e, retry_after)
+ .await
+ }
+ None => error_handlers::handle_retryable_error(&data, &e).await,
+ },
}
}
diff --git a/services/email_service/src/pubsub/gmail_ops/email_api_error/test.rs b/services/email_service/src/pubsub/gmail_ops/email_api_error/test.rs
index 7172cb202b..f03d0b004b 100644
--- a/services/email_service/src/pubsub/gmail_ops/email_api_error/test.rs
+++ b/services/email_service/src/pubsub/gmail_ops/email_api_error/test.rs
@@ -6,7 +6,7 @@ use super::*;
fn rate_limits_use_queue_routing_policy() {
let provider_rate_limit = EmailApiError::RateLimited {
retry_after: Some(Duration::from_secs(30)),
- origin: email_api_client::domain::models::RateLimitOrigin::Provider,
+ origin: email_api_client::domain::models::RateLimitOrigin::Provider { concurrency: false },
};
assert_eq!(error_policy(&provider_rate_limit), ErrorPolicy::RateLimited);
@@ -73,7 +73,7 @@ fn not_found_is_success_for_delete_label() {
fn modify_message_labels_rate_limited_and_transient_failures_do_not_revert() {
assert!(!is_permanent_mutation_error(&EmailApiError::RateLimited {
retry_after: None,
- origin: email_api_client::domain::models::RateLimitOrigin::Provider,
+ origin: email_api_client::domain::models::RateLimitOrigin::Provider { concurrency: false },
}));
assert!(!is_permanent_mutation_error(&EmailApiError::Transient {
message: "timeout".to_string(),
diff --git a/services/email_service/src/pubsub/inbox_sync/email_api_error.rs b/services/email_service/src/pubsub/inbox_sync/email_api_error.rs
index 682629d318..dec1f7076d 100644
--- a/services/email_service/src/pubsub/inbox_sync/email_api_error.rs
+++ b/services/email_service/src/pubsub/inbox_sync/email_api_error.rs
@@ -1,4 +1,4 @@
-use email_api_client::domain::models::{EmailApiError, RateLimitOrigin};
+use email_api_client::domain::models::EmailApiError;
use models_email::gmail::inbox_sync::{InboxSyncOperation, InboxSyncPubsubMessage};
use models_email::service::pubsub::{DetailedError, FailureReason, ProcessingError};
@@ -50,25 +50,6 @@ pub(crate) async fn handle_operation_error(
non_retryable(error)
}
-pub(crate) fn handle_gmail_message_error(error: EmailApiError) -> ProcessingError {
- // Local budget refusals drop the notification by design: no provider
- // quota was consumed and a later notification covers the same cursor
- // range, avoiding fan-out backpressure. A provider-origin 429 retries
- // (main's behavior) so an idle mailbox does not stay unsynced until its
- // next change.
- if matches!(
- error,
- EmailApiError::RateLimited {
- origin: RateLimitOrigin::Local,
- ..
- }
- ) {
- return non_retryable(error);
- }
-
- processing_error(error)
-}
-
fn processing_error(error: EmailApiError) -> ProcessingError {
if error.is_transient() || matches!(error, EmailApiError::RateLimited { .. }) {
ProcessingError::Retryable(detail(error))
diff --git a/services/email_service/src/pubsub/inbox_sync/email_api_error/test.rs b/services/email_service/src/pubsub/inbox_sync/email_api_error/test.rs
index bcf89f7df8..f90c8cc827 100644
--- a/services/email_service/src/pubsub/inbox_sync/email_api_error/test.rs
+++ b/services/email_service/src/pubsub/inbox_sync/email_api_error/test.rs
@@ -1,5 +1,7 @@
use std::time::Duration;
+use email_api_client::domain::models::RateLimitOrigin;
+
use super::*;
#[test]
@@ -15,38 +17,6 @@ fn retry_worker_uses_visibility_timeout_for_rate_limits() {
);
}
-#[test]
-fn gmail_message_local_rate_refusal_is_non_retryable() {
- let error = handle_gmail_message_error(EmailApiError::RateLimited {
- retry_after: Some(Duration::from_secs(10)),
- origin: RateLimitOrigin::Local,
- });
-
- assert!(matches!(
- error,
- ProcessingError::NonRetryable(DetailedError {
- reason: FailureReason::GmailApiRateLimited,
- ..
- })
- ));
-}
-
-#[test]
-fn gmail_message_provider_429_is_retried() {
- let error = handle_gmail_message_error(EmailApiError::RateLimited {
- retry_after: Some(Duration::from_secs(10)),
- origin: RateLimitOrigin::Provider,
- });
-
- assert!(matches!(
- error,
- ProcessingError::Retryable(DetailedError {
- reason: FailureReason::GmailApiRateLimited,
- ..
- })
- ));
-}
-
/// Pins the full `EmailApiError` → `ProcessingError` policy table for
/// inbox_sync operations (the non-rate-limit path of `handle_operation_error`
/// and the retry-worker rate-limit branch both route through
@@ -57,7 +27,7 @@ fn processing_error_policy_covers_every_variant() {
(
EmailApiError::RateLimited {
retry_after: Some(Duration::from_secs(5)),
- origin: RateLimitOrigin::Provider,
+ origin: RateLimitOrigin::Provider { concurrency: false },
},
true,
FailureReason::GmailApiRateLimited,
@@ -125,16 +95,3 @@ fn processing_error_policy_covers_every_variant() {
}
}
}
-
-#[test]
-fn outdated_cursor_is_a_terminal_history_failure() {
- let error = handle_gmail_message_error(EmailApiError::OutdatedCursor);
-
- assert!(matches!(
- error,
- ProcessingError::NonRetryable(DetailedError {
- reason: FailureReason::OutdatedHistoryId,
- ..
- })
- ));
-}
diff --git a/services/email_service/src/pubsub/inbox_sync/error_handlers.rs b/services/email_service/src/pubsub/inbox_sync/error_handlers.rs
index 39bc31ce3d..52b1ccddfa 100644
--- a/services/email_service/src/pubsub/inbox_sync/error_handlers.rs
+++ b/services/email_service/src/pubsub/inbox_sync/error_handlers.rs
@@ -12,8 +12,7 @@ pub async fn handle_non_retryable_error(
e: &DetailedError,
) -> anyhow::Result<()> {
// A rate-limited operation is only non-retryable here once it has been
- // re-enqueued to the retry queue, or when it is a notification that a
- // later one supersedes, so deleting it loses nothing.
+ // re-enqueued to the retry queue, so deleting it loses nothing.
if e.reason == FailureReason::GmailApiRateLimited {
tracing::debug!(error = %e, payload = format!("{:?}", data.operation), "Rate-limited inbox sync message deferred. The message will be deleted.");
} else {
diff --git a/services/email_service/src/pubsub/inbox_sync/operations/gmail_message.rs b/services/email_service/src/pubsub/inbox_sync/operations/gmail_message.rs
index e8fde95fbc..19fed01fb6 100644
--- a/services/email_service/src/pubsub/inbox_sync/operations/gmail_message.rs
+++ b/services/email_service/src/pubsub/inbox_sync/operations/gmail_message.rs
@@ -1,12 +1,12 @@
use crate::pubsub::context::PubSubContext;
-use crate::pubsub::inbox_sync::email_api_error::handle_gmail_message_error;
+use crate::pubsub::inbox_sync::email_api_error::handle_operation_error;
use crate::util::process_pre_insert::sync_labels::sync_labels;
use email_api_client::domain::models::{EmailApiError, SyncCursor};
use models_email::email::service::backfill::BackfillJob;
use models_email::email::service::backfill::{
BackfillOperation, BackfillPubsubMessage, InitPayload, JobScopedPayload,
};
-use models_email::gmail::inbox_sync::GmailMessagePayload;
+use models_email::gmail::inbox_sync::{GmailMessagePayload, InboxSyncOperation};
use models_email::service::link::{Link, UserProvider};
use models_email::service::pubsub::{DetailedError, FailureReason, ProcessingError};
use sqlx::PgPool;
@@ -59,14 +59,14 @@ pub async fn gmail_message(
return Ok(());
}
- // Gmail message notifications intentionally refuse labels/history work when quota is
- // exhausted. A later notification will cover the same cursor range, avoiding fan-out
- // backpressure while preserving the separate retry queue for child operations.
- let labels = ctx
- .email_api
- .list_labels(link.id)
- .await
- .map_err(handle_gmail_message_error)?;
+ // A rate-limited notification is requeued like its child operations rather
+ // than dropped: an idle mailbox may get no later notification to cover
+ // this cursor range.
+ let operation = || InboxSyncOperation::GmailMessage(payload.clone());
+ let labels = match ctx.email_api.list_labels(link.id).await {
+ Ok(labels) => labels,
+ Err(error) => return Err(handle_operation_error(ctx, link.id, operation(), error).await),
+ };
sync_labels(&ctx.db, link.id, &labels).await.map_err(|e| {
ProcessingError::Retryable(DetailedError {
reason: FailureReason::DatabaseQueryFailed,
@@ -82,9 +82,10 @@ pub async fn gmail_message(
Ok(batch) => batch,
Err(EmailApiError::OutdatedCursor) => {
schedule_stale_cursor_backfill(ctx, link, payload.history_id).await?;
- return Err(handle_gmail_message_error(EmailApiError::OutdatedCursor));
+ let error = EmailApiError::OutdatedCursor;
+ return Err(handle_operation_error(ctx, link.id, operation(), error).await);
}
- Err(error) => return Err(handle_gmail_message_error(error)),
+ Err(error) => return Err(handle_operation_error(ctx, link.id, operation(), error).await),
};
ctx.gmail_history
diff --git a/services/email_service/src/pubsub/inbox_sync/operations/upsert_message.rs b/services/email_service/src/pubsub/inbox_sync/operations/upsert_message.rs
index 1e19b197ff..506cbe8585 100644
--- a/services/email_service/src/pubsub/inbox_sync/operations/upsert_message.rs
+++ b/services/email_service/src/pubsub/inbox_sync/operations/upsert_message.rs
@@ -6,7 +6,9 @@ use crate::pubsub::util::{
};
use crate::pubsub::util::{cg_refresh_email, publish_email_event};
use crate::util::process_pre_insert::{process_message_pre_insert, process_threads_pre_insert};
-use crate::util::upload_attachment::{UploadAttachmentContext, upload_attachment};
+use crate::util::upload_attachment::{
+ UploadAttachmentContext, UploadAttachmentError, upload_attachment,
+};
use contacts::domain::ports::ContactsIngress;
use email::domain::events::{
EmailEventOrigin, EmailMacroEvent, MessageDraftSyncedMetadata, MessageReceivedMetadata,
@@ -15,6 +17,7 @@ use email::domain::events::{
use email::domain::models::{PreviewCursorQuery, PreviewView, PreviewViewStandardLabel};
use email::domain::ports::EmailRepo;
use email::outbound::EmailPgRepo;
+use email_api_client::domain::models::EmailApiError;
use email_db_client::attachments::provider::upload_filters::{
attachment_is_document, attachment_is_media,
};
@@ -309,7 +312,7 @@ pub async fn upsert_message(
publish_email_event(&ctx.macro_event_broker, &event);
}
- handle_attachment_upload(ctx, link, payload, &message.attachments).await?;
+ let refused_upload = handle_attachment_upload(ctx, link, payload, &message.attachments).await?;
handle_contacts_sync(
ctx,
@@ -349,24 +352,38 @@ pub async fn upsert_message(
.await?;
}
+ // Requeued only after the steps above, which are safe to repeat: the retry
+ // finds the message saved, so it doesn't notify again.
+ if let Some(error) = refused_upload {
+ return Err(handle_operation_error(
+ ctx,
+ link.id,
+ InboxSyncOperation::UpsertMessage(payload.clone()),
+ error,
+ )
+ .await);
+ }
+
Ok(())
}
+/// Uploads the message's new attachments, returning the rate limit that
+/// refused an attachment fetch, if one did.
#[tracing::instrument(skip(ctx, attachments), err)]
async fn handle_attachment_upload(
ctx: &PubSubContext,
link: &link::Link,
payload: &UpsertMessagePayload,
attachments: &[Attachment],
-) -> result::Result<(), ProcessingError> {
+) -> result::Result<Option<EmailApiError>, ProcessingError> {
if cfg!(not(feature = "attachment_upload")) {
- return Ok(());
+ return Ok(None);
}
// Keep this parsed-attachment gate at least as permissive as the database claim filters.
let eligibility = attachment_upload_eligibility(attachments);
if !eligibility.documents && !eligibility.media {
- return Ok(());
+ return Ok(None);
}
// upload attachments to Macro
@@ -424,6 +441,7 @@ async fn handle_attachment_upload(
})
})?;
+ let mut uploads = Vec::with_capacity(attachments.len());
for attachment in attachments {
// get the email addresses of the recipients of the message
let recipient_emails: Vec<String> = message_recipients
@@ -444,13 +462,15 @@ async fn handle_attachment_upload(
AttachmentUploadDestination::Dss
};
- let attachment_upload_args = AttachmentUploadArgs {
+ uploads.push(AttachmentUploadArgs {
recipient_emails,
attachment_metadata: attachment,
backfill: false,
upload_destination,
- };
+ });
+ }
+ return upload_claimed_attachments(&ctx.db, uploads, |args| async move {
let ctx_upload = UploadAttachmentContext {
db: &ctx.db,
email_api: &ctx.email_api,
@@ -459,15 +479,56 @@ async fn handle_attachment_upload(
system_properties_service: &ctx.system_properties_service,
link,
};
+ upload_attachment(ctx_upload, &args).await
+ })
+ .await;
+ }
- // keep processing if it fails, best effort
- if let Err(e) = upload_attachment(ctx_upload, &attachment_upload_args).await {
- tracing::error!("Failed to upload attachment to Macro: {e}");
+ Ok(None)
+}
+
+/// Uploads claimed attachments in order. Other upload failures are logged and
+/// skipped, but a fetch the rate limit refused stops the run: the refused
+/// attachment and the ones not yet tried lose their claims so a retry of the
+/// upsert claims them again, and the refusal is returned.
+async fn upload_claimed_attachments<F, Fut>(
+ db: &PgPool,
+ uploads: Vec<AttachmentUploadArgs>,
+ mut upload: F,
+) -> result::Result<Option<EmailApiError>, ProcessingError>
+where
+ F: FnMut(AttachmentUploadArgs) -> Fut,
+ Fut: Future<Output = result::Result<String, UploadAttachmentError>>,
+{
+ let mut uploads = uploads.into_iter();
+ while let Some(args) = uploads.next() {
+ let attachment_id = args.attachment_metadata.attachment_db_id;
+ match upload(args).await {
+ Ok(_) => {}
+ Err(UploadAttachmentError::RateLimited(error)) => {
+ let unfinished = std::iter::once(attachment_id)
+ .chain(uploads.map(|args| args.attachment_metadata.attachment_db_id));
+ for attachment_id in unfinished {
+ email_db_client::attachments::provider::upload::release_upload_claim(
+ db,
+ attachment_id,
+ )
+ .await
+ .map_err(|e| {
+ ProcessingError::Retryable(DetailedError {
+ reason: FailureReason::DatabaseQueryFailed,
+ source: e.context("Failed to release attachment upload claim"),
+ })
+ })?;
+ }
+ return Ok(Some(error));
}
+ // keep processing if it fails, best effort
+ Err(e) => tracing::error!("Failed to upload attachment to Macro: {e}"),
}
}
- Ok(())
+ Ok(None)
}
#[derive(Debug, Default, PartialEq, Eq)]
diff --git a/services/email_service/src/pubsub/inbox_sync/operations/upsert_message/test.rs b/services/email_service/src/pubsub/inbox_sync/operations/upsert_message/test.rs
index 5effdb84c4..9fdeb1e77d 100644
--- a/services/email_service/src/pubsub/inbox_sync/operations/upsert_message/test.rs
+++ b/services/email_service/src/pubsub/inbox_sync/operations/upsert_message/test.rs
@@ -1,5 +1,8 @@
use super::*;
+use email_api_client::domain::models::RateLimitOrigin;
+use email_db_client::attachments::provider::upload::new_email_media_atts;
use macro_db_migrator::MACRO_DB_MIGRATIONS;
+use std::time::Duration;
fn id(s: &str) -> MacroUserIdStr<'static> {
MacroUserIdStr::try_from(s.to_string()).unwrap()
@@ -399,3 +402,76 @@ async fn inserting_into_a_concurrently_deleted_thread_is_retryable(pool: PgPool)
other => panic!("expected a retryable insert failure, got {other:?}"),
}
}
+
+#[sqlx::test(
+ migrator = "MACRO_DB_MIGRATIONS",
+ fixtures(
+ path = "../../../../../../../crates/email_db_client/fixtures",
+ scripts("upload_claim_tests")
+ )
+)]
+async fn a_refused_attachment_fetch_is_claimed_again_by_the_retry(
+ pool: PgPool,
+) -> anyhow::Result<()> {
+ let link_id = Uuid::parse_str("00000000-0000-0000-0000-000000000c01")?;
+ let image = Uuid::parse_str("00000000-0000-0000-0000-00000c2a0201")?;
+ let video = Uuid::parse_str("00000000-0000-0000-0000-00000c2a0202")?;
+ let animation = Uuid::parse_str("00000000-0000-0000-0000-00000c2a0203")?;
+ sqlx::query!(
+ "INSERT INTO email_attachments (id, message_id, provider_attachment_id, filename, mime_type)
+ VALUES ($1, '00000000-0000-0000-0000-0000000ec201', 'claim-test-media-att-3', 'animation.gif', 'image/gif')",
+ animation
+ )
+ .execute(&pool)
+ .await?;
+
+ let claimed = new_email_media_atts(&pool, link_id, "claim-test-media-msg").await?;
+ assert_eq!(
+ claimed
+ .iter()
+ .map(|a| a.attachment_db_id)
+ .collect::<Vec<_>>(),
+ vec![image, video, animation]
+ );
+ let uploads = claimed
+ .into_iter()
+ .map(|attachment_metadata| AttachmentUploadArgs {
+ recipient_emails: vec![],
+ attachment_metadata,
+ backfill: false,
+ upload_destination: AttachmentUploadDestination::Sfs,
+ })
+ .collect();
+
+ // The image uploads; the gate then refuses the video's fetch.
+ let refusal = EmailApiError::RateLimited {
+ retry_after: Some(Duration::from_secs(1)),
+ origin: RateLimitOrigin::Local,
+ };
+ let mut attempted = Vec::new();
+ let refused = upload_claimed_attachments(&pool, uploads, |args| {
+ let attachment_id = args.attachment_metadata.attachment_db_id;
+ attempted.push(attachment_id);
+ let result = if attachment_id == image {
+ Ok("sfs-image".to_string())
+ } else {
+ Err(UploadAttachmentError::RateLimited(refusal.clone()))
+ };
+ async move { result }
+ })
+ .await;
+
+ assert!(matches!(refused, Ok(Some(ref error)) if *error == refusal));
+ assert_eq!(attempted, vec![image, video]);
+ // The retried upsert claims the refused and untried attachments, not the uploaded one.
+ let reclaimed = new_email_media_atts(&pool, link_id, "claim-test-media-msg").await?;
+ assert_eq!(
+ reclaimed
+ .iter()
+ .map(|a| a.attachment_db_id)
+ .collect::<Vec<_>>(),
+ vec![video, animation]
+ );
+
+ Ok(())
+}
diff --git a/services/email_service/src/server.rs b/services/email_service/src/server.rs
index c6b4059918..c18b30a288 100644
--- a/services/email_service/src/server.rs
+++ b/services/email_service/src/server.rs
@@ -9,7 +9,7 @@ use crate::calendar_refresh::ConnectionGatewayCalendarRefresh;
use crate::calendar_request_gate::RedisCalendarRequestGate;
use crate::calendar_tokens::CalendarTokenProviderAdapter;
use crate::outbound::email_api::{
- EmailServiceTokenSource, GmailApi, RateBudget, RedisProviderRateLimiter,
+ EmailServiceTokenSource, GmailApi, RateBudget, RedisGmailRequestGate,
};
use anyhow::Context;
use calendar_events::{
@@ -121,8 +121,6 @@ pub async fn run() -> anyhow::Result<()> {
let redis_client = crate::util::redis::RedisClient::new(
redis_inner_client,
- config.redis_rate_limit_reqs,
- config.redis_rate_limit_reqs_backfill,
config.redis_rate_limit_window_secs,
);
@@ -203,7 +201,12 @@ pub async fn run() -> anyhow::Result<()> {
auth_service_client.as_ref().clone(),
sqs_client.as_ref().clone(),
),
- RedisProviderRateLimiter::new(redis_client.clone(), RateBudget::Live),
+ RedisGmailRequestGate::new(
+ redis_client.inner.get_connection_manager().await.context(
+ "failed to get reconnecting redis connection for the gmail request gate",
+ )?,
+ RateBudget::Live,
+ ),
);
let redis_client = Arc::new(redis_client);
let gmail_token_state = GmailTokenState::new(GmailTokenProviderImpl::new(
diff --git a/services/email_service/src/util/redis.rs b/services/email_service/src/util/redis.rs
index c2230ded4a..299e4f37f7 100644
--- a/services/email_service/src/util/redis.rs
+++ b/services/email_service/src/util/redis.rs
@@ -8,31 +8,14 @@ pub mod rate_limit;
pub struct RedisClient {
pub inner: redis::Client,
- /// Maximum allowed Gmail API quota units within the rate limiting window, for normal operations.
- /// Gmail API per-user quota is 15,000 units per minute.
- pub rate_limit_units: u32,
-
- /// Maximum allowed Gmail API quota units within the rate limiting window, for backfill. Less
- /// than redis_rate_limit_reqs so we have room for normal gmail operations while backfilling
- /// is occurring
- pub rate_limit_units_backfill: u32,
-
- /// Duration of the sliding window for Gmail API rate limiting in seconds.
- /// Uses a 60-second window to align with Gmail's per-minute quota period.
+ /// Duration of the sliding window for Google Calendar rate limiting in seconds.
pub rate_limit_secs: u32,
}
impl RedisClient {
- pub fn new(
- inner: redis::Client,
- rate_limit_units: u32,
- rate_limit_units_backfill: u32,
- rate_limit_secs: u32,
- ) -> Self {
+ pub fn new(inner: redis::Client, rate_limit_secs: u32) -> Self {
Self {
inner,
- rate_limit_units,
- rate_limit_units_backfill,
rate_limit_secs,
}
}
diff --git a/services/email_service/src/util/redis/rate_limit.rs b/services/email_service/src/util/redis/rate_limit.rs
index 47ccf044ff..fcee7aacd1 100644
--- a/services/email_service/src/util/redis/rate_limit.rs
+++ b/services/email_service/src/util/redis/rate_limit.rs
@@ -1,100 +1,16 @@
use crate::util::redis::RedisClient;
-use models_email::gmail::operations::GmailApiOperation;
use redis::Script;
use std::time::{SystemTime, UNIX_EPOCH};
use uuid::Uuid;
-pub struct RateLimitArgs {
- pub user_id: Uuid,
- pub operation: GmailApiOperation,
- pub is_backfill: bool,
-}
-
-impl RedisClient {
- /// Checks if a Gmail API operation is rate-limited and returns the status.
- /// Uses a 60-second sliding window implemented via a Lua script for efficiency and atomicity.
- /// Using a Lua script provides several benefits: simpler logic compared to raw Redis commands, connection pooling
- /// via async multiplexed connections rather than dedicated connections, reduced network round trips,
- /// and guaranteed atomic execution. If anything goes wrong, return false and hope for the best
- ///
- /// # Arguments
- /// * `user_id` - UUID of the user to check rate limiting for
- /// * `operation` - The Gmail API operation to check rate limiting for
- ///
- /// # Returns
- /// Returns `true` if the operation should be rate limited (blocked), `false` otherwise.
- ///
- pub async fn is_rate_limited(&self, args: RateLimitArgs) -> bool {
- let RateLimitArgs {
- user_id,
- operation,
- is_backfill,
- } = args;
- // If we're backfilling, use a lower rate limit than if we are not. This is to allow some room
- // for normal inbox operations while backfill is occurring.
- let rate_limit_units = if is_backfill {
- self.rate_limit_units_backfill
- } else {
- self.rate_limit_units
- };
-
- let cost = operation.cost();
- let mut con = match self.inner.get_multiplexed_async_connection().await {
- Ok(conn) => conn,
- Err(e) => {
- tracing::warn!(
- "Failed to get Redis connection for rate limiting user_id {}: {}",
- user_id,
- e
- );
- return false;
- }
- };
-
- let lua_script = get_rate_limit_script_with_usage();
-
- let redis_key = format!("gmail-ratelimit:log:{}", user_id);
- let now_micros = SystemTime::now()
- .duration_since(UNIX_EPOCH)
- .expect("Time went backwards")
- .as_micros() as u64;
- let member_id = format!("{}:{}", cost, Uuid::new_v4());
-
- let script = Script::new(lua_script);
-
- let (is_limited, _current_units): (i32, u32) = match script
- .key(&redis_key)
- .arg(rate_limit_units)
- .arg(self.rate_limit_secs * 1_000_000)
- .arg(now_micros)
- .arg(cost)
- .arg(&member_id)
- .invoke_async(&mut con)
- .await
- {
- Ok(res) => res,
- Err(e) => {
- tracing::warn!(
- "Failed to execute rate limit script for user_id {}: {}",
- user_id.to_string(),
- e
- );
- return false;
- }
- };
-
- is_limited == 1
- }
-}
-
/// Google Calendar's per-user quota is 600 queries per minute; cap our
/// background sync at half of it so interactive calendar features keep room.
const CALENDAR_RATE_LIMIT_QUERIES: u32 = 300;
impl RedisClient {
- /// Checks if a Google Calendar API request is rate-limited, using the
- /// same atomic sliding-window Lua script as the Gmail limiter but with a
- /// calendar-specific key and Calendar's flat one-query-per-request cost.
+ /// Checks if a Google Calendar API request is rate-limited, using an
+ /// atomic sliding-window Lua script with a calendar-specific key and
+ /// Calendar's flat one-query-per-request cost.
pub async fn is_calendar_rate_limited(&self, email_link_id: Uuid) -> bool {
let mut con = match self.inner.get_multiplexed_async_connection().await {
Ok(conn) => conn,
diff --git a/services/email_service/src/util/upload_attachment.rs b/services/email_service/src/util/upload_attachment.rs
index 593909855e..5e6d602c70 100644
--- a/services/email_service/src/util/upload_attachment.rs
+++ b/services/email_service/src/util/upload_attachment.rs
@@ -28,7 +28,7 @@ mod test;
#[derive(Error, Debug)]
pub enum UploadAttachmentError {
#[error("Gmail API rate limit exceeded")]
- RateLimited,
+ RateLimited(EmailApiError),
#[error("Failed to fetch attachment data from Gmail: {0}")]
GmailFetchFailed(EmailApiError),
@@ -213,7 +213,7 @@ async fn fetch_gmail_attachment_data(
.get_attachment(link.id, &p.email_provider_id, &p.provider_attachment_id)
.await
.map_err(|error| match error {
- EmailApiError::RateLimited { .. } => UploadAttachmentError::RateLimited,
+ error @ EmailApiError::RateLimited { .. } => UploadAttachmentError::RateLimited(error),
// Preserve the provider error so API handlers can map it to the
// right status instead of collapsing everything to 500.
error => UploadAttachmentError::GmailFetchFailed(error),