Skip to content

:hedgehog: PostHog is the leading platform for building self-driving products. Our developer tools – AI observability, analytics, session replay, flags, experiments, error tracking, logs, and more – capture all the context agents need to diagnose problems, uncover opportunities, and ship fixes. Steer it all from Slack, web, desktop, or the MCP.

fix(workflows): send transactional email past the team sending cap (#127570)

Meikel Ratzcd7bda8b2079
diff --git a/docs/internal/workflows-email-sending-tiers.md b/docs/internal/workflows-email-sending-tiers.md
index 1fe69735e08b..bf9c3d7915cc 100644
--- a/docs/internal/workflows-email-sending-tiers.md
+++ b/docs/internal/workflows-email-sending-tiers.md
@@ -4,6 +4,7 @@ Every project has a trust tier that caps how much workflow email it can send: an
Projects start at the lowest tier and earn higher ones by sending cleanly; dirty rates, AWS tenant reputation findings, suspensions, and long dormancy move them down.
The tiers exist because all workflow email shares one SES account whose reputation pools every project's complaints.
Only workflows that send email are subject to the tiers; SMS, push, and webhook activity is never capped by them.
+Transactional email is never delayed by the send-time caps, but it is still charged against them (see Transactional email below).
## Where the pieces live
@@ -52,6 +53,30 @@ Pinned teams never move automatically.
While the tier is enforced, the sweep notifies teams it moved: a rate demotion sends an in-app notification plus an email to project admins, and an earned promotion sends an in-app notification only.
Decay, suspension drops, admin recomputes, and the backfill stay silent.
+## Transactional email
+
+An email step categorized as transactional is never delayed by the send-time buckets, in any mode.
+A password reset or a receipt that arrives two hours late has already failed the person waiting for it, so pacing it achieves nothing that dropping it would not.
+This matches the billing quota, which transactional email already bypasses.
+
+The send still drains both buckets by what it costs, so transactional volume displaces the project's marketing while tokens remain.
+A project that spends its whole allowance on transactional email has nothing left for marketing, which is the intended trade.
+
+The charge does not bound the exemption.
+The buckets record no debt, so once a project empties them its further transactional sends cost nothing and this cap stops pacing it.
+A project that labels all of its email transactional is therefore not paced per team at all.
+No other control bounds that volume per project either.
+The SES account rate limiter, which every send still passes, paces the whole account rather than one project.
+What is left is suspension, and the daily sweep demoting the project on its complaint and bounce rates after the fact.
+Making the cap itself bound the case needs overdraw in the shared token bucket, which is a change to `RateLimiterService` and is not done here.
+
+The category is self-declared in the email step, same as everywhere else it is read.
+That is not a new trust boundary: the same field already exempts a send from recipient opt-outs and from the billing quota, both larger levers than pacing.
+
+`cdp_team_email_cap_delayed_total{mode="transactional"}` counts transactional sends that a cap would have delayed.
+They are kept out of `mode="shadow"` so the shadow counts stay a true forecast of what enforce would delay.
+A project appearing there often is spending more on transactional than its tier allows, which is the signal to check the categorization or raise the tier.
+
## Rollout order
1. Merge and deploy with both modes `off`. The daily sweep starts computing and storing tiers immediately.
@@ -66,6 +91,7 @@ Decay, suspension drops, admin recomputes, and the backfill stay silent.
- The batch audience cap is decided when the batch is dispatched. Adding an email step to the workflow while a batch is queued does not re-cap it; the send-time buckets still cap every email at execution. This is why enforcement requires the worker caps to be deployed (see the rollout order).
- Test-panel sends bypass the team buckets on purpose, matching the per-workflow rate limit.
- Rescheduled emails keep their email queue priority while waiting. After a team-cap, workflow-pacing, or SES-throttle retry, the workflow restores its origin queue and priority when it next leaves the email queue. A push notification retried while the workflow is on the email queue behaves the same way and waits at the priority of its own message category.
+- Transactional sends are charged but never delayed, so a team whose transactional volume alone exceeds its tier can hold its own marketing at zero (see Transactional email below).
- The buckets are token buckets: a full idle bucket plus refill allows up to roughly twice the stated cap in the very first period. The bucket TTLs exceed the refill periods so this does not recur from idling.
- A denied send parks until every short bucket has refilled enough to cover it, instead of retrying on a fixed few-minute cadence.
The computed wait is capped at one hour and then jittered 1x to 2x, so a parked send can wait just under two hours between attempts.
diff --git a/nodejs/src/cdp/services/messaging/email.service.test.ts b/nodejs/src/cdp/services/messaging/email.service.test.ts
index 82decc3b4b5e..d432e9d7b16b 100644
--- a/nodejs/src/cdp/services/messaging/email.service.test.ts
+++ b/nodejs/src/cdp/services/messaging/email.service.test.ts
@@ -916,11 +916,13 @@ describe('EmailService', () => {
describe('team sending cap (enforce mode)', () => {
let claimAllOrNothingPair: jest.Mock
+ let claimUpTo: jest.Mock
let cappedService: EmailService
let cappedSendSpy: jest.SpyInstance
beforeEach(() => {
claimAllOrNothingPair = jest.fn()
+ claimUpTo = jest.fn().mockResolvedValue(0)
const configService = new TeamWorkflowsConfigService(hub.postgres, hub.pubSub)
jest.spyOn(configService, 'getEmailSendingTier').mockResolvedValue(0)
cappedService = new EmailService(
@@ -946,7 +948,7 @@ describe('EmailService', () => {
{ report: jest.fn() },
undefined,
null,
- { claimAllOrNothingPair } as unknown as RateLimiterService
+ { claimAllOrNothingPair, claimUpTo } as unknown as RateLimiterService
)
cappedSendSpy = jest.spyOn(cappedService.sesV2Client!, 'send') as any
cappedSendSpy.mockResolvedValue({ MessageId: 'test-message-id' })
@@ -956,6 +958,54 @@ describe('EmailService', () => {
jest.restoreAllMocks()
})
+ const buildEnforcedService = (
+ name: string,
+ hourlyCap: number,
+ dailyCap: number
+ ): { service: EmailService; sendSpy: jest.SpyInstance; limiter: RateLimiterService } => {
+ const redis = createRedisV2PoolFromConfig({
+ connection: hub.CDP_REDIS_HOST
+ ? {
+ url: hub.CDP_REDIS_HOST,
+ options: { port: hub.CDP_REDIS_PORT, password: hub.CDP_REDIS_PASSWORD },
+ }
+ : { url: hub.REDIS_URL },
+ poolMinSize: hub.REDIS_POOL_MIN_SIZE,
+ poolMaxSize: hub.REDIS_POOL_MAX_SIZE,
+ })
+ const limiter = new RateLimiterService(redis, { name })
+ const configService = new TeamWorkflowsConfigService(hub.postgres, hub.pubSub)
+ jest.spyOn(configService, 'getEmailSendingTier').mockResolvedValue(0)
+ const service = new EmailService(
+ {
+ sesAccessKeyId: hub.SES_ACCESS_KEY_ID,
+ sesSecretAccessKey: hub.SES_SECRET_ACCESS_KEY,
+ sesRegion: hub.SES_REGION,
+ sesEndpoint: hub.SES_ENDPOINT,
+ sesTrackedConfigurationSet: hub.SES_TRACKED_CONFIGURATION_SET,
+ sesUntrackedConfigurationSet: hub.SES_UNTRACKED_CONFIGURATION_SET,
+ teamEmailCapMode: 'enforce',
+ teamEmailTierHourlyCaps: [hourlyCap],
+ teamEmailTierDailyCaps: [dailyCap],
+ },
+ hub.integrationManager,
+ configService,
+ hub.ENCRYPTION_SALT_KEYS,
+ hub.SITE_URL,
+ new EmailTrackingCodeSigner(hub.ENCRYPTION_SALT_KEYS, hub.CDP_EMAIL_TRACKING_URL),
+ new EmailSuppressionService(hub.postgres, emailSuppressionConfigFromEnv()),
+ new RecipientsManagerService(hub.postgres),
+ null,
+ { report: jest.fn() },
+ undefined,
+ null,
+ limiter
+ )
+ const sendSpy = jest.spyOn(service.sesV2Client!, 'send') as any
+ sendSpy.mockResolvedValue({ MessageId: 'test-message-id' })
+ return { service, sendSpy, limiter }
+ }
+
it.each([
// A reserved slot is exclusive and is parked on exactly; spreading it would
// collide it with the slot in front.
@@ -1015,6 +1065,26 @@ describe('EmailService', () => {
expect(messages).toContain("Could not check this project's email sending limit")
})
+ it('sends a transactional email past the cap and charges the buckets anyway', async () => {
+ claimAllOrNothingPair.mockResolvedValue({
+ granted: false,
+ deniedIndex: 1,
+ retryAfterMs: 30 * 60 * 1000,
+ reserved: true,
+ })
+ invocation.hogFunction.metadata = { message_category_type: 'transactional' }
+
+ const result = await cappedService.executeSendEmail(invocation)
+
+ expect(result.finished).toBe(true)
+ expect(cappedSendSpy).toHaveBeenCalledTimes(1)
+ expect(result.invocation.queueScheduledAt).toBeUndefined()
+ expect(claimAllOrNothingPair).not.toHaveBeenCalled()
+ expect(claimUpTo).toHaveBeenCalledTimes(2)
+ // No cap was applied, so the customer must not be told one was.
+ expect(result.logs.map((log) => log.message).join(' ')).not.toContain('reached its email sending limit')
+ })
+
it('sends when the claim is granted', async () => {
claimAllOrNothingPair.mockResolvedValue({
granted: true,
@@ -1042,46 +1112,11 @@ describe('EmailService', () => {
it('spreads a capped team over distinct slots while another team keeps sending', async () => {
const hourlyCap = 360
const dailyCap = 8640
- const redis = createRedisV2PoolFromConfig({
- connection: hub.CDP_REDIS_HOST
- ? {
- url: hub.CDP_REDIS_HOST,
- options: { port: hub.CDP_REDIS_PORT, password: hub.CDP_REDIS_PASSWORD },
- }
- : { url: hub.REDIS_URL },
- poolMinSize: hub.REDIS_POOL_MIN_SIZE,
- poolMaxSize: hub.REDIS_POOL_MAX_SIZE,
- })
- const limiter = new RateLimiterService(redis, { name: 'team-email-cap-test' })
- const configService = new TeamWorkflowsConfigService(hub.postgres, hub.pubSub)
- jest.spyOn(configService, 'getEmailSendingTier').mockResolvedValue(0)
- const enforcedService = new EmailService(
- {
- sesAccessKeyId: hub.SES_ACCESS_KEY_ID,
- sesSecretAccessKey: hub.SES_SECRET_ACCESS_KEY,
- sesRegion: hub.SES_REGION,
- sesEndpoint: hub.SES_ENDPOINT,
- sesTrackedConfigurationSet: hub.SES_TRACKED_CONFIGURATION_SET,
- sesUntrackedConfigurationSet: hub.SES_UNTRACKED_CONFIGURATION_SET,
- teamEmailCapMode: 'enforce',
- teamEmailTierHourlyCaps: [hourlyCap],
- teamEmailTierDailyCaps: [dailyCap],
- },
- hub.integrationManager,
- configService,
- hub.ENCRYPTION_SALT_KEYS,
- hub.SITE_URL,
- new EmailTrackingCodeSigner(hub.ENCRYPTION_SALT_KEYS, hub.CDP_EMAIL_TRACKING_URL),
- new EmailSuppressionService(hub.postgres, emailSuppressionConfigFromEnv()),
- new RecipientsManagerService(hub.postgres),
- null,
- { report: jest.fn() },
- undefined,
- null,
- limiter
- )
- const enforcedSendSpy = jest.spyOn(enforcedService.sesV2Client!, 'send') as any
- enforcedSendSpy.mockResolvedValue({ MessageId: 'test-message-id' })
+ const {
+ service: enforcedService,
+ sendSpy: enforcedSendSpy,
+ limiter,
+ } = buildEnforcedService('team-email-cap-test', hourlyCap, dailyCap)
// Drain the capped team's hourly bucket so every send below is a denial. The
// 0.1 tokens/s refill makes the slot spacing 10s, far above test wall-clock.
@@ -1130,6 +1165,72 @@ describe('EmailService', () => {
expect(otherResult.finished).toBe(true)
expect(enforcedSendSpy).toHaveBeenCalledTimes(1)
})
+
+ // Enforce mode has only ever issued claimAllOrNothingPair against real Valkey, so
+ // transactional through the single-key claimUpTo has never run there. Real limiter.
+ it('charges a transactional send against the real buckets without delaying it', async () => {
+ const hourlyCap = 60
+ const dailyCap = 50
+ const {
+ service: enforcedService,
+ sendSpy: enforcedSendSpy,
+ limiter,
+ } = buildEnforcedService('team-email-cap-transactional-test', hourlyCap, dailyCap)
+
+ // A team of its own, so the buckets start cold whatever else touched Valkey.
+ const capTeam = (await createTestTeamFixture(hub.postgres)).team
+ await insertIntegration(hub.postgres, capTeam.id, {
+ id: capTeam.id + 1,
+ kind: 'email',
+ config: {
+ email: '[email protected]',
+ name: 'Test User',
+ domain: 'posthog.com',
+ verified: true,
+ provider: 'ses',
+ },
+ })
+ const sendFor = (id: string, transactional: boolean): CyclotronJobInvocationHogFunction => {
+ const next = createExampleInvocation({ team_id: capTeam.id, id: `function-${id}` })
+ next.id = `invocation-${id}`
+ next.state.vmState = { stack: [] } as any
+ next.queueParameters = createEmailParams()
+ next.queueParameters.from = { integrationId: capTeam.id + 1 }
+ if (transactional) {
+ next.hogFunction.metadata = { message_category_type: 'transactional' }
+ }
+ return next
+ }
+
+ // Empty the hourly bucket only, so the daily one can show the charge.
+ const buckets = teamEmailCapBuckets(capTeam.id, hourlyCap, dailyCap)
+ await limiter.claimUpTo({
+ key: buckets[0].key,
+ requested: hourlyCap,
+ capacity: buckets[0].capacity,
+ refillPerSecond: buckets[0].refillPerSecond,
+ })
+
+ // Control: the cap really is reached, so a marketing send parks.
+ const marketing = await enforcedService.executeSendEmail(sendFor('marketing', false))
+ expect(marketing.finished).toBe(false)
+ expect(enforcedSendSpy).not.toHaveBeenCalled()
+
+ const transactional = await enforcedService.executeSendEmail(sendFor('transactional', true))
+ expect(transactional.finished).toBe(true)
+ expect(transactional.invocation.queueScheduledAt).toBeUndefined()
+ expect(enforcedSendSpy).toHaveBeenCalledTimes(1)
+
+ // A range, not an exact count, because the daily bucket refills as the test runs.
+ const dailyLeft = await limiter.claimUpTo({
+ key: buckets[1].key,
+ requested: dailyCap,
+ capacity: buckets[1].capacity,
+ refillPerSecond: buckets[1].refillPerSecond,
+ })
+ expect(dailyLeft).toBeLessThan(dailyCap)
+ expect(dailyLeft).toBeGreaterThanOrEqual(dailyCap - 2)
+ })
})
})
describe('native email sending with maildev', () => {
diff --git a/nodejs/src/cdp/services/messaging/email.service.ts b/nodejs/src/cdp/services/messaging/email.service.ts
index 768117da547b..c6b38a090ddc 100644
--- a/nodejs/src/cdp/services/messaging/email.service.ts
+++ b/nodejs/src/cdp/services/messaging/email.service.ts
@@ -264,7 +264,7 @@ function pickReservedRetryDelayMs(retryAfterMs: number | null, refillPerSecond:
const teamEmailCapDelayedTotal = new Counter({
name: 'cdp_team_email_cap_delayed_total',
- help: 'Workflow email sends delayed by the team trust-tier sending cap (or that would have been, in shadow mode).',
+ help: 'Workflow email sends delayed by the team trust-tier sending cap, or that would have been. `mode=shadow` and `mode=transactional` both mean the send went out anyway.',
labelNames: ['tier', 'bucket', 'mode'],
})
@@ -716,6 +716,7 @@ export class EmailService {
// Project-wide pacing from the team's trust tier. Sits next to the per-workflow limit
// above and behaves the same way: it delays, never drops, and test sends bypass it.
+ // Transactional sends bypass the delay too, and are counted instead.
// Charged per recipient, not per send: SES counts every to/cc/bcc address against its
// own quota, so a send with many copies must spend that many tokens.
const capRecipients =
@@ -921,6 +922,13 @@ export class EmailService {
* the limiter, which delays the send, because we must not send when we cannot account for it.
* That second case returns a null `label`: the send waits, but no cap was reached, so the
* caller must not tell the customer it hit one.
+ *
+ * Transactional sends are charged but never delayed, because a password reset that arrives
+ * two hours late has already failed the person waiting on it. The charge does not bound the
+ * exemption: claimUpTo records no debt, so once a bucket empties further transactional sends
+ * cost nothing. No other control bounds that volume per team either. The SES limiter paces
+ * the whole account, so what is left is suspension and the daily tier job, which demotes a
+ * team on its complaint and bounce rates after the fact.
*/
private async claimTeamSendingBudget(
invocation: CyclotronJobInvocationHogFunction,
@@ -955,7 +963,9 @@ export class EmailService {
// it still waits for full capacity and pays everything there is.
const requested = Math.min(Math.max(1, recipients), buckets[0].capacity, buckets[1].capacity)
- if (mode === 'enforce') {
+ const isTransactional = invocation.hogFunction?.metadata?.message_category_type === 'transactional'
+
+ if (mode === 'enforce' && !isTransactional) {
// Both buckets in one atomic claim, granted whole or not at all. A denial consumes
// nothing, so a rescheduled multi-recipient send cannot burn the partial refill on
// every retry and starve the team's other emails while never sending itself.
@@ -987,8 +997,9 @@ export class EmailService {
}
}
- // Shadow mode: the send goes out regardless, so drain each bucket by what the send costs
- // and log the first cap that would have delayed it, measuring both against real traffic.
+ // Shadow mode, and transactional sends in any mode: the send goes out regardless, so
+ // drain each bucket and log the cap. Separate labels keep shadow a forecast of enforce.
+ const observedMode = isTransactional ? 'transactional' : mode
let firstDenial: TeamEmailCapBucket | null = null
for (const bucket of buckets) {
const granted = await this.teamEmailRateLimiter.claimUpTo({
@@ -1001,17 +1012,23 @@ export class EmailService {
if (granted >= requested) {
continue
}
- teamEmailCapDelayedTotal.inc({ tier: String(tier), bucket: bucket.name, mode })
+ teamEmailCapDelayedTotal.inc({ tier: String(tier), bucket: bucket.name, mode: observedMode })
firstDenial = firstDenial ?? bucket
}
if (firstDenial) {
- logger.info('📧', 'Team email sending cap would have delayed this send', {
- teamId: invocation.teamId,
- tier,
- bucket: firstDenial.name,
- cap: firstDenial.label,
- })
+ logger.info(
+ '📧',
+ isTransactional
+ ? 'Transactional email sent past the team sending cap'
+ : 'Team email sending cap would have delayed this send',
+ {
+ teamId: invocation.teamId,
+ tier,
+ bucket: firstDenial.name,
+ cap: firstDenial.label,
+ }
+ )
}
return null
}