Autonomous AI sales agent powered by Claude Code or Codex. Finds prospects, writes cold emails, sends campaigns, handles replies, books meetings. No API keys needed — runs on your Claude or Codex subscriptions.
Kill the Instantly dependency: native mail providers + outbox approval ladder
Ethan1a7c4760bbef
diff --git a/.env.example b/.env.exampleindex f539e3b..cd00ed3 100644--- a/.env.example+++ b/.env.example@@ -2,14 +2,37 @@# Harvey environment variables# Copy this file to .env and fill in your values: cp .env.example .env# Never commit .env — it is gitignored.-# Only INSTANTLY_API_KEY is required. Everything else is optional.+# Pick ONE email provider below (set channels.email.provider in harvey.yaml+# to match): gmail (recommended), smtp, or instantly. Everything else is+# optional but an email verifier (Reoon/ZeroBounce) is strongly recommended.# ============================================================# -------------------------------------------------------------# Instantly — cold email sending platform [REQUIRED]+# EMAIL PROVIDER — how Harvey sends and reads replies. Choose one.+#+# Option A: Gmail / Google Workspace [RECOMMENDED for <50 cold/day]+# Set provider: gmail in harvey.yaml. Use a mailbox on a DEDICATED+# secondary domain (never your main one), with SPF/DKIM/DMARC set up.+# 1. Google Cloud Console → new project → enable the Gmail API+# 2. Credentials → OAuth client ID → type "Desktop app"+# 3. Paste the client id/secret below, then run: harvey gmail auth+GMAIL_CLIENT_ID=+GMAIL_CLIENT_SECRET=++# Option B: Any SMTP + IMAP mailbox [AgentMail, Fastmail, Workspace app pw]+# Set provider: smtp in harvey.yaml. IMAP_* defaults to the SMTP_* values.+SMTP_HOST=+SMTP_PORT=587+SMTP_USERNAME=+SMTP_PASSWORD=+IMAP_HOST=+IMAP_PORT=993+IMAP_USERNAME=+IMAP_PASSWORD=++# Option C: Instantly — cold email platform [legacy]# Where to get it: https://app.instantly.ai → Settings → Integrations → API Keys-# Note: API access requires Instantly's Growth plan or higher.-# Used by: Sender agent (deploy campaigns), Handler agent (read replies).+# API access requires Instantly's Growth plan or higher.INSTANTLY_API_KEY=# ------------------------------------------------------------diff --git a/CLAUDE.md b/CLAUDE.mdindex ac384be..9c49076 100644--- a/CLAUDE.md+++ b/CLAUDE.md@@ -67,21 +67,34 @@ python -m playwright install chromium### Step 2: API Keys (`.env`)-Only Instantly is required. Ask for it first, then mention the optional ones:+Harvey needs an **email provider** (to send + read replies) and, strongly recommended, an **email verifier**. Pick one provider:```-INSTANTLY_API_KEY= # Required — from Instantly Settings → Integrations-LINKEDIN_EMAIL= # Optional — for LinkedIn prospecting-LINKEDIN_PASSWORD= # Optional — for LinkedIn prospecting-CLOUDFLARE_ACCOUNT_ID= # Optional — for deep JS-rendered website crawling-CLOUDFLARE_API_TOKEN= # Optional — for deep JS-rendered website crawling-REOON_API_KEY= # Optional — email verification (600 free/mo; best free tier)-ZEROBOUNCE_API_KEY= # Optional — email verification (100 free/mo; best for M365/Workspace catch-alls)-HUNTER_API_KEY= # Optional — email pattern lookup + verification (50 free/mo)-SERPER_API_KEY= # Optional — for reliable web search ($5/mo at serper.dev)+# --- Email provider: choose ONE, set channels.email.provider to match ---+GMAIL_CLIENT_ID= # Recommended: Gmail/Workspace. Then run 'harvey gmail auth'+GMAIL_CLIENT_SECRET=+# --- or SMTP+IMAP (AgentMail, Fastmail, Workspace app password) ---+SMTP_HOST= SMTP_PORT=587 SMTP_USERNAME= SMTP_PASSWORD=+IMAP_HOST= IMAP_PORT=993+# --- or legacy Instantly ---+INSTANTLY_API_KEY=++# --- Email verification (add at least one; else emails stay 'guess') ---+REOON_API_KEY= # 600 free/mo — best free tier+ZEROBOUNCE_API_KEY= # 100 free/mo — best for M365/Workspace catch-alls+HUNTER_API_KEY= # 50 free/mo + pattern lookup++# --- Optional ---+LINKEDIN_EMAIL= LINKEDIN_PASSWORD= # LinkedIn prospecting+CLOUDFLARE_ACCOUNT_ID= CLOUDFLARE_API_TOKEN= # deep JS crawling for training+SERPER_API_KEY= # reliable web search ($5/mo at serper.dev)```-**Email verification matters:** Harvey learns each company's email *pattern* and verifies ONE candidate rather than guessing (raw SMTP probing no longer works against Google Workspace / Microsoft 365). Add any one of Reoon / ZeroBounce / Hunter to get verified addresses. Without a verifier key, found emails are marked `guess` and are **never sent** — so at least one key is strongly recommended before running campaigns.+**Recommended provider — Gmail:** for <50 cold emails/day, a real Google Workspace mailbox on a *dedicated secondary domain* (never the main one) is the most deliverable, cheapest (~$7/mo) option. Set `channels.email.provider: gmail` in harvey.yaml, put the OAuth client id/secret in `.env`, then run `harvey gmail auth` (one-time browser login). SMTP works with any mailbox (AgentMail, Fastmail). Instantly still works as a legacy option.++**Email verification matters:** Harvey learns each company's email *pattern* and verifies ONE candidate rather than guessing (raw SMTP probing no longer works against Google Workspace / Microsoft 365). Without a verifier key, found emails are marked `guess` and are **never sent**.++**Approval by default:** with a native provider, every outgoing email waits in the **Outbox** for your approval (dashboard Outbox tab, or `harvey outbox`). Once you trust the output, set `channels.email.require_approval: false` in harvey.yaml for full autopilot.After getting the Instantly API key, test it:```bash@@ -169,8 +182,8 @@ Users will come back with questions and tasks. Common ones:### Sub-Agents- **Scout**: Python does all web searching (DuckDuckGo → Bing → Google → Serper API) and email resolution (pattern-first: cache → scraped mailto → Hunter domain search → default, then verify one candidate via Reoon/ZeroBounce/Hunter/SMTP; catch-alls flagged `risky`). Claude only scores/personalizes found data. Scout also collects **buying signals**: tech stack detected on each company's site (HubSpot, Shopify, Intercom, ~35 tools — zero extra requests) and hiring signals from careers pages. With `pip install python-jobspy` (optional), a job-board strategy discovers companies actively hiring for roles in `icp.hiring_signals` (falls back to `icp.titles`) — the strongest in-market signal. Signals land in prospects' personalization notes and boost their score.- **Writer**: Generates 3-email sequences (Email 1 < 75 words, Email 2 < 75, Email 3 < 40). Strict ban list on AI patterns.-- **Sender**: Deploys to Instantly API. Enforces daily send limits.-- **Handler**: Classifies reply intent, advances conversation stage, auto-responds. Has reply deduplication.+- **Sender**: Native providers (gmail/smtp) render merge vars per prospect and STAGE each email into the `outbox` (pending_review → approved → sent) with a scheduled send time; each heartbeat drains due, approved items with human-like pacing, the daily cap, the deterministic pre-send gate, and stop-on-reply. Legacy Instantly path deploys campaigns via API. Enforces daily send limits either way.+- **Handler**: Polls the provider (or Instantly) for replies, dedups them, classifies intent, advances conversation stage, and queues auto-responses through the same outbox approval ladder. On native providers it also detects bounces → marks the address invalid, cancels the prospect's queued sends, and trips a global kill switch past a bounce-rate threshold.- **Analyst**: Runs on idle cycles. Generates `data/analytics.json` with pipeline stats and insights.### Conversation Stages@@ -191,6 +204,10 @@ harvey usage # Claude quota gauges + per-agent token/cost reportharvey usage --reconcile # Also backfill usage from Claude Code transcriptsharvey export # Deliverable prospects → sequencer-ready CSV (prospects.csv)harvey export --all # Full raw list, no filters+harvey gmail auth # One-time Gmail OAuth (when provider: gmail)+harvey gmail test # Verify the Gmail connection+harvey outbox # Review queued emails; --approve <id> / --approve-all / --reject <id>+harvey sending pause|resume # Kill switch for all outbound```### Common Issuesdiff --git a/harvey.yaml b/harvey.yamlindex ad7b488..7003026 100644--- a/harvey.yaml+++ b/harvey.yaml@@ -47,8 +47,11 @@ icp:channels:email:enabled: true- provider: "instantly"+ provider: "instantly" # gmail (recommended) | smtp | instantlymax_daily_sends: 50+ require_approval: true # native providers: emails wait in the Outbox+ send_to_risky: false # also send to catch-all domains (off = safest)+ max_bounce_rate: 0.05 # pause all sending if bounces exceed thislinkedin:enabled: truemax_daily_connections: 20diff --git a/harvey/agents/handler.py b/harvey/agents/handler.pyindex b23865f..6377aa7 100644--- a/harvey/agents/handler.py+++ b/harvey/agents/handler.py@@ -1,16 +1,29 @@-"""Handler — monitors replies and manages conversations."""+"""Handler — monitors replies and manages conversations.++Works against Instantly (legacy) or a native mail provider (Gmail/SMTP).+The native path also owns bounce handling: a bounce marks the address+invalid, cancels the prospect's queued sends, and — past a bounce-rate+threshold — flips the global kill switch so a bad list can't torch the+sending domain while nobody's watching.+"""import logging-from datetime import datetime+from datetime import datetime, timezonefrom harvey.brain import Brainfrom harvey.config import HarveyConfig, EnvConfigfrom harvey.integrations.instantly import InstantlyClient+from harvey.integrations.mail_provider import NATIVE_PROVIDERS, get_mail_providerfrom harvey.models.conversation import Conversation, Messagefrom harvey.state import StateManagerlogger = logging.getLogger("harvey.handler")+KILL_SWITCH_KEY = "sending_paused"+BOUNCE_COUNT_KEY = "bounce_count"+# Kill switch only engages after this many sends (a tiny sample lies).+MIN_SENDS_FOR_KILL_SWITCH = 10+INTENT_LABELS = {"interested","objection",@@ -77,8 +90,16 @@ class Handler:self.state = stateself.config = configself.instantly = InstantlyClient(env.instantly_api_key)+ self.provider = get_mail_provider(config, env)self.skills = ""+ @property+ def is_native(self) -> bool:+ return (+ self.config.channels.email.provider in NATIVE_PROVIDERS+ and self.provider is not None+ )+async def run(self):"""Check for new replies and handle them."""logger.info("Handler: Checking for replies...")@@ -89,6 +110,10 @@ class Handler:if not self.config.channels.email.enabled:return+ if self.is_native:+ await self._run_native()+ return+# 1. Fetch active campaignsactive_campaigns = await self.state.get_campaigns_by_status("active")if not active_campaigns:@@ -142,7 +167,7 @@ class Handler:logger.info(f"Handler: Reply from {lead_email}")try:- await self._process_reply(lead_email, reply_text, reply_uuid, campaign)+ await self._process_reply(lead_email, reply_text, reply_uuid, campaign.id)finally:# ALWAYS mark processed — even on early exits (opt-out, OOO,# unknown prospect) — so the same reply is never re-handled@@ -151,7 +176,14 @@ class Handler:await self.state.mark_reply_processed(reply_uuid)return True- async def _process_reply(self, lead_email: str, reply_text: str, reply_uuid: str, campaign):+ async def _process_reply(+ self,+ lead_email: str,+ reply_text: str,+ reply_uuid: str,+ campaign_id: str = "",+ reply_meta: dict | None = None,+ ):"""Classify, record, and respond to a single reply."""# Find the prospect by email (indexed lookup)prospect = await self.state.get_prospect_by_email(lead_email)@@ -159,8 +191,18 @@ class Handler:logger.warning(f"Handler: No prospect found for {lead_email}")return- # Update prospect status+ # Update prospect status + stop-on-reply: a human answered, so every+ # queued sequence email for them is now wrong to send.await self.state.update_prospect_status(prospect.id, "replied")+ try:+ cancelled = await self.state.cancel_pending_outbox_for_prospect(prospect.id)+ if cancelled:+ logger.info(+ f"Handler: cancelled {cancelled} queued email(s) for "+ f"{lead_email} (they replied)."+ )+ except Exception as e:+ logger.debug(f"Handler: outbox cancel failed: {e}")# 1. Classify intent. Hard keyword checks run FIRST and override# the LLM — opt-outs and legal threats must never be missed.@@ -183,7 +225,7 @@ class Handler:convo = Conversation(id="",prospect_id=prospect.id,- campaign_id=campaign.id,+ campaign_id=campaign_id,channel="email",thread=[Message(sender="prospect", content=reply_text),@@ -257,8 +299,13 @@ class Handler:logger.warning(f"Handler: Could not generate response for {lead_email}")return- # Send via Instantly- if reply_uuid:+ # Send the response — natively through the outbox approval ladder,+ # or immediately via Instantly (legacy).+ if self.is_native:+ await self._queue_native_reply(+ response, prospect, convo, reply_meta or {}, intent+ )+ elif reply_uuid:result = await self.instantly.send_reply(reply_uuid, response)if result is not None:# Add our response to conversation@@ -279,6 +326,132 @@ class Handler:},)+ # ── Native provider path (Gmail / SMTP) ──++ async def _run_native(self):+ """Poll the mailbox, split bounces from human replies, handle both."""+ if not self.provider.is_configured():+ logger.debug(+ f"Handler: provider '{self.provider.name}' not configured yet."+ )+ return++ try:+ inbound = await self.provider.get_replies()+ except Exception as e:+ logger.error(f"Handler: fetching replies failed: {e}")+ return++ handled = 0+ for msg in inbound:+ dedup_key = msg.provider_id or msg.message_id+ if not dedup_key or await self.state.is_reply_processed(dedup_key):+ continue+ try:+ if msg.is_bounce:+ await self._handle_bounce(msg)+ elif msg.body.strip():+ await self._process_reply(+ msg.from_email,+ msg.body.strip(),+ reply_uuid="",+ reply_meta={+ "thread_ref": msg.thread_ref,+ "message_id": msg.message_id,+ "subject": msg.subject,+ },+ )+ handled += 1+ except Exception as e:+ logger.error(f"Handler: error processing {msg.from_email}: {e}")+ finally:+ await self.state.mark_reply_processed(dedup_key)++ if handled:+ logger.info(f"Handler: processed {handled} inbound message(s).")+ else:+ logger.info("Handler: no new replies.")++ async def _handle_bounce(self, msg):+ """A bounce is a data bug AND a reputation threat. Fix both."""+ outbox_item = await self.state.find_outbox_by_message_id(msg.in_reply_to)+ prospect = None+ if outbox_item:+ prospect = await self.state.get_prospect(outbox_item["prospect_id"])++ if prospect:+ await self.state.update_prospect_email(+ prospect.id, prospect.email, "invalid"+ )+ cancelled = await self.state.cancel_pending_outbox_for_prospect(+ prospect.id, reason="bounced"+ )+ logger.warning(+ f"Handler: BOUNCE for {prospect.email} — marked invalid, "+ f"cancelled {cancelled} queued email(s)."+ )+ else:+ logger.warning(+ f"Handler: bounce received ({msg.subject[:60]}) but couldn't "+ "match it to a sent email."+ )++ bounces = await self.state.increment_setting(BOUNCE_COUNT_KEY)+ total_sent = await self.state.count_outbox_sent()+ await self.state.log_action(+ action_type="bounce",+ agent="handler",+ details={"prospect": prospect.email if prospect else "unknown",+ "bounces": bounces, "total_sent": total_sent},+ )++ max_rate = getattr(self.config.channels.email, "max_bounce_rate", 0.05)+ if (+ max_rate > 0+ and total_sent >= MIN_SENDS_FOR_KILL_SWITCH+ and bounces / total_sent > max_rate+ ):+ reason = (+ f"bounce rate {bounces}/{total_sent} exceeded "+ f"{max_rate:.0%} — check list quality before resuming"+ )+ await self.state.set_setting(KILL_SWITCH_KEY, reason)+ logger.error(f"Handler: KILL SWITCH ENGAGED — {reason}")++ async def _queue_native_reply(+ self, response: str, prospect, convo, reply_meta: dict, intent: str+ ):+ """Route Harvey's reply through the outbox (approval ladder applies)."""+ require_approval = getattr(+ self.config.channels.email, "require_approval", True+ )+ subject = reply_meta.get("subject", "")+ if subject and not subject.lower().startswith("re:"):+ subject = f"Re: {subject}"++ item_id = await self.state.add_outbox_item(+ prospect_id=prospect.id,+ conversation_id=convo.id,+ kind="reply",+ to_email=prospect.email,+ subject=subject or "Re: your note",+ body=response,+ send_at=datetime.now(timezone.utc).replace(tzinfo=None).isoformat(),+ status="pending_review" if require_approval else "approved",+ provider=self.provider.name if self.provider else "",+ thread_ref=reply_meta.get("thread_ref", ""),+ in_reply_to=reply_meta.get("message_id", ""),+ )+ if item_id:+ mode = "queued for your approval" if require_approval else "queued to send"+ logger.info(f"Handler: reply to {prospect.email} {mode} ({intent}).")+ await self.state.log_action(+ action_type="reply_queued",+ agent="handler",+ details={"prospect_email": prospect.email, "intent": intent,+ "response_preview": response[:100]},+ )+async def _classify_intent(self, reply_text: str, prospect) -> str:"""Ask the brain to classify the reply's intent."""prompt = f"""Classify this email reply into exactly ONE category:diff --git a/harvey/agents/sender.py b/harvey/agents/sender.pyindex 39634a9..988bc66 100644--- a/harvey/agents/sender.py+++ b/harvey/agents/sender.py@@ -1,16 +1,41 @@-"""Sender — deploys campaigns via Instantly."""-+"""Sender — deploys campaigns via Instantly, or sends natively via the+configured mail provider (Gmail / SMTP) through the outbox approval ladder.++Native flow:+ 1. Draft campaigns are STAGED: merge variables rendered per prospect,+ one outbox row per (prospect, step) with a scheduled send_at.+ Rows start as pending_review (copilot) or approved (autopilot).+ 2. Each heartbeat DRAINS due approved rows: kill-switch check, stop-on-+ reply check, deterministic pre-send gate, then provider.send_email+ with jittered pacing and the daily cap.+"""++import asyncioimport logging+import randomimport re-from datetime import date+from datetime import date, datetime, timedelta, timezonefrom harvey.brain import Brainfrom harvey.config import HarveyConfig, EnvConfig+from harvey.gate import pre_send_checkfrom harvey.integrations.instantly import InstantlyClient+from harvey.integrations.mail_provider import NATIVE_PROVIDERS, get_mail_providerfrom harvey.state import StateManagerlogger = logging.getLogger("harvey.sender")+# Max sends drained per heartbeat cycle — spreads volume through the day+# instead of bursting the daily cap in one minute.+MAX_SENDS_PER_CYCLE = 8+SEND_JITTER_SECONDS = (4, 15)++# Prospect pipeline statuses that mean "stop emailing this person".+STOP_STATUSES = {"replied", "opted_out", "lost", "meeting", "closed"}++KILL_SWITCH_KEY = "sending_paused"+BOUNCE_COUNT_KEY = "bounce_count"+EMAIL_RE = re.compile(r"^[A-Za-z0-9._%+\-]+@[A-Za-z0-9.\-]+\.[A-Za-z]{2,}$")# Only prospects in these statuses may be added as leads. This guarantees@@ -36,15 +61,28 @@ class Sender:self.state = stateself.config = configself.instantly = InstantlyClient(env.instantly_api_key)+ self.provider = get_mail_provider(config, env)+ # Disabled in tests to skip inter-send sleeps.+ self.send_pacing = True++ @property+ def is_native(self) -> bool:+ return (+ self.config.channels.email.provider in NATIVE_PROVIDERS+ and self.provider is not None+ )async def run(self):- """Deploy draft campaigns to Instantly."""- logger.info("Sender: Checking for campaigns to deploy...")-+ """Deploy campaigns — natively through the outbox, or via Instantly."""if not self.config.channels.email.enabled:logger.info("Sender: Email channel disabled. Skipping.")return+ if self.is_native:+ await self._run_native()+ return++ logger.info("Sender: Checking for campaigns to deploy...")draft_campaigns = await self.state.get_campaigns_by_status("draft")if not draft_campaigns:logger.info("Sender: No draft campaigns to deploy.")@@ -272,3 +310,223 @@ class Sender:) as cursor:row = await cursor.fetchone()return row[0] if row else 0++ # ── Native provider flow (Gmail / SMTP via the outbox) ──++ async def _run_native(self):+ """Stage new campaigns into the outbox, then drain what's due."""+ paused = await self.state.get_setting(KILL_SWITCH_KEY)+ if paused:+ logger.warning(+ f"Sender: SENDING PAUSED ({paused}). Resume from the dashboard "+ "or `harvey sending resume` once the cause is fixed."+ )+ return++ for campaign in await self.state.get_campaigns_by_status("draft"):+ try:+ await self._stage_campaign_native(campaign)+ except Exception as e:+ logger.error(f"Sender: staging '{campaign.name}' failed: {e}")++ await self._drain_due()++ def _render(self, text: str, prospect) -> str:+ """Fill merge variables. The pre-send gate rejects any leftovers."""+ replacements = {+ "first_name": prospect.first_name,+ "last_name": prospect.last_name,+ "company": prospect.company,+ "title": prospect.title,+ "personalization": prospect.personalization_notes,+ }+ for key, value in replacements.items():+ text = text.replace("{{" + key + "}}", value or "")+ text = text.replace("{{ " + key + " }}", value or "")+ return text.strip()++ async def _stage_campaign_native(self, campaign):+ """Render + schedule a draft campaign's emails into the outbox."""+ if not self._validate_sequence(campaign):+ await self.state.update_campaign(campaign.id, status="failed")+ return++ require_approval = getattr(+ self.config.channels.email, "require_approval", True+ )+ initial_status = "pending_review" if require_approval else "approved"+ allow_risky = getattr(self.config.channels.email, "send_to_risky", False)+ sendable_email = (+ SENDABLE_EMAIL_STATUSES_WITH_RISKY if allow_risky+ else SENDABLE_EMAIL_STATUSES+ )++ staged = 0+ skipped = 0+ now = datetime.now(timezone.utc).replace(tzinfo=None)+ seen_emails: set[str] = set()++ for prospect_id in campaign.prospect_ids:+ prospect = await self.state.get_prospect(prospect_id)+ if not prospect or not prospect.email:+ continue+ email = prospect.email.strip().lower()+ if not EMAIL_RE.match(email) or email in seen_emails:+ continue+ if prospect.status not in SENDABLE_STATUSES:+ continue+ if (prospect.email_status or "guess") not in sendable_email:+ skipped += 1+ continue+ seen_emails.add(email)++ cumulative_days = 0+ for step in campaign.sequence:+ cumulative_days += max(0, step.delay_days)+ send_at = now + timedelta(days=cumulative_days)+ item_id = await self.state.add_outbox_item(+ prospect_id=prospect.id,+ campaign_id=campaign.id,+ step=step.step,+ to_email=email,+ subject=self._render(step.subject, prospect),+ body=self._render(step.body, prospect),+ send_at=send_at.isoformat(),+ status=initial_status,+ provider=self.provider.name,+ )+ if item_id:+ staged += 1+ await self.state.update_prospect_status(prospect.id, "queued")++ await self.state.update_campaign(campaign.id, status="active")+ if skipped:+ logger.info(+ f"Sender: held back {skipped} prospect(s) with unverified "+ f"emails from '{campaign.name}'."+ )+ if staged:+ mode = "awaiting your approval" if require_approval else "approved"+ logger.info(+ f"Sender: staged {staged} email(s) for '{campaign.name}' "+ f"({mode}). Review in the dashboard Outbox tab."+ )+ await self.state.log_action(+ action_type="stage_campaign",+ agent="sender",+ details={"campaign": campaign.name, "staged": staged, "mode": mode},+ )++ async def _drain_due(self):+ """Send due, approved outbox items through the provider."""+ if not self.provider.is_configured():+ logger.warning(+ f"Sender: mail provider '{self.provider.name}' is not configured "+ "yet — outbox is holding. See CLAUDE.md → provider setup."+ )+ return++ max_daily = self.config.channels.email.max_daily_sends+ sent_today = await self.state.count_outbox_sent_today()+ budget = min(MAX_SENDS_PER_CYCLE, max_daily - sent_today)+ if budget <= 0:+ logger.info(f"Sender: daily send cap reached ({sent_today}/{max_daily}).")+ return++ now = datetime.now(timezone.utc).replace(tzinfo=None).isoformat()+ due = await self.state.get_outbox(+ status="approved", due_before=now, limit=budget * 3+ )+ if not due:+ return++ allow_risky = getattr(self.config.channels.email, "send_to_risky", False)+ sent = 0+ for item in due:+ if sent >= budget:+ break++ prospect = await self.state.get_prospect(item["prospect_id"])+ if prospect is None:+ await self.state.update_outbox_item(+ item["id"], status="failed", error="prospect missing"+ )+ continue++ # Stop-on-reply / opt-out: never continue a sequence after contact.+ if item["kind"] == "sequence" and prospect.status in STOP_STATUSES:+ await self.state.update_outbox_item(+ item["id"], status="cancelled",+ error=f"prospect status '{prospect.status}'",+ )+ continue++ gate = pre_send_check(+ item["to_email"], item["subject"], item["body"],+ prospect=prospect, allow_risky=allow_risky, kind=item["kind"],+ )+ if not gate:+ await self.state.update_outbox_item(+ item["id"], status="failed",+ error="gate: " + "; ".join(gate.reasons)[:300],+ )+ logger.warning(+ f"Sender: gate blocked email to {item['to_email']}: "+ f"{gate.reasons}"+ )+ continue++ result = await self.provider.send_email(+ item["to_email"], item["subject"], item["body"],+ thread_ref=item.get("thread_ref", ""),+ in_reply_to=item.get("in_reply_to", ""),+ )+ if not result.ok:+ await self.state.update_outbox_item(+ item["id"], status="failed", error=result.error[:300]+ )+ continue++ sent += 1+ now_iso = datetime.now(timezone.utc).replace(tzinfo=None).isoformat()+ await self.state.update_outbox_item(+ item["id"], status="sent", sent_at=now_iso,+ message_id=result.message_id, thread_ref=result.thread_ref,+ )+ if prospect.status in ("new", "queued"):+ await self.state.update_prospect_status(prospect.id, "contacted")++ # A sent reply belongs in its conversation thread.+ if item["kind"] == "reply" and item.get("conversation_id"):+ try:+ convo = await self.state.get_conversation(item["conversation_id"])+ if convo:+ from harvey.models.conversation import Message+ convo.thread.append(+ Message(sender="harvey", content=item["body"])+ )+ await self.state.update_conversation(+ convo.id, thread_json=convo.thread_json()+ )+ except Exception as e:+ logger.debug(f"Sender: convo append failed: {e}")+ await self.state.log_action(+ action_type="email_sent",+ agent="sender",+ details={+ "to": item["to_email"], "step": item["step"],+ "kind": item["kind"], "provider": self.provider.name,+ },+ )+ logger.info(+ f"Sender: sent step {item['step']} to {item['to_email']} "+ f"via {self.provider.name}."+ )+ # Human-ish pacing BETWEEN sends (not after the last, and never+ # in tests where pacing is disabled).+ if self.send_pacing and sent < budget:+ await asyncio.sleep(random.uniform(*SEND_JITTER_SECONDS))++ if sent:+ logger.info(f"Sender: {sent} email(s) sent this cycle "+ f"({sent_today + sent}/{max_daily} today).")diff --git a/harvey/agents/writer.py b/harvey/agents/writer.pyindex 77955f8..2f9c059 100644--- a/harvey/agents/writer.py+++ b/harvey/agents/writer.py@@ -1,21 +1,48 @@-"""Writer — crafts personalized email sequences."""+"""Writer — crafts personalized email sequences.++With a native mail provider (Gmail/SMTP), email 1 is written PER PROSPECT+from grounded facts — the company's own description, detected tech stack,+and buying signals — instead of one merge-tag template shared by a whole+batch. Mass-templated "token-swap" mail is exactly what inbox filters now+cluster and junk; a specific, verifiable first line is what earns replies.+Steps 2-3 stay campaign-level (proof point + breakup are less personal by+design). Every draft still passes the deterministic pre-send gate.+"""import jsonimport logging+from datetime import datetime, timezonefrom harvey.brain import Brainfrom harvey.config import HarveyConfig+from harvey.integrations.mail_provider import NATIVE_PROVIDERSfrom harvey.models.campaign import Campaign, EmailStepfrom harvey.state import StateManagerlogger = logging.getLogger("harvey.writer")+# Native mode: one Claude call per prospect for email 1, so batches stay+# small — the rest of the 'new' pool is picked up on later cycles.+NATIVE_BATCH_CAP = 8+LEGACY_BATCH_CAP = 50+class Writer:- def __init__(self, brain: Brain, state: StateManager, config: HarveyConfig):+ def __init__(+ self,+ brain: Brain,+ state: StateManager,+ config: HarveyConfig,+ env=None,+ ):self.brain = brainself.state = stateself.config = config+ self.env = env++ @property+ def is_native(self) -> bool:+ return self.config.channels.email.provider in NATIVE_PROVIDERSasync def run(self):"""Create email campaigns for prospects that need outreach."""@@ -77,6 +104,11 @@ class Writer:for p in prospects:await self.state.update_prospect_status(p.id, "queued")+ # Native providers: replace the shared template for email 1 with+ # a per-prospect draft grounded in that company's actual facts.+ if self.is_native:+ await self._personalize_first_emails(campaign_id, prospects)+await self.state.log_action(action_type="write_campaign",agent="writer",@@ -92,6 +124,123 @@ class Writer:f"{len(sequence)} emails for {len(prospects)} prospects.")+ async def _personalize_first_emails(self, campaign_id: str, prospects: list):+ """Draft a grounded, per-prospect email 1 and stage it in the outbox.++ The outbox unique index means the sender's later template staging+ can't overwrite these rows — a personalized draft always wins the+ (campaign, prospect, step 1) slot. If a draft fails, the slot stays+ empty and the sender falls back to the campaign template.+ """+ require_approval = getattr(+ self.config.channels.email, "require_approval", True+ )+ status = "pending_review" if require_approval else "approved"+ now = datetime.now(timezone.utc).replace(tzinfo=None).isoformat()+ provider = self.config.channels.email.provider++ drafted = 0+ for prospect in prospects:+ try:+ draft = await self._write_personal_email(prospect)+ except Exception as e:+ logger.warning(f"Writer: personal draft failed for {prospect.email}: {e}")+ continue+ if not draft:+ continue+ item_id = await self.state.add_outbox_item(+ prospect_id=prospect.id,+ campaign_id=campaign_id,+ step=1,+ to_email=prospect.email,+ subject=draft["subject"],+ body=draft["body"],+ send_at=now,+ status=status,+ provider=provider,+ )+ if item_id:+ drafted += 1++ if drafted:+ logger.info(+ f"Writer: drafted {drafted} personalized first email(s) "+ f"({'awaiting approval' if require_approval else 'approved'})."+ )++ async def _write_personal_email(self, prospect) -> dict | None:+ """One grounded draft for one person. Facts in, one email out."""+ facts = [+ f"- Name: {prospect.full_name()}",+ f"- Title: {prospect.title}",+ f"- Company: {prospect.company}",+ ]+ if prospect.industry:+ facts.append(f"- Industry: {prospect.industry}")++ company = None+ if prospect.company_id:+ try:+ company = await self.state.get_company(prospect.company_id)+ except Exception:+ company = None+ if company:+ if company.description:+ facts.append(f"- What the company says about itself: {company.description}")+ if company.tech_stack:+ facts.append(f"- Tools detected on their website: {', '.join(company.tech_stack[:6])}")+ for signal in company.signals[:3]:+ facts.append(+ f"- Signal ({signal.get('type', 'signal')}): {signal.get('detail', '')}"+ )+ if prospect.personalization_notes:+ facts.append(f"- Research notes: {prospect.personalization_notes}")++ prompt = self.brain.load_prompt(+ "writer",+ product_name=self.config.product.name,+ product_description=self.config.product.description,+ product_benefits="\n".join(f"- {b}" for b in self.config.product.key_benefits),+ product_pricing=self.config.product.pricing,+ persona_name=self.config.persona.name,+ persona_company=self.config.persona.company,+ persona_role=self.config.persona.role,+ persona_tone=self.config.persona.tone,+ ) or ""+ if self.skills:+ prompt += "\n\n" + self.skills++ prompt += f"""++Write ONE cold email (the very first touch) to this specific person.++FACTS — everything you know about them. Every claim about the prospect in+your email must come from these facts. If a fact isn't listed here, you+don't know it — do not invent funding rounds, mutual contacts, metrics,+or anything else:+{chr(10).join(facts)}++Requirements:+- Under 75 words. One specific observation from the FACTS above, one+ question. No pitch, no product name.+- Write the actual text (no merge variables — you know their name/company).+- Subject: lowercase, 2-5 words.+- Follow every STRICT EMAIL RULE above.++Return ONLY JSON: {{"subject": "...", "body": "..."}}"""++ result = await self.brain.think_json(+ prompt, session_id="harvey-writer",+ agent="writer", task="personal_email",+ )+ if not isinstance(result, dict):+ return None+ subject = str(result.get("subject") or "").strip()+ body = str(result.get("body") or "").strip()+ if not subject or not body:+ return None+ return {"subject": subject[:120], "body": body[:2000]}+async def _write_sequence(self, prospects: list) -> list[EmailStep]:"""Ask the brain to write a 3-email sequence."""# Build context about the prospects@@ -218,12 +367,8 @@ Return ONLY a JSON array (no markdown fences, no commentary):batches[key] = []batches[key].append(prospect)- # Cap each batch at 50 prospects (Instantly best practice)- capped = {}- for key, group in batches.items():- if len(group) > 50:- capped[key] = group[:50]- else:- capped[key] = group-- return capped+ # Cap batch size. Native mode stays small because email 1 is drafted+ # per prospect (one Claude call each); leftovers ride the next cycle.+ # Small segments also outperform blasts on deliverability.+ cap = NATIVE_BATCH_CAP if self.is_native else LEGACY_BATCH_CAP+ return {key: group[:cap] for key, group in batches.items()}diff --git a/harvey/cli.py b/harvey/cli.pyindex 813c73e..1e296bb 100644--- a/harvey/cli.py+++ b/harvey/cli.py@@ -255,6 +255,96 @@ def cmd_export(args):asyncio.run(_export())+def cmd_gmail(args):+ """Gmail provider utilities (auth / test)."""+ from harvey.config import load_env++ env = load_env()+ if args.gmail_action == "auth":+ from harvey.integrations.gmail import run_auth_flow+ ok = run_auth_flow(env.gmail_client_id, env.gmail_client_secret)+ sys.exit(0 if ok else 1)++ if args.gmail_action == "test":+ from harvey.config import load_config+ from harvey.integrations.gmail import GmailProvider++ async def _test():+ provider = GmailProvider(load_config(), env)+ ok, detail = await provider.test_connection()+ print(f"\n {'✓' if ok else '✗'} {detail}\n")+ sys.exit(0 if ok else 1)+ asyncio.run(_test())+++def cmd_outbox(args):+ """Review and approve queued outgoing emails."""+ from harvey.state import StateManager++ async def _outbox():+ state = StateManager()+ await state.init_db()++ if args.approve_all:+ n = await state.approve_outbox()+ print(f"\n Approved {n} email(s). They'll send on schedule.\n")+ return+ if args.approve:+ n = await state.approve_outbox(args.approve)+ print(f"\n {'Approved.' if n else 'No pending item with that id.'}\n")+ return+ if args.reject:+ await state.update_outbox_item(args.reject, status="rejected")+ print("\n Rejected.\n")+ return++ paused = await state.get_setting("sending_paused")+ if paused:+ print(f"\n ⚠ SENDING PAUSED: {paused}")+ print(" Resume with: harvey sending resume")++ pending = await state.get_outbox(status="pending_review", limit=50)+ approved = await state.get_outbox(status="approved", limit=10)+ print(f"\n Outbox — {len(pending)} awaiting approval, "+ f"{len(approved)}+ approved/scheduled")+ print(" " + "=" * 60)+ for item in pending:+ print(f"\n [{item['id']}] step {item['step']} ({item['kind']}) "+ f"→ {item['to_email']} (send {item['send_at'][:16]})")+ print(f" Subject: {item['subject']}")+ body_preview = (item["body"][:200] + "...") if len(item["body"]) > 200 else item["body"]+ for line in body_preview.splitlines():+ print(f" {line}")+ if pending:+ print("\n Approve: harvey outbox --approve <id> | all: harvey outbox --approve-all")+ print(" Reject: harvey outbox --reject <id>\n")+ else:+ print(" Nothing awaiting review.\n")++ asyncio.run(_outbox())+++def cmd_sending(args):+ """Pause/resume the sending kill switch."""+ from harvey.state import StateManager++ async def _run():+ state = StateManager()+ await state.init_db()+ if args.sending_action == "pause":+ await state.set_setting("sending_paused", "paused manually")+ print("\n Sending paused. Nothing will leave the outbox.\n")+ elif args.sending_action == "resume":+ await state.set_setting("sending_paused", "")+ await state.set_setting("bounce_count", "0")+ print("\n Sending resumed (bounce counter reset).\n")+ else:+ paused = await state.get_setting("sending_paused")+ print(f"\n Sending: {'PAUSED — ' + paused if paused else 'active'}\n")++ asyncio.run(_run())++def main():from harvey.paths import PROJECT_ROOTproject_root = str(PROJECT_ROOT)@@ -313,6 +403,25 @@ def main():sub.add_argument("--all", action="store_true", help="Export everything, no filters")sub.set_defaults(func=cmd_export)+ # harvey gmail auth|test+ sub = subparsers.add_parser("gmail", help="Gmail provider setup")+ sub.add_argument("gmail_action", choices=["auth", "test"],+ help="auth: one-time OAuth; test: verify connection")+ sub.set_defaults(func=cmd_gmail)++ # harvey outbox+ sub = subparsers.add_parser("outbox", help="Review/approve queued emails")+ sub.add_argument("--approve", metavar="ID", default="", help="Approve one item")+ sub.add_argument("--approve-all", action="store_true", help="Approve all pending")+ sub.add_argument("--reject", metavar="ID", default="", help="Reject one item")+ sub.set_defaults(func=cmd_outbox)++ # harvey sending pause|resume|status+ sub = subparsers.add_parser("sending", help="Kill switch: pause/resume sending")+ sub.add_argument("sending_action", nargs="?", default="status",+ choices=["pause", "resume", "status"])+ sub.set_defaults(func=cmd_sending)+# harvey usagesub = subparsers.add_parser("usage", help="Show Claude usage and quota")sub.add_argument("--days", type=int, default=30, help="Breakdown window (default: 30)")diff --git a/harvey/config.py b/harvey/config.pyindex ca85b0a..0a0b94e 100644--- a/harvey/config.py+++ b/harvey/config.py@@ -64,12 +64,21 @@ class ICPConfig(BaseModel):class EmailChannelConfig(BaseModel):enabled: bool = True+ # "instantly" (legacy), "gmail" (Gmail/Workspace via API — recommended),+ # or "smtp" (any SMTP+IMAP mailbox: AgentMail, Fastmail, ...)provider: str = "instantly"max_daily_sends: int = 50# When True, also send to catch-all ("risky") domains, not just verified# mailboxes. Off by default — catch-alls accept everything, so a bad# guess still bounces.send_to_risky: bool = False+ # Copilot mode (native providers): every outgoing email waits in the+ # outbox for your approval (dashboard → Outbox, or `harvey outbox`).+ # Set false for full autopilot once you trust the output.+ require_approval: bool = True+ # Kill switch: pause all sending when bounces exceed this fraction of+ # sent mail (measured over the trailing sends). 0 disables the switch.+ max_bounce_rate: float = 0.05@field_validator("max_daily_sends")@classmethod@@ -154,6 +163,17 @@ class EnvConfig(BaseModel):serper_api_key: str = ""reoon_api_key: str = ""zerobounce_api_key: str = ""+ # Native mail providers (channels.email.provider: gmail | smtp)+ gmail_client_id: str = ""+ gmail_client_secret: str = ""+ smtp_host: str = ""+ smtp_port: int = 587+ smtp_username: str = ""+ smtp_password: str = ""+ imap_host: str = ""+ imap_port: int = 993+ imap_username: str = ""+ imap_password: str = ""def _format_validation_error(e: ValidationError) -> str:@@ -217,12 +237,17 @@ def load_env() -> EnvConfig:serper_api_key=os.getenv("SERPER_API_KEY", "").strip(),reoon_api_key=os.getenv("REOON_API_KEY", "").strip(),zerobounce_api_key=os.getenv("ZEROBOUNCE_API_KEY", "").strip(),+ gmail_client_id=os.getenv("GMAIL_CLIENT_ID", "").strip(),+ gmail_client_secret=os.getenv("GMAIL_CLIENT_SECRET", "").strip(),+ smtp_host=os.getenv("SMTP_HOST", "").strip(),+ smtp_port=int(os.getenv("SMTP_PORT", "587").strip() or 587),+ smtp_username=os.getenv("SMTP_USERNAME", "").strip(),+ smtp_password=os.getenv("SMTP_PASSWORD", "").strip(),+ imap_host=os.getenv("IMAP_HOST", "").strip(),+ imap_port=int(os.getenv("IMAP_PORT", "993").strip() or 993),+ imap_username=os.getenv("IMAP_USERNAME", "").strip(),+ imap_password=os.getenv("IMAP_PASSWORD", "").strip(),)- if not env.instantly_api_key:- logger.warning(- "INSTANTLY_API_KEY is not set — email sending will be disabled "- "until it's added to .env."- )return envdiff --git a/harvey/dashboard.py b/harvey/dashboard.pyindex 6885eb3..cf4562a 100644--- a/harvey/dashboard.py+++ b/harvey/dashboard.py@@ -572,6 +572,82 @@ async def get_prospects():return rows+def _state():+ from harvey.state import StateManager+ return StateManager(db_path=str(DB_PATH))++[email protected]("/api/outbox")+async def get_outbox_api():+ """Outbox queue + kill-switch state for the Outbox tab."""+ try:+ state = _state()+ await state.init_db()+ return {+ "paused": await state.get_setting("sending_paused"),+ "pending": await state.get_outbox(status="pending_review", limit=100),+ "approved": await state.get_outbox(status="approved", limit=50),+ "sent": (await query_db(+ "SELECT * FROM outbox WHERE status = 'sent' "+ "ORDER BY sent_at DESC LIMIT 25")),+ "failed": (await query_db(+ "SELECT * FROM outbox WHERE status IN ('failed','rejected','cancelled') "+ "ORDER BY updated_at DESC LIMIT 25")),+ }+ except Exception as e:+ return {"error": str(e)}++[email protected]("/api/outbox/approve-all")+async def outbox_approve_all():+ try:+ state = _state()+ await state.init_db()+ n = await state.approve_outbox()+ return {"success": True, "approved": n}+ except Exception as e:+ return JSONResponse({"success": False, "message": str(e)}, status_code=500)++[email protected]("/api/outbox/{item_id}/approve")+async def outbox_approve(item_id: str):+ try:+ state = _state()+ await state.init_db()+ n = await state.approve_outbox(item_id)+ return {"success": bool(n)}+ except Exception as e:+ return JSONResponse({"success": False, "message": str(e)}, status_code=500)++[email protected]("/api/outbox/{item_id}/reject")+async def outbox_reject(item_id: str):+ try:+ state = _state()+ await state.init_db()+ await state.update_outbox_item(item_id, status="rejected")+ return {"success": True}+ except Exception as e:+ return JSONResponse({"success": False, "message": str(e)}, status_code=500)++[email protected]("/api/sending/{action}")+async def sending_toggle(action: str):+ if action not in ("pause", "resume"):+ return JSONResponse({"success": False, "message": "unknown action"}, status_code=400)+ try:+ state = _state()+ await state.init_db()+ if action == "pause":+ await state.set_setting("sending_paused", "paused from dashboard")+ else:+ await state.set_setting("sending_paused", "")+ await state.set_setting("bounce_count", "0")+ return {"success": True}+ except Exception as e:+ return JSONResponse({"success": False, "message": str(e)}, status_code=500)++@app.get("/api/export/prospects.csv")async def export_prospects(all: bool = False, min_score: int = 0, email_status: str = ""):"""Sequencer-ready CSV download of the prospect list."""@@ -1097,6 +1173,7 @@ DASHBOARD_HTML = r"""<!DOCTYPE html><button onclick="showTab('companies', this)">Companies</button><button onclick="showTab('prospects', this)">Contacts</button><button onclick="showTab('campaigns', this)">Campaigns</button>+ <button onclick="showTab('outbox', this)">Outbox</button><button onclick="showTab('conversations', this)">Conversations</button><button onclick="showTab('activity', this)">Activity</button><button onclick="showTab('usage', this)">Usage</button>@@ -1146,6 +1223,18 @@ DASHBOARD_HTML = r"""<!DOCTYPE html><div id="campaigns-list"></div></div>+<!-- Outbox -->+<div id="outbox" class="section">+ <div class="section-head" style="display:flex;align-items:flex-end;justify-content:space-between;gap:16px">+ <div><h2>Outbox</h2><p>Every outgoing email waits here for approval before it sends. Approve once you're happy — Harvey handles the schedule.</p></div>+ <div style="display:flex;gap:8px;flex-shrink:0">+ <button class="btn btn-primary btn-sm" onclick="outboxApproveAll()">Approve all pending</button>+ </div>+ </div>+ <div id="outbox-banner"></div>+ <div id="outbox-list"></div>+</div>+<!-- Conversations --><div id="conversations" class="section"><div class="section-head"><h2>Conversations</h2><p>Every reply, and how Harvey handled it.</p></div>@@ -1421,6 +1510,7 @@ function loadCurrentTab() {case 'companies': if (!companyDrill) loadCompanies(); break;case 'prospects': loadProspects(); break;case 'campaigns': loadCampaigns(); break;+ case 'outbox': loadOutbox(); break;case 'conversations': loadConversations(); break;case 'activity': loadActivity(); break;case 'usage': loadUsage(); break;@@ -1739,6 +1829,94 @@ async function loadUsage() {}}+async function loadOutbox() {+ const data = await api('/api/outbox');+ const banner = document.getElementById('outbox-banner');+ const list = document.getElementById('outbox-list');+ if (!data) { list.innerHTML = offlineState(); return; }++ if (data.paused) {+ banner.innerHTML = '<div class="card" style="border-color:var(--red)">' ++ '<h2 style="color:var(--red)">⚠ Sending paused</h2>' ++ '<p style="color:var(--text-2);font-size:13px;margin-bottom:12px">' + escHtml(data.paused) + '</p>' ++ '<button class="btn btn-secondary btn-sm" onclick="sendingToggle(\'resume\')">Resume sending</button></div>';+ } else {+ banner.innerHTML = '<div style="display:flex;justify-content:flex-end;margin-bottom:12px">' ++ '<button class="btn btn-secondary btn-sm" onclick="sendingToggle(\'pause\')">Pause all sending</button></div>';+ }++ let html = '';+ const pending = data.pending || [];+ if (pending.length) {+ html += '<div class="card"><h2>Awaiting approval (' + pending.length + ')</h2>';+ for (const item of pending) {+ html += '<div style="border:1px solid var(--border);border-radius:10px;padding:14px;margin-bottom:10px">' ++ '<div style="display:flex;justify-content:space-between;gap:10px;align-items:center;margin-bottom:8px">' ++ '<div style="font-size:13px"><b>' + escHtml(item.to_email) + '</b>' ++ ' <span class="muted">· step ' + item.step + ' (' + escHtml(item.kind) + ')' ++ ' · sends ' + formatDate(item.send_at) + '</span></div>' ++ '<div style="display:flex;gap:6px;flex-shrink:0">' ++ '<button class="btn btn-primary btn-sm" onclick="outboxAct(\'' + item.id + '\',\'approve\')">Approve</button>' ++ '<button class="btn btn-secondary btn-sm" onclick="outboxAct(\'' + item.id + '\',\'reject\')">Reject</button>' ++ '</div></div>' ++ '<div style="font-size:13px;font-weight:600;margin-bottom:6px">' + escHtml(item.subject) + '</div>' ++ '<div style="font-size:13px;color:var(--text-2);white-space:pre-wrap">' + escHtml(item.body) + '</div>' ++ '</div>';+ }+ html += '</div>';+ } else {+ html += '<div class="card"><h2>Awaiting approval</h2>' ++ '<p style="color:var(--text-3);font-size:13px">Nothing to review. New drafts land here before they can send.</p></div>';+ }++ const table = (title, rows, cols) => {+ if (!rows || !rows.length) return '';+ let h = '<div class="card"><h2>' + title + '</h2><div class="table-card"><table><thead><tr>' ++ cols.map(c => '<th>' + c[0] + '</th>').join('') + '</tr></thead><tbody>';+ for (const r of rows) {+ h += '<tr>' + cols.map(c => '<td' + (c[2] ? ' class="muted"' : '') + '>' ++ escHtml(String(c[1](r) ?? '')) + '</td>').join('') + '</tr>';+ }+ return h + '</tbody></table></div></div>';+ };++ html += table('Approved & scheduled', data.approved, [+ ['To', r => r.to_email], ['Step', r => r.step], ['Subject', r => r.subject],+ ['Sends', r => formatDate(r.send_at), true],+ ]);+ html += table('Recently sent', data.sent, [+ ['To', r => r.to_email], ['Step', r => r.step], ['Subject', r => r.subject],+ ['Sent', r => formatDate(r.sent_at), true],+ ]);+ html += table('Failed / rejected / cancelled', data.failed, [+ ['To', r => r.to_email], ['Status', r => r.status],+ ['Reason', r => r.error, true], ['Updated', r => formatDate(r.updated_at), true],+ ]);++ list.innerHTML = html;+}++async function outboxAct(id, action) {+ const data = await api('/api/outbox/' + encodeURIComponent(id) + '/' + action, {method: 'POST'});+ if (data && data.success) showToast(action === 'approve' ? 'Approved — will send on schedule.' : 'Rejected.', 'success');+ else showToast('Action failed.', 'error');+ loadOutbox();+}++async function outboxApproveAll() {+ const data = await api('/api/outbox/approve-all', {method: 'POST'});+ if (data && data.success) showToast('Approved ' + data.approved + ' email(s).', 'success');+ else showToast('Approve-all failed.', 'error');+ loadOutbox();+}++async function sendingToggle(action) {+ const data = await api('/api/sending/' + action, {method: 'POST'});+ if (data && data.success) showToast(action === 'pause' ? 'Sending paused.' : 'Sending resumed.', 'success');+ else showToast('Failed.', 'error');+ loadOutbox();+}+async function loadCompanies() {companyDrill = false;const el = document.getElementById('companies-list');@@ -1929,6 +2107,7 @@ setInterval(() => {case 'conversations': loadConversations(); break;case 'activity': loadActivity(); break;case 'usage': loadUsage(); break;+ case 'outbox': loadOutbox(); break;case 'controls': loadLogs(); break;// settings & help: never auto-refreshed (user may be typing)}diff --git a/harvey/gate.py b/harvey/gate.pynew file mode 100644index 0000000..319f4ed--- /dev/null+++ b/harvey/gate.py@@ -0,0 +1,107 @@+"""Deterministic pre-send gate — the last line before an email leaves.++LLM instructions are a floor, not a guarantee. Every outgoing email passes+these non-LLM checks at send time; a failure blocks the send and records+why, so a bad generation can never reach a prospect. This is the pattern+the surviving commercial AI-SDR vendors converged on after the 2025+autonomous-agent flameouts: constraints in the prompt, verification in+code.+"""++import re+from dataclasses import dataclass, field++# Spam-filter and AI-tell phrases that must never appear in outgoing mail.+# Mirrors prompts/writer.md; enforced here because prompts can be ignored.+BANNED_PHRASES = (+ "act now", "buy now", "click here", "click below", "limited time",+ "limited offer", "exclusive deal", "special offer", "risk-free",+ "no obligation", "100% free", "make money", "double your",+ "guaranteed results", "winner", "congratulations",+ "i hope this finds you well", "hope this email finds you well",+ "i hope this email finds you", "picture this", "what if i told you",+ "in today's fast-paced world", "game-changer", "game changer",+ "cutting-edge solution", "revolutionize your", "unlock your",+ "supercharge your", "as an ai", "i'm an ai", "i am an ai",+ "as a language model",+)++MAX_SUBJECT_LEN = 90+MAX_BODY_WORDS = 220 # hard ceiling; sequences aim far lower+MAX_LINKS = 1++EMAIL_RE = re.compile(r"^[A-Za-z0-9._%+\-]+@[A-Za-z0-9.\-]+\.[A-Za-z]{2,}$")+MERGE_TAG_RE = re.compile(r"\{\{\s*\w+\s*\}\}|\{\s*\w+\s*\}(?!\})")+URL_RE = re.compile(r"https?://\S+", re.IGNORECASE)++DELIVERABLE_STATUSES = {"verified", "risky"}+++@dataclass+class GateResult:+ ok: bool+ reasons: list[str] = field(default_factory=list)++ def __bool__(self):+ return self.ok+++def pre_send_check(+ to_email: str,+ subject: str,+ body: str,+ prospect=None,+ allow_risky: bool = False,+ kind: str = "sequence",+) -> GateResult:+ """Run every deterministic check. Returns ok=False with reasons on any hit."""+ reasons: list[str] = []+ subject = (subject or "").strip()+ body = (body or "").strip()+ to_email = (to_email or "").strip().lower()++ # Recipient sanity+ if not EMAIL_RE.match(to_email):+ reasons.append(f"invalid recipient address '{to_email}'")+ if prospect is not None:+ if (prospect.email or "").strip().lower() != to_email:+ reasons.append("recipient does not match the prospect record")+ allowed = DELIVERABLE_STATUSES if allow_risky else {"verified"}+ # Replies go back to someone who emailed US — deliverability proven.+ if kind == "sequence" and (prospect.email_status or "guess") not in allowed:+ reasons.append(+ f"email_status '{prospect.email_status or 'guess'}' is not deliverable"+ )++ # Content sanity+ if not subject:+ reasons.append("empty subject")+ if not body:+ reasons.append("empty body")+ if len(subject) > MAX_SUBJECT_LEN:+ reasons.append(f"subject too long ({len(subject)} > {MAX_SUBJECT_LEN})")+ word_count = len(body.split())+ if word_count > MAX_BODY_WORDS:+ reasons.append(f"body too long ({word_count} words > {MAX_BODY_WORDS})")++ # Unrendered merge tags — the classic mass-mail embarrassment+ leftover = MERGE_TAG_RE.findall(subject + " " + body)+ if leftover:+ reasons.append(f"unrendered merge tags: {sorted(set(leftover))[:3]}")++ # Banned phrases (spam triggers + AI tells)+ lowered = (subject + " " + body).lower()+ hits = [p for p in BANNED_PHRASES if p in lowered]+ if hits:+ reasons.append(f"banned phrases: {hits[:3]}")++ # Link budget: cold email with links goes to spam+ links = URL_RE.findall(body)+ if len(links) > MAX_LINKS:+ reasons.append(f"too many links ({len(links)} > {MAX_LINKS})")++ # No HTML — Harvey sends plain text only+ if re.search(r"<\s*(html|body|div|table|img|a)\b", body, re.IGNORECASE):+ reasons.append("HTML markup in body (plain text only)")++ return GateResult(ok=not reasons, reasons=reasons)diff --git a/harvey/integrations/gmail.py b/harvey/integrations/gmail.pynew file mode 100644index 0000000..f97ccac--- /dev/null+++ b/harvey/integrations/gmail.py@@ -0,0 +1,337 @@+"""Gmail / Google Workspace mailbox provider — raw REST, no SDK weight.++Setup (one-time):+1. Google Cloud Console → create a project → enable the Gmail API.+2. OAuth consent screen (internal or testing) → create an OAuth client of+ type "Desktop app". Put its id/secret in .env as GMAIL_CLIENT_ID /+ GMAIL_CLIENT_SECRET.+3. Run ``harvey gmail auth`` — opens a browser, stores the refresh token+ in data/gmail_token.json. Done: Harvey can send and read.++Deliverability notes: use a mailbox on a dedicated SECONDARY domain (never+your main one), configure SPF/DKIM/DMARC, ramp volume slowly, and watch+Google Postmaster Tools. Harvey's outbox pacing + kill switches handle the+rest.+"""++import base64+import json+import logging+import time+import webbrowser+from email.message import EmailMessage+from email.utils import parseaddr++import httpx++from harvey.paths import PROJECT_ROOT+from harvey.integrations.mail_provider import (+ InboundMessage,+ MailProvider,+ SendResult,+ looks_like_bounce,+)++logger = logging.getLogger("harvey.gmail")++TOKEN_FILE = PROJECT_ROOT / "data" / "gmail_token.json"+API_BASE = "https://gmail.googleapis.com/gmail/v1/users/me"+TOKEN_URL = "https://oauth2.googleapis.com/token"+AUTH_URL = "https://accounts.google.com/o/oauth2/v2/auth"+SCOPE = "https://www.googleapis.com/auth/gmail.modify"+LOOPBACK_PORT = 8765++TIMEOUT = httpx.Timeout(connect=10.0, read=30.0, write=30.0, pool=10.0)+++def _load_token() -> dict | None:+ try:+ if TOKEN_FILE.is_file():+ data = json.loads(TOKEN_FILE.read_text())+ if isinstance(data, dict) and data.get("refresh_token"):+ return data+ except (OSError, json.JSONDecodeError) as e:+ logger.warning(f"Could not read Gmail token file: {e}")+ return None+++def _save_token(data: dict):+ TOKEN_FILE.parent.mkdir(parents=True, exist_ok=True)+ TOKEN_FILE.write_text(json.dumps(data, indent=2))+ try:+ TOKEN_FILE.chmod(0o600)+ except OSError:+ pass+++class GmailProvider(MailProvider):+ name = "gmail"++ def __init__(self, config, env):+ self.config = config+ self.client_id = getattr(env, "gmail_client_id", "")+ self.client_secret = getattr(env, "gmail_client_secret", "")+ self._access_token = ""+ self._access_expires = 0.0++ def is_configured(self) -> bool:+ return bool(self.client_id and self.client_secret and _load_token())++ async def _token(self) -> str | None:+ """Valid access token, refreshing via the stored refresh token."""+ if self._access_token and time.time() < self._access_expires - 60:+ return self._access_token+ stored = _load_token()+ if not stored:+ logger.error("Gmail: not authorized. Run 'harvey gmail auth'.")+ return None+ try:+ async with httpx.AsyncClient(timeout=TIMEOUT) as client:+ resp = await client.post(TOKEN_URL, data={+ "client_id": self.client_id,+ "client_secret": self.client_secret,+ "refresh_token": stored["refresh_token"],+ "grant_type": "refresh_token",+ })+ except httpx.HTTPError as e:+ logger.error(f"Gmail token refresh failed: {e}")+ return None+ if resp.status_code != 200:+ logger.error(+ f"Gmail token refresh rejected ({resp.status_code}). "+ "Re-run 'harvey gmail auth'."+ )+ return None+ data = resp.json()+ self._access_token = data.get("access_token", "")+ self._access_expires = time.time() + float(data.get("expires_in", 3600))+ return self._access_token or None++ async def _request(self, method: str, path: str, **kwargs):+ token = await self._token()+ if not token:+ return None+ headers = {"Authorization": f"Bearer {token}"}+ try:+ async with httpx.AsyncClient(timeout=TIMEOUT) as client:+ resp = await client.request(+ method, f"{API_BASE}{path}", headers=headers, **kwargs+ )+ except httpx.HTTPError as e:+ logger.error(f"Gmail request failed ({path}): {e}")+ return None+ if resp.status_code >= 400:+ logger.error(f"Gmail API {resp.status_code} on {path}: {resp.text[:200]}")+ return None+ try:+ return resp.json()+ except ValueError:+ return {}++ async def send_email(+ self,+ to_email: str,+ subject: str,+ body: str,+ thread_ref: str = "",+ in_reply_to: str = "",+ ) -> SendResult:+ msg = EmailMessage()+ persona = self.config.persona+ msg["To"] = to_email+ msg["From"] = f"{persona.name} <{persona.email}>" if persona.email else persona.name+ msg["Subject"] = subject+ if in_reply_to:+ msg["In-Reply-To"] = in_reply_to+ msg["References"] = in_reply_to+ msg.set_content(body)++ raw = base64.urlsafe_b64encode(msg.as_bytes()).decode()+ payload: dict = {"raw": raw}+ if thread_ref:+ payload["threadId"] = thread_ref++ data = await self._request("POST", "/messages/send", json=payload)+ if data is None:+ return SendResult(ok=False, error="gmail send failed")++ # Fetch the assigned RFC Message-ID for bounce/thread matching later.+ message_id = ""+ gmail_id = data.get("id", "")+ if gmail_id:+ meta = await self._request(+ "GET", f"/messages/{gmail_id}",+ params={"format": "metadata", "metadataHeaders": "Message-ID"},+ )+ for h in ((meta or {}).get("payload") or {}).get("headers", []):+ if h.get("name", "").lower() == "message-id":+ message_id = h.get("value", "")+ return SendResult(+ ok=True,+ message_id=message_id or gmail_id,+ thread_ref=data.get("threadId", ""),+ )++ async def get_replies(self, limit: int = 50) -> list[InboundMessage]:+ listing = await self._request(+ "GET", "/messages",+ params={"q": "in:inbox -from:me newer_than:7d", "maxResults": limit},+ )+ if not listing:+ return []+ out = []+ for stub in listing.get("messages", [])[:limit]:+ data = await self._request(+ "GET", f"/messages/{stub['id']}", params={"format": "full"}+ )+ if not data:+ continue+ msg = self._parse_message(data)+ if msg:+ out.append(msg)+ return out++ @staticmethod+ def _parse_message(data: dict) -> InboundMessage | None:+ payload = data.get("payload") or {}+ headers = {+ h.get("name", "").lower(): h.get("value", "")+ for h in payload.get("headers", [])+ }+ from_email = parseaddr(headers.get("from", ""))[1].lower()+ if not from_email:+ return None+ subject = headers.get("subject", "")+ body = GmailProvider._extract_text(payload) or data.get("snippet", "")+ return InboundMessage(+ provider_id=data.get("id", ""),+ from_email=from_email,+ subject=subject,+ body=body[:5000],+ message_id=headers.get("message-id", ""),+ in_reply_to=headers.get("in-reply-to", ""),+ thread_ref=data.get("threadId", ""),+ date=headers.get("date", ""),+ is_bounce=looks_like_bounce(from_email, subject),+ headers=headers,+ )++ @staticmethod+ def _extract_text(payload: dict) -> str:+ """Depth-first hunt for the text/plain part of a MIME tree."""+ def decode(part) -> str:+ data = (part.get("body") or {}).get("data", "")+ if not data:+ return ""+ try:+ return base64.urlsafe_b64decode(data + "===").decode(errors="replace")+ except Exception:+ return ""++ if payload.get("mimeType", "").startswith("text/plain"):+ return decode(payload)+ for part in payload.get("parts", []) or []:+ text = GmailProvider._extract_text(part)+ if text:+ return text+ if payload.get("mimeType", "").startswith("text/html"):+ import re+ html = decode(payload)+ return re.sub(r"<[^>]+>", " ", html)+ return ""++ async def test_connection(self) -> tuple[bool, str]:+ if not self.client_id or not self.client_secret:+ return False, "GMAIL_CLIENT_ID / GMAIL_CLIENT_SECRET missing from .env"+ if not _load_token():+ return False, "Not authorized yet — run 'harvey gmail auth'"+ data = await self._request("GET", "/profile")+ if data and data.get("emailAddress"):+ return True, f"Connected as {data['emailAddress']}"+ return False, "Token refresh or profile fetch failed"+++def run_auth_flow(client_id: str, client_secret: str) -> bool:+ """One-time OAuth (installed-app loopback). Blocks until the browser+ round-trip completes; stores the refresh token for GmailProvider."""+ import http.server+ import secrets+ import threading+ import urllib.parse++ if not client_id or not client_secret:+ print(" Set GMAIL_CLIENT_ID and GMAIL_CLIENT_SECRET in .env first.")+ print(" (Google Cloud Console → APIs → Credentials → OAuth client, Desktop app)")+ return False++ state = secrets.token_urlsafe(16)+ redirect_uri = f"http://localhost:{LOOPBACK_PORT}"+ auth_url = AUTH_URL + "?" + urllib.parse.urlencode({+ "client_id": client_id,+ "redirect_uri": redirect_uri,+ "response_type": "code",+ "scope": SCOPE,+ "access_type": "offline",+ "prompt": "consent",+ "state": state,+ })++ result: dict = {}+ done = threading.Event()++ class Handler(http.server.BaseHTTPRequestHandler):+ def do_GET(self):+ query = urllib.parse.parse_qs(urllib.parse.urlparse(self.path).query)+ if query.get("state", [""])[0] != state:+ self.send_response(400); self.end_headers()+ return+ result["code"] = query.get("code", [""])[0]+ result["error"] = query.get("error", [""])[0]+ self.send_response(200)+ self.send_header("Content-Type", "text/html")+ self.end_headers()+ self.wfile.write(+ b"<h2>Harvey is connected to Gmail.</h2>You can close this tab.")+ done.set()++ def log_message(self, *args):+ pass++ server = http.server.HTTPServer(("localhost", LOOPBACK_PORT), Handler)+ thread = threading.Thread(target=server.serve_forever, daemon=True)+ thread.start()++ print(f"\n Opening browser for Google sign-in...\n {auth_url}\n")+ webbrowser.open(auth_url)+ if not done.wait(timeout=300):+ server.shutdown()+ print(" Timed out waiting for the browser round-trip.")+ return False+ server.shutdown()++ if result.get("error") or not result.get("code"):+ print(f" Authorization failed: {result.get('error') or 'no code returned'}")+ return False++ resp = httpx.post(TOKEN_URL, data={+ "client_id": client_id,+ "client_secret": client_secret,+ "code": result["code"],+ "grant_type": "authorization_code",+ "redirect_uri": redirect_uri,+ }, timeout=30)+ if resp.status_code != 200:+ print(f" Token exchange failed ({resp.status_code}): {resp.text[:200]}")+ return False+ data = resp.json()+ if not data.get("refresh_token"):+ print(" Google returned no refresh token. Remove Harvey from your"+ " account's third-party access list and retry.")+ return False+ _save_token({+ "refresh_token": data["refresh_token"],+ "scope": data.get("scope", SCOPE),+ "obtained_at": int(time.time()),+ })+ print(f" ✓ Gmail authorized. Token stored at {TOKEN_FILE}")+ return Truediff --git a/harvey/integrations/mail_provider.py b/harvey/integrations/mail_provider.pynew file mode 100644index 0000000..7813d57--- /dev/null+++ b/harvey/integrations/mail_provider.py@@ -0,0 +1,99 @@+"""Mail provider abstraction — Harvey owns its sending, Instantly optional.++A MailProvider gives Harvey full inbox control: send, read replies,+thread. Two native providers ship:++- ``gmail`` — a Google Workspace / Gmail mailbox via the Gmail REST API+ (recommended: a real mailbox on a dedicated secondary domain is the+ most deliverable way to send <50 cold emails/day).+- ``smtp`` — any mailbox that speaks SMTP + IMAP (AgentMail, Fastmail,+ a Workspace app-mailbox, self-hosted). Configuration via .env.++``instantly`` remains supported through the legacy Sender/Handler path —+it isn't a MailProvider because Instantly owns sequencing/sending itself.+"""++import logging+from abc import ABC, abstractmethod+from dataclasses import dataclass, field++logger = logging.getLogger("harvey.mail")++NATIVE_PROVIDERS = ("gmail", "smtp")+++@dataclass+class SendResult:+ ok: bool+ message_id: str = "" # RFC Message-ID (or provider id) of the sent mail+ thread_ref: str = "" # provider thread handle (Gmail threadId, etc.)+ error: str = ""+++@dataclass+class InboundMessage:+ provider_id: str # provider-unique id, used for reply dedup+ from_email: str+ subject: str = ""+ body: str = ""+ message_id: str = "" # RFC Message-ID header+ in_reply_to: str = "" # RFC In-Reply-To header (bounce/thread matching)+ thread_ref: str = ""+ date: str = ""+ is_bounce: bool = False+ headers: dict = field(default_factory=dict)+++BOUNCE_SENDERS = ("mailer-daemon", "postmaster", "mail delivery subsystem")+BOUNCE_SUBJECTS = (+ "delivery status notification", "undeliverable", "returned mail",+ "delivery failure", "failure notice", "could not be delivered",+ "delivery incomplete", "address not found",+)+++def looks_like_bounce(from_email: str, subject: str) -> bool:+ f = (from_email or "").lower()+ s = (subject or "").lower()+ return any(m in f for m in BOUNCE_SENDERS) or any(m in s for m in BOUNCE_SUBJECTS)+++class MailProvider(ABC):+ """A mailbox Harvey can send from and read replies out of."""++ name: str = "base"++ @abstractmethod+ def is_configured(self) -> bool:+ """Whether required credentials are present (no network)."""++ @abstractmethod+ async def send_email(+ self,+ to_email: str,+ subject: str,+ body: str,+ thread_ref: str = "",+ in_reply_to: str = "",+ ) -> SendResult:+ """Send one plain-text email. Reply threading when refs are given."""++ @abstractmethod+ async def get_replies(self, limit: int = 50) -> list[InboundMessage]:+ """Fetch recent inbound messages (dedup is the caller's job)."""++ @abstractmethod+ async def test_connection(self) -> tuple[bool, str]:+ """Cheap end-to-end credential check. Returns (ok, detail)."""+++def get_mail_provider(config, env) -> MailProvider | None:+ """Build the configured native provider, or None for instantly/unknown."""+ provider = (config.channels.email.provider or "").strip().lower()+ if provider == "gmail":+ from harvey.integrations.gmail import GmailProvider+ return GmailProvider(config, env)+ if provider == "smtp":+ from harvey.integrations.smtp_mail import SmtpImapProvider+ return SmtpImapProvider(config, env)+ return Nonediff --git a/harvey/integrations/smtp_mail.py b/harvey/integrations/smtp_mail.pynew file mode 100644index 0000000..9b55a3f--- /dev/null+++ b/harvey/integrations/smtp_mail.py@@ -0,0 +1,188 @@+"""Generic SMTP + IMAP mailbox provider.++Works with any real mailbox: AgentMail (their IMAP/SMTP relay), Fastmail,+a Google Workspace account with an app password, or self-hosted mail.++.env:+ SMTP_HOST= IMAP_HOST=+ SMTP_PORT=587 IMAP_PORT=993+ SMTP_USERNAME= IMAP_USERNAME= (defaults to SMTP_USERNAME)+ SMTP_PASSWORD= IMAP_PASSWORD= (defaults to SMTP_PASSWORD)++Sends as the configured persona (name + email). IMAP polling fetches+UNSEEN inbox messages and marks them seen after retrieval.+"""++import asyncio+import email+import email.policy+import imaplib+import logging+from email.message import EmailMessage+from email.utils import make_msgid, parseaddr++import aiosmtplib++from harvey.integrations.mail_provider import (+ InboundMessage,+ MailProvider,+ SendResult,+ looks_like_bounce,+)++logger = logging.getLogger("harvey.smtp")+++class SmtpImapProvider(MailProvider):+ name = "smtp"++ def __init__(self, config, env):+ self.config = config+ self.smtp_host = getattr(env, "smtp_host", "")+ self.smtp_port = int(getattr(env, "smtp_port", 587) or 587)+ self.smtp_user = getattr(env, "smtp_username", "")+ self.smtp_pass = getattr(env, "smtp_password", "")+ self.imap_host = getattr(env, "imap_host", "") or self.smtp_host+ self.imap_port = int(getattr(env, "imap_port", 993) or 993)+ self.imap_user = getattr(env, "imap_username", "") or self.smtp_user+ self.imap_pass = getattr(env, "imap_password", "") or self.smtp_pass++ def is_configured(self) -> bool:+ return bool(self.smtp_host and self.smtp_user and self.smtp_pass)++ async def send_email(+ self,+ to_email: str,+ subject: str,+ body: str,+ thread_ref: str = "",+ in_reply_to: str = "",+ ) -> SendResult:+ persona = self.config.persona+ msg = EmailMessage()+ msg["To"] = to_email+ msg["From"] = f"{persona.name} <{persona.email or self.smtp_user}>"+ msg["Subject"] = subject+ message_id = make_msgid(domain=(persona.email or self.smtp_user).split("@")[-1])+ msg["Message-ID"] = message_id+ if in_reply_to:+ msg["In-Reply-To"] = in_reply_to+ msg["References"] = in_reply_to+ msg.set_content(body)++ try:+ await aiosmtplib.send(+ msg,+ hostname=self.smtp_host,+ port=self.smtp_port,+ username=self.smtp_user,+ password=self.smtp_pass,+ start_tls=(self.smtp_port == 587),+ use_tls=(self.smtp_port == 465),+ timeout=30,+ )+ except Exception as e:+ logger.error(f"SMTP send failed to {to_email}: {e}")+ return SendResult(ok=False, error=str(e)[:300])+ return SendResult(ok=True, message_id=message_id, thread_ref=message_id)++ async def get_replies(self, limit: int = 50) -> list[InboundMessage]:+ loop = asyncio.get_event_loop()+ try:+ return await loop.run_in_executor(None, self._fetch_unseen, limit)+ except Exception as e:+ logger.error(f"IMAP fetch failed: {e}")+ return []++ def _fetch_unseen(self, limit: int) -> list[InboundMessage]:+ out: list[InboundMessage] = []+ conn = imaplib.IMAP4_SSL(self.imap_host, self.imap_port)+ try:+ conn.login(self.imap_user, self.imap_pass)+ conn.select("INBOX")+ status, data = conn.search(None, "UNSEEN")+ if status != "OK":+ return []+ ids = data[0].split()[-limit:]+ for msg_id in ids:+ status, parts = conn.fetch(msg_id, "(RFC822)")+ if status != "OK" or not parts or not isinstance(parts[0], tuple):+ continue+ parsed = self._parse_rfc822(parts[0][1])+ if parsed:+ out.append(parsed)+ finally:+ try:+ conn.logout()+ except Exception:+ pass+ return out++ @staticmethod+ def _parse_rfc822(raw: bytes) -> InboundMessage | None:+ try:+ msg = email.message_from_bytes(raw, policy=email.policy.default)+ except Exception:+ return None+ from_email = parseaddr(str(msg.get("From", "")))[1].lower()+ if not from_email:+ return None+ subject = str(msg.get("Subject", ""))++ body = ""+ try:+ part = msg.get_body(preferencelist=("plain", "html"))+ if part is not None:+ body = part.get_content()+ if part.get_content_type() == "text/html":+ import re+ body = re.sub(r"<[^>]+>", " ", body)+ except Exception:+ pass++ message_id = str(msg.get("Message-ID", "")).strip()+ return InboundMessage(+ provider_id=message_id or f"{from_email}:{subject}",+ from_email=from_email,+ subject=subject,+ body=(body or "")[:5000],+ message_id=message_id,+ in_reply_to=str(msg.get("In-Reply-To", "")).strip(),+ thread_ref=str(msg.get("In-Reply-To", "")).strip(),+ date=str(msg.get("Date", "")),+ is_bounce=looks_like_bounce(from_email, subject),+ )++ async def test_connection(self) -> tuple[bool, str]:+ if not self.is_configured():+ return False, "SMTP_HOST / SMTP_USERNAME / SMTP_PASSWORD missing from .env"+ # SMTP handshake+ try:+ smtp = aiosmtplib.SMTP(+ hostname=self.smtp_host, port=self.smtp_port,+ start_tls=(self.smtp_port == 587),+ use_tls=(self.smtp_port == 465), timeout=15,+ )+ await smtp.connect()+ await smtp.login(self.smtp_user, self.smtp_pass)+ await smtp.quit()+ except Exception as e:+ return False, f"SMTP login failed: {str(e)[:200]}"+ # IMAP handshake+ loop = asyncio.get_event_loop()++ def _imap_check():+ conn = imaplib.IMAP4_SSL(self.imap_host, self.imap_port)+ try:+ conn.login(self.imap_user, self.imap_pass)+ finally:+ try:+ conn.logout()+ except Exception:+ pass++ try:+ await loop.run_in_executor(None, _imap_check)+ except Exception as e:+ return False, f"SMTP OK but IMAP login failed: {str(e)[:200]}"+ return True, f"SMTP + IMAP connected as {self.smtp_user}"diff --git a/harvey/main.py b/harvey/main.pyindex 1f0c8ec..d4d1aac 100644--- a/harvey/main.py+++ b/harvey/main.py@@ -134,7 +134,7 @@ async def heartbeat(stop_event: asyncio.Event | None = None):from harvey.agents.analyst import Analystscout = Scout(brain, state, config, env)- writer = Writer(brain, state, config)+ writer = Writer(brain, state, config, env)sender = Sender(brain, state, config, env)handler = Handler(brain, state, config, env)analyst = Analyst(state)@@ -195,6 +195,12 @@ async def heartbeat(stop_event: asyncio.Event | None = None):elif action == "idle":tasks.append(("analyze", analyst.run()))+ # Native mail providers drain the outbox every cycle — due sends+ # and approved replies must go out on schedule regardless of the+ # cycle's primary action.+ if sender.is_native and not any(n == "send_campaign" for n, _ in tasks):+ tasks.append(("send_outbox", sender.run()))+if len(tasks) > 1:logger.info(f"Running {len(tasks)} agents in parallel: {[t[0] for t in tasks]}")diff --git a/harvey/state.py b/harvey/state.pyindex c9eacbe..3e7002d 100644--- a/harvey/state.py+++ b/harvey/state.py@@ -247,6 +247,49 @@ MIGRATIONS: list[str] = [ALTER TABLE companies ADD COLUMN tech_stack_json TEXT DEFAULT '[]';ALTER TABLE companies ADD COLUMN signals_json TEXT DEFAULT '[]';""",+ # ── v6: outbox (approval ladder + native sending) and settings KV ──+ """+ -- Every outgoing email becomes an outbox row first. Status ladder:+ -- pending_review -> approved -> sent+ -- (or rejected / cancelled / failed)+ -- The unique index on (campaign_id, prospect_id, step) is the+ -- double-send guard: retries and re-stages physically cannot+ -- duplicate a send.+ CREATE TABLE IF NOT EXISTS outbox (+ id TEXT PRIMARY KEY,+ campaign_id TEXT DEFAULT '',+ prospect_id TEXT DEFAULT '',+ conversation_id TEXT DEFAULT '',+ step INTEGER DEFAULT 1,+ kind TEXT DEFAULT 'sequence',+ to_email TEXT DEFAULT '',+ subject TEXT DEFAULT '',+ body TEXT DEFAULT '',+ status TEXT DEFAULT 'pending_review',+ send_at TIMESTAMP,+ sent_at TIMESTAMP,+ provider TEXT DEFAULT '',+ message_id TEXT DEFAULT '',+ thread_ref TEXT DEFAULT '',+ in_reply_to TEXT DEFAULT '',+ error TEXT DEFAULT '',+ created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,+ updated_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP+ );++ CREATE UNIQUE INDEX IF NOT EXISTS uq_outbox_campaign_step+ ON outbox(campaign_id, prospect_id, step)+ WHERE campaign_id != '' AND kind = 'sequence';+ CREATE INDEX IF NOT EXISTS idx_outbox_status_send_at ON outbox(status, send_at);+ CREATE INDEX IF NOT EXISTS idx_outbox_prospect ON outbox(prospect_id);++ -- Simple key/value store for operational flags (kill switch, counters).+ CREATE TABLE IF NOT EXISTS settings (+ key TEXT PRIMARY KEY,+ value TEXT DEFAULT '',+ updated_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP+ );+ """,]# Column whitelists for dynamic UPDATEs (prevents SQL injection via kwargs).@@ -600,6 +643,180 @@ class StateManager:)await db.commit()+ # ── Outbox (approval ladder + native sending) ──++ async def add_outbox_item(+ self,+ *,+ prospect_id: str,+ to_email: str,+ subject: str,+ body: str,+ send_at: str,+ status: str = "pending_review",+ campaign_id: str = "",+ conversation_id: str = "",+ step: int = 1,+ kind: str = "sequence",+ provider: str = "",+ thread_ref: str = "",+ in_reply_to: str = "",+ ) -> str | None:+ """Queue one outgoing email. Returns its id, or None when the+ (campaign, prospect, step) slot already exists — the double-send guard."""+ item_id = _new_id()+ async with self._connect() as db:+ cursor = await db.execute(+ """INSERT OR IGNORE INTO outbox+ (id, campaign_id, prospect_id, conversation_id, step, kind,+ to_email, subject, body, status, send_at, provider,+ thread_ref, in_reply_to)+ VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)""",+ (+ item_id, campaign_id, prospect_id, conversation_id,+ int(step), kind, _norm(to_email), subject, body,+ status, send_at, provider, thread_ref, in_reply_to,+ ),+ )+ await db.commit()+ return item_id if cursor.rowcount > 0 else None++ async def get_outbox(+ self,+ status: str | None = None,+ due_before: str | None = None,+ limit: int = 200,+ ) -> list[dict]:+ where, params = [], []+ if status:+ where.append("status = ?")+ params.append(status)+ if due_before:+ where.append("(send_at IS NULL OR send_at <= ?)")+ params.append(due_before)+ sql = "SELECT * FROM outbox"+ if where:+ sql += " WHERE " + " AND ".join(where)+ sql += " ORDER BY send_at ASC, created_at ASC LIMIT ?"+ params.append(int(limit))+ async with self._connect() as db:+ db.row_factory = aiosqlite.Row+ async with db.execute(sql, params) as cursor:+ return [dict(r) for r in await cursor.fetchall()]++ async def get_outbox_item(self, item_id: str) -> dict | None:+ async with self._connect() as db:+ db.row_factory = aiosqlite.Row+ async with db.execute(+ "SELECT * FROM outbox WHERE id = ?", (item_id,)+ ) as cursor:+ row = await cursor.fetchone()+ return dict(row) if row else None++ _OUTBOX_COLUMNS = frozenset({+ "status", "error", "message_id", "thread_ref", "sent_at",+ "subject", "body", "send_at", "provider",+ })++ async def update_outbox_item(self, item_id: str, **kwargs):+ fields = {k: v for k, v in kwargs.items() if k in self._OUTBOX_COLUMNS}+ if not fields:+ return+ sets = ", ".join(f"{k} = ?" for k in fields)+ async with self._connect() as db:+ await db.execute(+ f"UPDATE outbox SET {sets}, updated_at = ? WHERE id = ?",+ (*fields.values(), _utcnow().isoformat(), item_id),+ )+ await db.commit()++ async def approve_outbox(self, item_id: str | None = None) -> int:+ """Approve one pending item, or ALL pending when item_id is None."""+ async with self._connect() as db:+ if item_id:+ cursor = await db.execute(+ "UPDATE outbox SET status = 'approved', updated_at = ? "+ "WHERE id = ? AND status = 'pending_review'",+ (_utcnow().isoformat(), item_id),+ )+ else:+ cursor = await db.execute(+ "UPDATE outbox SET status = 'approved', updated_at = ? "+ "WHERE status = 'pending_review'",+ (_utcnow().isoformat(),),+ )+ await db.commit()+ return cursor.rowcount++ async def cancel_pending_outbox_for_prospect(+ self, prospect_id: str, reason: str = "stop_on_reply"+ ) -> int:+ """Stop-on-reply: cancel everything queued for a prospect who replied."""+ async with self._connect() as db:+ cursor = await db.execute(+ "UPDATE outbox SET status = 'cancelled', error = ?, updated_at = ? "+ "WHERE prospect_id = ? AND status IN ('pending_review', 'approved')",+ (reason, _utcnow().isoformat(), prospect_id),+ )+ await db.commit()+ return cursor.rowcount++ async def count_outbox_sent(self) -> int:+ async with self._connect() as db:+ async with db.execute(+ "SELECT COUNT(*) FROM outbox WHERE status = 'sent'"+ ) as cursor:+ return (await cursor.fetchone())[0]++ async def count_outbox_sent_today(self) -> int:+ async with self._connect() as db:+ async with db.execute(+ "SELECT COUNT(*) FROM outbox WHERE status = 'sent' "+ "AND date(sent_at) = date('now')"+ ) as cursor:+ return (await cursor.fetchone())[0]++ async def find_outbox_by_message_id(self, message_id: str) -> dict | None:+ if not message_id:+ return None+ async with self._connect() as db:+ db.row_factory = aiosqlite.Row+ async with db.execute(+ "SELECT * FROM outbox WHERE message_id = ?", (message_id,)+ ) as cursor:+ row = await cursor.fetchone()+ return dict(row) if row else None++ # ── Settings (operational flags: kill switch, counters) ──++ async def get_setting(self, key: str, default: str = "") -> str:+ async with self._connect() as db:+ async with db.execute(+ "SELECT value FROM settings WHERE key = ?", (key,)+ ) as cursor:+ row = await cursor.fetchone()+ return row[0] if row else default++ async def set_setting(self, key: str, value: str):+ async with self._connect() as db:+ await db.execute(+ """INSERT INTO settings (key, value, updated_at)+ VALUES (?, ?, CURRENT_TIMESTAMP)+ ON CONFLICT(key) DO UPDATE SET+ value = excluded.value, updated_at = CURRENT_TIMESTAMP""",+ (key, str(value)),+ )+ await db.commit()++ async def increment_setting(self, key: str, by: int = 1) -> int:+ current = await self.get_setting(key, "0")+ try:+ value = int(current) + by+ except ValueError:+ value = by+ await self.set_setting(key, str(value))+ return value+async def prospect_exists(self, email: str = "", linkedin_url: str = "",first_name: str = "", last_name: str = "", company: str = "",@@ -771,6 +988,19 @@ class StateManager:await db.commit()return convo.id+ async def get_conversation(self, convo_id: str) -> Conversation | None:+ async with self._connect() as db:+ db.row_factory = aiosqlite.Row+ async with db.execute(+ "SELECT * FROM conversations WHERE id = ?", (convo_id,)+ ) as cursor:+ row = await cursor.fetchone()+ if not row:+ return None+ d = dict(row)+ d["thread"] = Conversation.thread_from_json(d.pop("thread_json"))+ return Conversation(**d)+async def get_conversations_by_status(self, status: str) -> list[Conversation]:async with self._connect() as db:db.row_factory = aiosqlite.Rowdiff --git a/tests/test_outbox_native.py b/tests/test_outbox_native.pynew file mode 100644index 0000000..6e56607--- /dev/null+++ b/tests/test_outbox_native.py@@ -0,0 +1,351 @@+"""Tests for the native sending path: gate, outbox lifecycle, stage/drain,+stop-on-reply, bounces, and the kill switch."""++import os+import tempfile+from datetime import datetime, timedelta, timezone++import pytest+import pytest_asyncio++from harvey.gate import pre_send_check+from harvey.models.campaign import Campaign, EmailStep+from harvey.models.prospect import Prospect+from harvey.state import StateManager+from harvey.integrations.mail_provider import (+ InboundMessage,+ MailProvider,+ SendResult,+ looks_like_bounce,+)+++def _now_iso():+ return datetime.now(timezone.utc).replace(tzinfo=None).isoformat()+++@pytest_asyncio.fixture+async def state():+ with tempfile.TemporaryDirectory() as tmpdir:+ sm = StateManager(os.path.join(tmpdir, "test.db"))+ await sm.init_db()+ yield sm+++class FakeProvider(MailProvider):+ name = "fake"++ def __init__(self):+ self.sent = []+ self.inbound = []+ self.fail_next = False++ def is_configured(self):+ return True++ async def send_email(self, to_email, subject, body, thread_ref="", in_reply_to=""):+ if self.fail_next:+ self.fail_next = False+ return SendResult(ok=False, error="smtp boom")+ self.sent.append({"to": to_email, "subject": subject, "body": body,+ "thread_ref": thread_ref, "in_reply_to": in_reply_to})+ return SendResult(ok=True, message_id=f"<m{len(self.sent)}@x>", thread_ref="t1")++ async def get_replies(self, limit=50):+ return self.inbound++ async def test_connection(self):+ return True, "fake"+++class Cfg:+ class persona:+ name = "Harvey"; email = "[email protected]"; company = "X"; role = "BD"; tone = "direct"++ class product:+ name = "P"; description = "d"; pricing = "$"; key_benefits = ["b"]+ objection_responses = {}++ class channels:+ class email:+ enabled = True+ provider = "smtp"+ max_daily_sends = 50+ send_to_risky = False+ require_approval = True+ max_bounce_rate = 0.05++ class linkedin:+ enabled = False+++class Env:+ instantly_api_key = ""+++def make_sender(state, provider, require_approval=True):+ from harvey.agents.sender import Sender+ cfg = Cfg()+ cfg.channels.email.require_approval = require_approval+ sender = Sender(brain=None, state=state, config=cfg, env=Env())+ sender.provider = provider # override registry lookup+ sender.send_pacing = False # no inter-send sleeps in tests+ return sender+++class StubBrain:+ """Minimal brain: canned intent + reply text, no subprocess."""+ def __init__(self, intent="question", reply="Sure, happy to help."):+ self._intent = intent+ self._reply = reply++ def load_skills_for_agent(self, *_):+ return ""++ def load_prompt(self, *_a, **_k):+ return ""++ async def think(self, prompt, session_id=None, agent="", task=""):+ return self._intent if task == "classify_intent" or "category" in prompt.lower() else self._reply+++def make_handler(state, provider, intent="question"):+ from harvey.agents.handler import Handler+ handler = Handler(brain=StubBrain(intent=intent), state=state, config=Cfg(), env=Env())+ handler.provider = provider+ return handler+++async def seed_prospect(state, email="[email protected]", status="new", estatus="verified"):+ return await state.add_prospect(Prospect(+ first_name="Jane", last_name="Doe", title="VP", company="Acme",+ email=email, email_status=estatus, email_verified=(estatus == "verified"),+ status=status,+ ))+++async def seed_campaign(state, prospect_ids):+ campaign = Campaign(+ id="", name="test-campaign", channel="email",+ sequence=[+ EmailStep(step=1, subject="hi {{first_name}}", body="Body for {{company}}. Question?", delay_days=0),+ EmailStep(step=2, subject="follow", body="Another angle {{first_name}}.", delay_days=3),+ ],+ prospect_ids=prospect_ids, status="draft",+ )+ campaign.id = await state.add_campaign(campaign)+ return campaign+++# ── gate ──+++def test_gate_blocks_bad_content():+ p = Prospect(first_name="J", last_name="D", title="VP",+ email="[email protected]", email_status="verified")+ ok = pre_send_check("[email protected]", "hello", "Short and fine. Question?", p)+ assert ok.ok++ bad = pre_send_check("[email protected]", "hello", "Hi {{first_name}}, act now!", p)+ assert not bad.ok+ assert any("merge" in r for r in bad.reasons)+ assert any("banned" in r for r in bad.reasons)++ guess = Prospect(first_name="J", last_name="D", title="VP",+ email="[email protected]", email_status="guess")+ blocked = pre_send_check("[email protected]", "hello", "Fine body.", guess)+ assert not blocked.ok++ # Replies to someone who emailed us are exempt from the status gate+ reply_ok = pre_send_check("[email protected]", "Re: hi", "Thanks!", guess, kind="reply")+ assert reply_ok.ok+++def test_gate_link_and_html_limits():+ r = pre_send_check("[email protected]", "s", "See https://a.com and https://b.com")+ assert any("links" in x for x in r.reasons)+ r2 = pre_send_check("[email protected]", "s", "<div>hello</div>")+ assert any("HTML" in x for x in r2.reasons)+++def test_bounce_detection():+ assert looks_like_bounce("[email protected]", "anything")+ assert looks_like_bounce("[email protected]", "Delivery Status Notification (Failure)")+ assert not looks_like_bounce("[email protected]", "Re: your note")+++# ── staging ──+++async def test_stage_campaign_renders_and_respects_approval(state):+ pid = await seed_prospect(state)+ campaign = await seed_campaign(state, [pid])+ sender = make_sender(state, FakeProvider(), require_approval=True)++ await sender._run_native()++ pending = await state.get_outbox(status="pending_review")+ assert len(pending) == 2 # both steps staged+ step1 = next(i for i in pending if i["step"] == 1)+ assert step1["subject"] == "hi Jane"+ assert "Acme" in step1["body"]+ # Nothing sent while awaiting approval+ assert sender.provider.sent == []++ # Idempotent: re-running stages nothing new+ campaign2 = await state.get_campaigns_by_status("active")+ assert campaign2[0].id == campaign.id+ await sender._run_native()+ assert len(await state.get_outbox(status="pending_review")) == 2+++async def test_stage_skips_unverified(state):+ pid = await seed_prospect(state, email="[email protected]", estatus="guess")+ await seed_campaign(state, [pid])+ sender = make_sender(state, FakeProvider())+ await sender._run_native()+ assert await state.get_outbox(status="pending_review") == []+++# ── drain ──+++async def test_drain_sends_approved_due_items(state):+ pid = await seed_prospect(state)+ await seed_campaign(state, [pid])+ provider = FakeProvider()+ sender = make_sender(state, provider, require_approval=False)++ await sender._run_native() # stage as approved + drain in one pass++ sent = await state.get_outbox(status="sent")+ assert len(sent) == 1 # step 1 due now; step 2 in 3 days+ assert provider.sent[0]["to"] == "[email protected]"+ assert provider.sent[0]["subject"] == "hi Jane"+ prospect = await state.get_prospect(pid)+ assert prospect.status == "contacted"+ approved_left = await state.get_outbox(status="approved")+ assert len(approved_left) == 1 and approved_left[0]["step"] == 2+++async def test_drain_respects_kill_switch(state):+ pid = await seed_prospect(state)+ await seed_campaign(state, [pid])+ provider = FakeProvider()+ sender = make_sender(state, provider, require_approval=False)+ await state.set_setting("sending_paused", "test pause")++ await sender._run_native()+ assert provider.sent == []+ assert await state.get_outbox(status="sent") == []+++async def test_drain_gate_failure_marks_failed(state):+ pid = await seed_prospect(state)+ await state.add_outbox_item(+ prospect_id=pid, to_email="[email protected]",+ subject="hello", body="Unrendered {{first_name}} tag",+ send_at=_now_iso(), status="approved", campaign_id="c1", step=1,+ )+ provider = FakeProvider()+ sender = make_sender(state, provider)+ await sender._drain_due()+ failed = await state.get_outbox(status="failed")+ assert len(failed) == 1+ assert "gate" in failed[0]["error"]+ assert provider.sent == []+++async def test_send_failure_recorded(state):+ pid = await seed_prospect(state)+ await state.add_outbox_item(+ prospect_id=pid, to_email="[email protected]",+ subject="hello", body="Fine body. Question?",+ send_at=_now_iso(), status="approved", campaign_id="c1", step=1,+ )+ provider = FakeProvider()+ provider.fail_next = True+ sender = make_sender(state, provider)+ await sender._drain_due()+ failed = await state.get_outbox(status="failed")+ assert failed and "smtp boom" in failed[0]["error"]+++# ── handler: stop-on-reply, bounces, kill switch ──+++async def test_reply_cancels_pending_outbox(state):+ pid = await seed_prospect(state, status="contacted")+ await state.add_outbox_item(+ prospect_id=pid, to_email="[email protected]", subject="s2",+ body="b", send_at=_now_iso(), status="approved", campaign_id="c1", step=2,+ )+ provider = FakeProvider()+ provider.inbound = [InboundMessage(+ provider_id="in1", from_email="[email protected]",+ subject="Re: hi", body="unsubscribe please",+ )]+ handler = make_handler(state, provider)+ await handler._run_native()++ cancelled = await state.get_outbox(status="cancelled")+ assert len(cancelled) == 1+ prospect = await state.get_prospect(pid)+ assert prospect.status == "opted_out" # opt-out keywords honored+++async def test_bounce_marks_invalid_and_kill_switch(state):+ pid = await seed_prospect(state, status="contacted")+ # A sent item whose message-id the bounce references+ item_id = await state.add_outbox_item(+ prospect_id=pid, to_email="[email protected]", subject="s",+ body="b", send_at=_now_iso(), status="approved", campaign_id="c1", step=1,+ )+ await state.update_outbox_item(+ item_id, status="sent", message_id="<m1@x>", sent_at=_now_iso()+ )+ # Ten more sent items so the kill-switch minimum-sample rule is met+ for i in range(10):+ oid = await state.add_outbox_item(+ prospect_id=pid, to_email="[email protected]", subject="s", body="b",+ send_at=_now_iso(), status="approved", campaign_id=f"c{i+2}", step=1,+ )+ await state.update_outbox_item(oid, status="sent", sent_at=_now_iso())++ provider = FakeProvider()+ provider.inbound = [InboundMessage(+ provider_id="b1", from_email="[email protected]",+ subject="Delivery Status Notification (Failure)",+ body="couldn't be delivered", in_reply_to="<m1@x>", is_bounce=True,+ )]+ handler = make_handler(state, provider)+ await handler._run_native()++ prospect = await state.get_prospect(pid)+ assert prospect.email_status == "invalid"+ # 1 bounce / 11 sent = 9% > 5% threshold with >= 10 sends → kill switch+ assert await state.get_setting("sending_paused") != ""+++async def test_inbound_dedup(state):+ await seed_prospect(state, status="contacted")+ provider = FakeProvider()+ provider.inbound = [InboundMessage(+ provider_id="in1", from_email="[email protected]",+ subject="Re: hi", body="we're all set thanks",+ )]+ handler = make_handler(state, provider, intent="not_interested")+ await handler._run_native()+ first = await state.get_conversations_by_status("closed")+ await handler._run_native() # same message again → ignored via dedup+ second = await state.get_conversations_by_status("closed")+ assert len(first) == len(second) == 1