Skip to content

A curated collection of practical AI projects implementing OCR systems, RAG, AI agents, and other AI use cases.

Add customer support voice agent tutorial (Telnyx AI Assistant Builder)

Sumanth077014292df314e
diff --git a/voice_apps/customer_support_voice_agent/.env.example b/voice_apps/customer_support_voice_agent/.env.example
new file mode 100644
index 0000000..f453269
--- /dev/null
+++ b/voice_apps/customer_support_voice_agent/.env.example
@@ -0,0 +1,8 @@
+# Telnyx API credentials
+# Get from portal.telnyx.com > API Keys
+TELNYX_API_KEY=your_telnyx_api_key_here
+TELNYX_PUBLIC_KEY=your_ed25519_public_key_here
+
+# Server config
+HOST=127.0.0.1
+PORT=8000
diff --git a/voice_apps/customer_support_voice_agent/.gitignore b/voice_apps/customer_support_voice_agent/.gitignore
new file mode 100644
index 0000000..65de21a
--- /dev/null
+++ b/voice_apps/customer_support_voice_agent/.gitignore
@@ -0,0 +1,9 @@
+.env
+__pycache__/
+*.pyc
+*.pyo
+.venv/
+venv/
+*.egg-info/
+dist/
+.DS_Store
diff --git a/voice_apps/customer_support_voice_agent/README.md b/voice_apps/customer_support_voice_agent/README.md
new file mode 100644
index 0000000..0c90af8
--- /dev/null
+++ b/voice_apps/customer_support_voice_agent/README.md
@@ -0,0 +1,284 @@
+# Customer Support Voice Agent (Telnyx AI Assistant Builder)
+
+![Demo](assets/demo.gif)
+
+A voice AI agent that answers phone calls and handles customer support conversations. Configured in the Telnyx portal (no-code) with a FastAPI webhook that injects real-time context into every call.
+
+## Overview
+
+This project uses the **Telnyx AI Assistant Builder** to configure the agent, model, voice, system prompt and phone number entirely in the Telnyx portal. Our FastAPI server contributes a **Dynamic Variables webhook**: Telnyx calls it at the start of every call, and we return live data (system status, queue wait time, business hours) that gets injected into the agent's system prompt as `{{variable}}` placeholders.
+
+**What Telnyx portal handles:** STT, LLM inference, TTS, phone number routing, conversation history, embeddable browser widget.
+
+**What our code handles:** real-time context injection via Dynamic Variables webhook.
+
+## How It Works
+
+![How It Works](assets/how_it_works.png)
+
+## Tech Stack
+
+| Layer | Tool |
+|---|---|
+| Agent configuration | Telnyx AI Assistant Builder (no-code portal) |
+| LLM | `moonshotai/Kimi-K2.5` via Telnyx (no external API key) |
+| STT | Telnyx native |
+| TTS | Telnyx native |
+| Context webhook | FastAPI + Uvicorn |
+| Browser demo | Telnyx embeddable widget (from portal Widget tab) |
+| Package manager | uv |
+
+## Prerequisites
+
+- Python 3.10 or higher
+- [uv](https://docs.astral.sh/uv/) (`pip install uv`)
+- [ngrok](https://ngrok.com) to expose your webhook during development
+- A [Telnyx account](https://portal.telnyx.com/sign-up) with a funded balance
+
+## Telnyx Portal Setup
+
+All of this is done at [portal.telnyx.com](https://portal.telnyx.com).
+
+### Step 1: Get your API credentials
+
+- Copy your **API Key** (starts with `KEY...`)
+- Click the **Public Key** tab on the same page and copy your **Ed25519 Public Key**
+
+### Step 2: Buy a phone number
+
+- Search for a US number and purchase it
+
+### Step 3: Create your AI Assistant
+
+- Left sidebar → **AI** section → **AI Assistants** → **Create New**
+- Choose **Blank Template**
+- Fill in the **Instructions** field (paste the system prompt below):
+
+```
+You are a customer support agent for a full-service retail bank serving personal
+and business customers.
+
+Keep all responses under 2 sentences -- callers are listening, not reading.
+Speak calmly and professionally. If a question is not covered in the knowledge
+base below, acknowledge it warmly and offer to transfer the caller to a specialist.
+
+System status: {{system_status}}
+Active incidents: {{todays_incidents}}
+Queue wait: {{current_queue_wait}}
+Business hours: {{business_hours}}
+Support email: {{support_email}}
+
+--- KNOWLEDGE BASE ---
+
+ACCOUNTS
+We offer three personal account tiers. The Everyday Checking account carries no
+minimum balance, includes a free debit card and online bill pay, and reimburses up
+to $10 in ATM fees each month. The Premier Checking account charges a $12 monthly
+fee that is waived when the customer maintains a $2,500 minimum daily balance or
+receives a recurring direct deposit of at least $1,000 per month; it also earns
+0.05% APY. The Savings account currently earns 4.25% APY with no minimum balance
+required; customers may make up to 6 withdrawals per month before a $5 excess
+withdrawal fee applies.
+
+DEBIT AND CREDIT CARDS
+Customers who lose a card or believe it has been stolen should report it immediately
+by calling the 24-hour card hotline at 1-800-555-0199 or by freezing the card
+instantly through Card Controls in the mobile app. A replacement card arrives within
+5 to 7 business days. Transaction disputes must be filed within 60 days of the
+statement date and are typically resolved within 10 business days. Temporary card
+freezes and unfreezes are available at any time in the mobile app -- no call required.
+
+TRANSFERS AND PAYMENTS
+Outgoing domestic wire transfers submitted before 3:00 PM EST on a business day are
+processed the same day; transfers submitted after the cut-off are processed the
+following business day. The fee is $25 for outgoing and $15 for incoming domestic
+wires. Zelle transfers between enrolled customers typically arrive within minutes.
+ACH transfers to external accounts take 1 to 3 business days. Customers can link up
+to 5 external accounts under Transfers in Online Banking.
+
+LOANS
+Personal loans are available from $2,000 to $50,000 with fixed APRs between 7.99%
+and 24.99% based on credit profile and term length. Auto loans for new and used
+vehicles start at 5.49% APR for 60-month terms. Mortgage pre-qualification is
+available online and takes approximately 2 business days. All loan applications can
+be started on our website or at any branch.
+
+ONLINE BANKING AND MOBILE APP
+Customers who forget their Online Banking password can reset it on our website by
+selecting Forgot Password; a verification code is sent to the registered email or
+phone number. After 5 failed login attempts the account is locked and must be
+unlocked by calling support or visiting a branch with a valid government-issued ID.
+The mobile app supports mobile check deposit, card controls, spending insights, and
+Zelle. Checks deposited before 8:00 PM EST on a business day are available the
+next business day.
+
+SECURITY AND FRAUD
+We will never ask for a full Social Security number, password, or one-time passcode
+over the phone or by text. Customers who suspect fraud should call the 24-hour fraud
+line at 1-800-555-0911 immediately. Accounts flagged for suspicious activity are
+frozen automatically and the customer is notified by both text and email. Two-factor
+authentication is strongly recommended and can be enabled under Security Settings in
+Online Banking.
+
+FEES
+Monthly maintenance fees are $0 for Everyday Checking, $12 for Premier Checking
+(waived with qualifying activity), and $0 for the Savings account. Overdraft
+protection transfers cost $10 per transfer. Stop payment requests are $30 per item.
+Paper statement delivery costs $3 per month; electronic statements are always free.
+There is no fee for incoming ACH transfers or standard online bill payments.
+
+BRANCHES AND ATMS
+We operate 47 branches across the Northeast, open Monday through Friday 9 AM to
+5 PM EST and Saturday 9 AM to 1 PM EST. Customers have surcharge-free access to
+over 55,000 ATMs in the Allpoint network nationwide. Branch and ATM locations are
+available on our website or through the mobile app.
+```
+
+- Set the **Greeting**:
+
+```
+Hi, thanks for calling customer support. How can I help you today?
+```
+
+- Under **Agent** tab: set Model to `moonshotai/Kimi-K2.5`
+- Leave Voice settings as default for now
+- Click **Save**
+
+### Step 4: Assign your phone number
+
+After creation, assign the number you bought in Step 2 to this assistant.
+
+### Step 5: Test directly in the portal
+
+Inside your assistant, click the **Test** button. The portal has a built-in web call interface, no phone, no ngrok needed. Speak to your agent right there.
+
+### Step 6: Set the Dynamic Variables webhook URL
+
+In your assistant settings, look for the **Dynamic Variables** section:
+
+- Set the webhook URL to: `https://your-ngrok-url.ngrok.io/webhooks/dynamic-variables`
+- You will update this after running ngrok in the setup steps below
+
+### Step 7: Open the browser widget demo
+
+`demo.html` already contains the live widget embed. Open it in any browser, no server needed, it loads the widget directly from Telnyx's CDN and connects to the assistant via WebRTC.
+
+## Installation
+
+### 1. Clone the repository
+
+```bash
+git clone https://github.com/Sumanth077/Hands-On-AI-Engineering.git
+cd Hands-On-AI-Engineering/voice_apps/saas_customer_support_voice_agent
+```
+
+### 2. Create virtual environment
+
+```bash
+uv venv
+source .venv/bin/activate # Windows: .venv\Scripts\activate
+```
+
+### 3. Install dependencies
+
+```bash
+uv pip install -e .
+```
+
+### 4. Set up environment variables
+
+```bash
+cp .env.example .env
+```
+
+Edit `.env`:
+
+```bash
+TELNYX_API_KEY=KEYxxxxxxxxxxxxxxxx
+TELNYX_PUBLIC_KEY=your_ed25519_public_key_here
+```
+
+## Running the Webhook Server
+
+```bash
+python main.py
+```
+
+Server starts at `http://127.0.0.1:8000`.
+
+In a separate terminal, expose it with ngrok:
+
+```bash
+ngrok http 8000
+```
+
+Copy the HTTPS URL (e.g. `https://abc123.ngrok.io`) and update the Dynamic Variables webhook URL in your Telnyx AI Assistant to `https://abc123.ngrok.io/webhooks/dynamic-variables`.
+
+Health check:
+
+```bash
+curl http://localhost:8000/health
+```
+
+## Project Structure
+
+```
+customer_support_voice_agent/
+├── main.py # FastAPI server -- Dynamic Variables webhook
+├── demo.html # Browser demo -- open in any browser to try the agent via WebRTC
+├── pyproject.toml # uv dependencies
+├── .env.example # Environment variable template
+├── .gitignore
+└── README.md
+```
+
+## Customising the Dynamic Variables
+
+Open `main.py` and update `get_live_context()` to pull real data from your systems:
+
+```python
+def get_live_context() -> dict:
+ # Replace with real lookups:
+ return {
+ "system_status": fetch_status_page(), # your status API
+ "current_queue_wait": fetch_queue_wait(), # your support queue
+ "business_hours": "Mon-Fri, 9AM-6PM EST",
+ "support_email": "[email protected]",
+ "todays_incidents": fetch_incidents(), # PagerDuty, OpsGenie
+ }
+```
+
+## Demo
+
+The primary way to interact with this agent is by calling the Telnyx phone number you assigned in Step 4. The caller dials in, the Dynamic Variables webhook fires, and the AI handles the conversation end-to-end -- STT, LLM, TTS, all through Telnyx.
+
+For this tutorial we also ship `demo.html`, a browser widget that connects to the same assistant over WebRTC so you can try it without making a phone call. See the [Telnyx AI Agent Widget docs](https://developers.telnyx.com/docs/inference/ai-assistants/ai-agent-widget) for details on how the embed works.
+
+Open `demo.html` in any browser -- no server needed.
+
+### Try asking
+
+The demo uses a fictional bank as the support persona. All of the following questions are grounded in the knowledge base in the Instructions field:
+
+- "I think my debit card was stolen -- what should I do?"
+- "What is the current savings account interest rate?"
+- "How long does a wire transfer take, and what does it cost?"
+- "My online banking account is locked. How do I get back in?"
+- "How do I dispute a charge on my account?"
+- "Can I freeze my card from the app without calling?"
+- "What are your branch hours?"
+
+### How the knowledge base works
+
+The knowledge base lives inline in the assistant's **Instructions** field in the Telnyx portal -- no separate KB service, no vector database, no retrieval pipeline. The model reads the full KB on every call and uses it to answer questions accurately.
+
+This is the right approach for demos and small-to-medium support playbooks (under ~8000 tokens). For production at scale, Telnyx also supports a dedicated Knowledge Bases feature where you upload documents and the assistant retrieves relevant chunks automatically.
+
+## Resources
+
+- [Telnyx AI Assistant Builder docs](https://developers.telnyx.com/docs/inference/ai-assistants/no-code-voice-assistant)
+- [Dynamic Variables docs](https://developers.telnyx.com/docs/inference/ai-assistants/dynamic-variables)
+- [Available models](https://developers.telnyx.com/docs/inference/models) -- `moonshotai/Kimi-K2.5` is the recommended balance of intelligence and cost
+- [Telnyx Portal](https://portal.telnyx.com)
+
+[Back to Top](#saas-customer-support-voice-agent-telnyx-ai-assistant-builder)
diff --git a/voice_apps/customer_support_voice_agent/assets/demo.gif b/voice_apps/customer_support_voice_agent/assets/demo.gif
new file mode 100644
index 0000000..e185945
Binary files /dev/null and b/voice_apps/customer_support_voice_agent/assets/demo.gif differ
diff --git a/voice_apps/customer_support_voice_agent/assets/how_it_works.png b/voice_apps/customer_support_voice_agent/assets/how_it_works.png
new file mode 100644
index 0000000..11fb6a4
Binary files /dev/null and b/voice_apps/customer_support_voice_agent/assets/how_it_works.png differ
diff --git a/voice_apps/customer_support_voice_agent/demo.html b/voice_apps/customer_support_voice_agent/demo.html
new file mode 100644
index 0000000..668ed1d
--- /dev/null
+++ b/voice_apps/customer_support_voice_agent/demo.html
@@ -0,0 +1,80 @@
+<!DOCTYPE html>
+<html lang="en">
+<head>
+ <meta charset="UTF-8" />
+ <meta name="viewport" content="width=device-width, initial-scale=1.0" />
+ <title>Customer Support Voice Agent</title>
+ <style>
+ *, *::before, *::after { box-sizing: border-box; margin: 0; padding: 0; }
+
+ body {
+ font-family: -apple-system, BlinkMacSystemFont, "Segoe UI", Roboto, sans-serif;
+ background: linear-gradient(135deg, #0f172a 0%, #1e1b4b 100%);
+ min-height: 100vh;
+ display: flex;
+ flex-direction: column;
+ align-items: center;
+ justify-content: center;
+ padding: 40px 24px;
+ }
+
+ .hero {
+ text-align: center;
+ max-width: 480px;
+ margin-bottom: 40px;
+ }
+
+ .badge {
+ display: inline-block;
+ background: rgba(99, 102, 241, 0.15);
+ color: #a5b4fc;
+ font-size: 12px;
+ font-weight: 600;
+ padding: 4px 12px;
+ border-radius: 999px;
+ margin-bottom: 16px;
+ letter-spacing: 0.3px;
+ border: 1px solid rgba(99, 102, 241, 0.25);
+ }
+
+ h1 {
+ font-size: 28px;
+ font-weight: 700;
+ color: #f1f5f9;
+ line-height: 1.3;
+ }
+
+ .widget-wrap {
+ display: flex;
+ justify-content: center;
+ }
+
+ .footer {
+ margin-top: 32px;
+ font-size: 12px;
+ color: #475569;
+ text-align: center;
+ }
+ </style>
+</head>
+<body>
+
+ <div class="hero">
+ <div class="badge">Live Demo</div>
+ <h1>Customer Support Voice Agent</h1>
+ </div>
+
+ <div class="widget-wrap">
+ <telnyx-ai-agent
+ agent-id="assistant-82094bd1-1a27-42b0-9dbb-52c56d2fbe62"
+ environment="production">
+ </telnyx-ai-agent>
+ <script async src="https://unpkg.com/@telnyx/ai-agent-widget@next"></script>
+ </div>
+
+ <p class="footer">
+ Built with Telnyx AI Assistant Builder &middot; FastAPI Dynamic Variables Webhook &middot; Kimi K2.5
+ </p>
+
+</body>
+</html>
diff --git a/voice_apps/customer_support_voice_agent/main.py b/voice_apps/customer_support_voice_agent/main.py
new file mode 100644
index 0000000..c2cd70a
--- /dev/null
+++ b/voice_apps/customer_support_voice_agent/main.py
@@ -0,0 +1,108 @@
+#!/usr/bin/env python3
+"""SaaS Customer Support Voice Agent -- Dynamic Variables Webhook.
+
+Telnyx AI Assistant Builder calls this webhook at the start of each call.
+We return real-time context that gets injected into the assistant's system
+prompt as {{variable}} placeholders -- no static hardcoded values needed.
+
+In production, replace the stub values with real database/API lookups.
+"""
+
+import os
+import time
+
+import telnyx
+from dotenv import load_dotenv
+from fastapi import FastAPI, HTTPException, Request
+from fastapi.responses import JSONResponse
+
+load_dotenv()
+
+# ---------------------------------------------------------------------------
+# Config
+# ---------------------------------------------------------------------------
+
+TELNYX_API_KEY: str = os.getenv("TELNYX_API_KEY", "")
+TELNYX_PUBLIC_KEY: str = os.getenv("TELNYX_PUBLIC_KEY", "")
+
+telnyx_client = telnyx.Telnyx(api_key=TELNYX_API_KEY, public_key=TELNYX_PUBLIC_KEY)
+
+# ---------------------------------------------------------------------------
+# App
+# ---------------------------------------------------------------------------
+
+app = FastAPI(
+ title="SaaS Customer Support -- Dynamic Variables Webhook",
+ description="Injects real-time SaaS context into a Telnyx AI Assistant at call start.",
+ version="1.0.0",
+)
+
+# ---------------------------------------------------------------------------
+# Dynamic Variables webhook
+# ---------------------------------------------------------------------------
+
+
+def get_live_context() -> dict:
+ """Return real-time context for the AI assistant system prompt.
+
+ These values replace {{variable}} placeholders in the assistant's
+ Instructions field (configured in the Telnyx portal).
+
+ In production, replace these stubs with real lookups:
+ - system_status -> query your status page API
+ - current_queue_wait -> query your support queue
+ - todays_incidents -> query PagerDuty, OpsGenie, etc.
+ """
+ return {
+ "system_status": "All systems operational",
+ "current_queue_wait": "Under 2 minutes",
+ "business_hours": "Monday to Friday, 9AM to 5PM EST; Saturday, 9AM to 1PM EST",
+ "todays_incidents": "None",
+ "support_email": "[email protected]",
+ }
+
+
[email protected]("/webhooks/dynamic-variables")
+async def dynamic_variables(request: Request) -> JSONResponse:
+ """Telnyx calls this endpoint at the start of each call.
+
+ Telnyx sends a POST with call metadata. We respond with a dict of
+ key/value pairs that are injected into the assistant's system prompt
+ via {{variable}} placeholders.
+
+ Docs: https://developers.telnyx.com/docs/inference/ai-assistants/dynamic-variables
+ """
+ # Verify Ed25519 signature before trusting the request
+ raw_body = await request.body()
+ try:
+ telnyx_client.webhooks.unwrap(raw_body.decode("utf-8"), dict(request.headers))
+ except Exception:
+ raise HTTPException(status_code=401, detail="Invalid webhook signature")
+
+ # Return live context -- Telnyx injects these as {{variable}} in the prompt
+ return JSONResponse(get_live_context())
+
+
+# ---------------------------------------------------------------------------
+# Health check
+# ---------------------------------------------------------------------------
+
+
+async def health() -> JSONResponse:
+ return JSONResponse({"status": "ok", "timestamp": int(time.time())})
+
+
+# ---------------------------------------------------------------------------
+# Entry point
+# ---------------------------------------------------------------------------
+
+if __name__ == "__main__":
+ import uvicorn
+
+ uvicorn.run(
+ "main:app",
+ host=os.getenv("HOST", "127.0.0.1"),
+ port=int(os.getenv("PORT", "8000")),
+ reload=False,
+ )
diff --git a/voice_apps/customer_support_voice_agent/pyproject.toml b/voice_apps/customer_support_voice_agent/pyproject.toml
new file mode 100644
index 0000000..d999546
--- /dev/null
+++ b/voice_apps/customer_support_voice_agent/pyproject.toml
@@ -0,0 +1,11 @@
+[project]
+name = "saas-customer-support-voice-agent"
+version = "1.0.0"
+description = "SaaS Customer Support Voice Agent -- Dynamic Variables webhook for Telnyx AI Assistant Builder"
+requires-python = ">=3.10"
+dependencies = [
+ "fastapi>=0.115.0",
+ "uvicorn[standard]>=0.32.0",
+ "python-dotenv>=1.0.0",
+ "telnyx>=4.0.0",
+]