Skip to content

A curated collection of practical AI projects implementing OCR systems, RAG, AI agents, and other AI use cases.

feat: add Voice GitHub Agent - voice-controlled agent for GitHub repo triage

Tiioluwania3a5c2d8d53c
diff --git a/ai_agents/voice-github-agent/.env.example b/ai_agents/voice-github-agent/.env.example
new file mode 100644
index 0000000..54eb377
--- /dev/null
+++ b/ai_agents/voice-github-agent/.env.example
@@ -0,0 +1,11 @@
+# Get this from https://www.assemblyai.com/dashboard/signup
+# Used for both the Sync API (transcription) and the LLM Gateway (agent).
+ASSEMBLYAI_API_KEY=
+
+# A GitHub personal access token. Needs at least:
+# - classic token: "repo" scope
+# - fine-grained token: Contents (read) + Issues (read & write) on the repo below
+GITHUB_TOKEN=
+
+# The single repo this v1 operates on, as "owner/name", e.g. "octocat/hello-world"
+GITHUB_REPO=
diff --git a/ai_agents/voice-github-agent/README.md b/ai_agents/voice-github-agent/README.md
new file mode 100644
index 0000000..ecf6553
--- /dev/null
+++ b/ai_agents/voice-github-agent/README.md
@@ -0,0 +1,133 @@
+# Voice GitHub Agent
+
+> Speak an instruction, and a tool-calling agent inspects your repo and acts on it.
+
+![Voice GitHub Agent Demo](assets/demo.gif)
+
+## Overview
+
+Voice GitHub Agent turns a spoken instruction into real GitHub actions, so you can check in on a repository without switching to a keyboard. You speak a request, it gets transcribed, and a tool-calling agent decides which GitHub tools to run, from reading commits and diffs to opening issues and posting comments, then reports back what it found and did. It is built for developers who want a fast, hands-free way to triage a repo instead of clicking through commits and issues one at a time.
+
+## Features
+
+- **Real voice input**: click a record button in the browser, speak, click again to stop, no terminal and no typing the instruction
+- **Sync transcription**: AssemblyAI's Sync API returns a finished transcript in the same HTTP call, no polling
+- **A genuine tool-calling agent**: `qwen3-next-80b-a3b` on AssemblyAI's LLM Gateway decides which GitHub tools to call, not a scripted if/else
+- **Real GitHub actions**: reads commits, diffs, and issues, and can open issues and post comments through the GitHub REST API
+- **Turn-limited agent loop**: capped at 8 tool-calling turns so a confused run fails loud instead of looping forever
+- **Formatted markdown output**: the summary and activity feed render through marked.js, so bold text, inline code, and numbered lists show up formatted instead of as raw markdown syntax
+
+## Tech Stack
+
+**Frameworks & Libraries:**
+- Flask
+- openai (Chat Completions client, pointed at AssemblyAI's LLM Gateway)
+- marked.js (markdown rendering in the browser)
+
+**Additional Tools:**
+- AssemblyAI Sync API, transcription (Universal-3.5 Pro)
+- AssemblyAI LLM Gateway, agent model (qwen3-next-80b-a3b)
+- GitHub REST API, commits, diffs, issues, comments
+- Browser MediaRecorder / Web Audio API, mic capture and WAV encoding, no server-side audio libraries needed
+
+## Prerequisites
+
+- Python 3.9 or higher
+- API keys for:
+ - [ ] AssemblyAI (free signup: assemblyai.com/dashboard/signup)
+ - [ ] GitHub personal access token (repo scope, or fine-grained with Contents read + Issues read/write)
+
+## Installation
+
+### 1. Clone Repository
+
+```bash
+git clone https://github.com/Sumanth077/Hands-On-AI-Engineering.git
+cd Hands-On-AI-Engineering/ai_agents/voice-github-agent
+```
+
+### 2. Create Virtual Environment
+
+```bash
+python -m venv venv
+```
+
+Activate it:
+
+```bash
+# Windows
+venv\Scripts\activate
+
+# macOS/Linux
+source venv/bin/activate
+```
+
+### 3. Install Dependencies
+
+```bash
+pip install -r requirements.txt
+```
+
+### 4. Set Up Environment Variables
+
+```bash
+cp .env.example .env
+```
+
+Edit `.env` and add your keys:
+
+```bash
+ASSEMBLYAI_API_KEY=your_assemblyai_api_key_here
+GITHUB_TOKEN=your_github_token_here
+GITHUB_REPO=owner/name
+```
+
+`GITHUB_REPO` is the single repository the agent operates on, as `owner/name`, for example `octocat/hello-world`. Never commit `.env` to version control.
+
+## Usage
+
+### Running the Application
+
+```bash
+python app.py
+```
+
+Then open http://localhost:5000 in your browser.
+
+### Example Usage
+
+**Input:** (spoken) "Check the latest commits and tell me what changed. If anything looks broken, open an issue for it."
+
+**Output:** The transcript appears in the UI styled like text landing in a notes app, followed by an activity feed listing each GitHub action taken in plain language (for example "Checked the last 5 commits" and "Opened issue #42"), followed by a final written summary of what was checked, found, and done.
+
+## Project Structure
+
+```
+voice-github-agent/
+├── app.py # Flask app: serves the UI and the /api/run endpoint
+├── transcribe.py # AssemblyAI Sync API wrapper
+├── agent.py # LLM Gateway client and tool-calling agent loop
+├── github_tools.py # GitHub REST API tool functions and their schemas
+├── templates/
+│ └── index.html # Single-page UI
+├── static/
+│ ├── app.js # Recording, client-side WAV encoding, and rendering the result
+│ └── style.css # Visual styling
+├── prompts/
+│ └── system_prompt.txt # System prompt for the agent
+├── requirements.txt # Python dependencies
+├── .env.example # Environment variables template
+└── README.md # This file
+```
+
+## How It Works
+
+**Technical Details:**
+
+The browser records the microphone with `MediaRecorder`, decodes the result with the Web Audio API, and encodes it as a 16-bit PCM mono WAV entirely client-side in `static/app.js`, since AssemblyAI's Sync API requires WAV and this way no server-side audio library is needed. That WAV is POSTed to `/api/run` in `app.py`.
+
+`transcribe.py` sends the WAV to AssemblyAI's Sync API in a single HTTP request and gets a finished transcript back in the same response, with no job to poll. `app.py` hands that transcript to `agent.py`, which runs the tool-calling loop: it sends the transcript, the system prompt from `prompts/system_prompt.txt`, and the GitHub tool schemas to the LLM Gateway on `qwen3-next-80b-a3b`. Whenever the model returns tool calls instead of a final answer, the loop executes them for real against `github_tools.py`, feeds the results back as tool messages, and asks again, up to 8 turns. `run_agent()` returns both the final summary and the full list of tool calls made along the way.
+
+`app.py` returns `{ transcript, tool_calls, summary }` as JSON. `static/app.js` renders the transcript first, then a plain-language activity feed built from the tool calls, then the summary. The summary and activity feed text are escaped before being parsed as markdown with marked.js, so formatting like bold text and numbered lists renders correctly while any injected HTML in the model's output is neutralized instead of executed.
+
+[⬆ Back to Top](#voice-github-agent)
diff --git a/ai_agents/voice-github-agent/agent.py b/ai_agents/voice-github-agent/agent.py
new file mode 100644
index 0000000..c8e7ac6
--- /dev/null
+++ b/ai_agents/voice-github-agent/agent.py
@@ -0,0 +1,94 @@
+"""Agent loop: sends the transcript + tool schema to AssemblyAI's LLM
+Gateway on qwen3-next-80b-a3b (the model in the gateway's lineup that
+supports tool calling / agentic workflows), and executes whichever GitHub
+tools the model decides to call, until it returns a final answer.
+
+Docs:
+ https://www.assemblyai.com/docs/llm-gateway/quickstart
+ https://www.assemblyai.com/docs/llm-gateway/tool-calling
+ https://www.assemblyai.com/docs/llm-gateway/agentic-workflows
+"""
+import json
+import os
+
+from openai import OpenAI
+
+from github_tools import TOOL_FUNCTIONS, TOOL_SCHEMAS
+
+MODEL = "qwen3-next-80b-a3b"
+MAX_TURNS = 8 # safety cap so a confused model can't loop forever
+
+_PROMPT_PATH = os.path.join(os.path.dirname(__file__), "prompts", "system_prompt.txt")
+with open(_PROMPT_PATH, encoding="utf-8") as f:
+ SYSTEM_PROMPT = f.read()
+
+
+def _client() -> OpenAI:
+ api_key = os.environ.get("ASSEMBLYAI_API_KEY")
+ if not api_key:
+ raise RuntimeError("ASSEMBLYAI_API_KEY is not set. Check your .env file.")
+ return OpenAI(base_url="https://llm-gateway.assemblyai.com/v1", api_key=api_key)
+
+
+def run_agent(spoken_instruction: str) -> dict:
+ """Runs the tool-calling loop. Returns a dict with the agent's final
+ summary text and the list of tool calls made along the way, each as
+ {"name": str, "args": dict, "result": Any}."""
+ client = _client()
+
+ messages = [
+ {"role": "system", "content": SYSTEM_PROMPT},
+ {"role": "user", "content": spoken_instruction},
+ ]
+ tool_call_log = []
+
+ for _ in range(MAX_TURNS):
+ response = client.chat.completions.create(
+ model=MODEL,
+ messages=messages,
+ tools=TOOL_SCHEMAS,
+ max_tokens=2000,
+ )
+ message = response.choices[0].message
+ tool_calls = message.tool_calls
+
+ if not tool_calls:
+ return {
+ "summary": message.content or "(no summary returned)",
+ "tool_calls": tool_call_log,
+ }
+
+ # Record the assistant's tool-call request in the conversation history.
+ messages.append(message.model_dump(exclude_none=True))
+
+ for call in tool_calls:
+ name = call.function.name
+ try:
+ args = json.loads(call.function.arguments or "{}")
+ except json.JSONDecodeError:
+ args = {}
+
+ print(f" -> calling tool: {name}({args})")
+ fn = TOOL_FUNCTIONS.get(name)
+ if fn is None:
+ result = {"error": f"Unknown tool '{name}'"}
+ else:
+ try:
+ result = fn(**args)
+ except Exception as exc: # tool errors get fed back to the model, not raised
+ result = {"error": str(exc)}
+
+ tool_call_log.append({"name": name, "args": args, "result": result})
+
+ messages.append(
+ {
+ "role": "tool",
+ "tool_call_id": call.id,
+ "content": json.dumps(result),
+ }
+ )
+
+ return {
+ "summary": "Stopped after reaching the turn limit without a final answer — check the tool calls above.",
+ "tool_calls": tool_call_log,
+ }
diff --git a/ai_agents/voice-github-agent/app.py b/ai_agents/voice-github-agent/app.py
new file mode 100644
index 0000000..c6fa3c7
--- /dev/null
+++ b/ai_agents/voice-github-agent/app.py
@@ -0,0 +1,86 @@
+"""Voice GitHub Agent — web app entry point.
+
+Flow:
+ 1. Browser records a spoken instruction and encodes it as WAV client-side.
+ 2. POST /api/run receives that WAV, transcribes it with AssemblyAI's Sync
+ API, hands the transcript to the tool-calling agent, and returns the
+ transcript, the tool calls made, and the agent's final summary as JSON.
+
+Run: python app.py, then open http://localhost:5000
+"""
+import os
+import tempfile
+
+from dotenv import find_dotenv, load_dotenv
+
+# load_dotenv() does NOT override a variable that's already set in the real
+# OS environment — if GITHUB_REPO/GITHUB_TOKEN were ever `export`ed (or set
+# in a parent shell/terminal) in this session, that value silently wins over
+# .env every time, regardless of how many times .env is edited or the app is
+# restarted. If GITHUB_REPO ever looks stale, check `echo $GITHUB_REPO` /
+# `$env:GITHUB_REPO` in the shell you're launching `python app.py` from
+# before assuming .env itself is wrong.
+load_dotenv()
+
+print(f"[startup] .env resolved to: {find_dotenv() or '(none found)'}")
+print(f"[startup] GITHUB_REPO = {os.environ.get('GITHUB_REPO')!r}")
+
+from flask import Flask, jsonify, render_template, request
+
+from agent import run_agent
+from transcribe import transcribe_audio
+
+REQUIRED_ENV_VARS = ("ASSEMBLYAI_API_KEY", "GITHUB_TOKEN", "GITHUB_REPO")
+
+app = Flask(__name__)
+
+
+def index():
+ return render_template("index.html")
+
+
[email protected]("/api/run", methods=["POST"])
+def api_run():
+ missing = [v for v in REQUIRED_ENV_VARS if not os.environ.get(v)]
+ if missing:
+ return jsonify({"error": f"Missing env var(s): {', '.join(missing)}. Check your .env file."}), 500
+
+ audio_file = request.files.get("audio")
+ if audio_file is None:
+ return jsonify({"error": "No audio file uploaded."}), 400
+
+ with tempfile.TemporaryDirectory() as tmp:
+ wav_path = os.path.join(tmp, "instruction.wav")
+ audio_file.save(wav_path)
+
+ try:
+ transcription = transcribe_audio(wav_path)
+ except Exception as exc:
+ return jsonify({"error": f"Transcription failed: {exc}"}), 502
+
+ transcript = (transcription.get("text") or "").strip()
+ if not transcript:
+ return jsonify({"error": "Got an empty transcript — try again and speak a bit longer."}), 400
+
+ try:
+ agent_result = run_agent(transcript)
+ except Exception as exc:
+ return jsonify({"error": f"Agent failed: {exc}", "transcript": transcript}), 502
+
+ return jsonify(
+ {
+ "transcript": transcript,
+ "tool_calls": agent_result["tool_calls"],
+ "summary": agent_result["summary"],
+ }
+ )
+
+
+if __name__ == "__main__":
+ missing = [v for v in REQUIRED_ENV_VARS if not os.environ.get(v)]
+ if missing:
+ raise SystemExit(
+ f"Missing env var(s): {', '.join(missing)}. Copy .env.example to .env and fill it in."
+ )
+ app.run(debug=True, port=5000)
diff --git a/ai_agents/voice-github-agent/assets/demo.gif b/ai_agents/voice-github-agent/assets/demo.gif
new file mode 100644
index 0000000..d2f7366
Binary files /dev/null and b/ai_agents/voice-github-agent/assets/demo.gif differ
diff --git a/ai_agents/voice-github-agent/github_tools.py b/ai_agents/voice-github-agent/github_tools.py
new file mode 100644
index 0000000..06e581a
--- /dev/null
+++ b/ai_agents/voice-github-agent/github_tools.py
@@ -0,0 +1,201 @@
+"""GitHub tools the agent can call, plus the OpenAI-style schemas that
+describe them to the model.
+
+v1 scope: one fixed repo (GITHUB_REPO, "owner/name") and a personal access
+token (GITHUB_TOKEN). Enough to run "check recent changes, flag issues,
+summarize" end to end. Letting the spoken instruction name a different repo
+is a natural follow-up once this is working.
+"""
+import os
+
+import requests
+
+API_ROOT = "https://api.github.com"
+
+
+def _repo() -> str:
+ repo = os.environ.get("GITHUB_REPO")
+ if not repo:
+ raise RuntimeError("GITHUB_REPO is not set (expected 'owner/name'). Check your .env file.")
+ return repo
+
+
+def _headers() -> dict:
+ token = os.environ.get("GITHUB_TOKEN")
+ if not token:
+ raise RuntimeError("GITHUB_TOKEN is not set. Check your .env file.")
+ return {
+ "Authorization": f"Bearer {token}",
+ "Accept": "application/vnd.github+json",
+ "X-GitHub-Api-Version": "2022-11-28",
+ }
+
+
+def _request(method: str, path: str, **kwargs) -> dict:
+ response = requests.request(method, f"{API_ROOT}{path}", headers=_headers(), timeout=30, **kwargs)
+ if not response.ok:
+ raise RuntimeError(f"GitHub API {method} {path} failed ({response.status_code}): {response.text[:300]}")
+ return response.json() if response.text else {}
+
+
+def list_recent_commits(count: int = 5) -> dict:
+ """List the most recent commits on the repo's default branch."""
+ count = max(1, min(count, 20))
+ data = _request("GET", f"/repos/{_repo()}/commits", params={"per_page": count})
+ return {
+ "commits": [
+ {
+ "sha": c["sha"][:10],
+ "message": c["commit"]["message"].splitlines()[0],
+ "author": (c["commit"]["author"] or {}).get("name"),
+ "date": (c["commit"]["author"] or {}).get("date"),
+ "url": c["html_url"],
+ }
+ for c in data
+ ]
+ }
+
+
+def get_commit_diff(sha: str) -> dict:
+ """Get the file-level diff for one commit, to see what actually changed."""
+ data = _request("GET", f"/repos/{_repo()}/commits/{sha}")
+ files = data.get("files", [])
+ return {
+ "sha": sha,
+ "message": data.get("commit", {}).get("message", ""),
+ "files": [
+ {
+ "filename": f.get("filename"),
+ "status": f.get("status"),
+ "additions": f.get("additions"),
+ "deletions": f.get("deletions"),
+ # Truncate long patches so a big diff doesn't blow the context budget.
+ "patch": (f.get("patch") or "")[:4000],
+ }
+ for f in files
+ ],
+ }
+
+
+def list_open_issues(count: int = 10) -> dict:
+ """List open issues (pull requests are excluded)."""
+ count = max(1, min(count, 30))
+ data = _request("GET", f"/repos/{_repo()}/issues", params={"state": "open", "per_page": count})
+ return {
+ "issues": [
+ {"number": i["number"], "title": i["title"], "url": i["html_url"]}
+ for i in data
+ if "pull_request" not in i
+ ]
+ }
+
+
+def create_issue(title: str, body: str = "") -> dict:
+ """Open a new GitHub issue."""
+ data = _request("POST", f"/repos/{_repo()}/issues", json={"title": title, "body": body})
+ return {"number": data["number"], "url": data["html_url"]}
+
+
+def add_comment(issue_number: int, body: str) -> dict:
+ """Add a comment to an existing issue or pull request."""
+ data = _request("POST", f"/repos/{_repo()}/issues/{issue_number}/comments", json={"body": body})
+ return {"url": data["html_url"]}
+
+
+TOOL_FUNCTIONS = {
+ "list_recent_commits": list_recent_commits,
+ "get_commit_diff": get_commit_diff,
+ "list_open_issues": list_open_issues,
+ "create_issue": create_issue,
+ "add_comment": add_comment,
+}
+
+TOOL_SCHEMAS = [
+ {
+ "type": "function",
+ "function": {
+ "name": "list_recent_commits",
+ "description": "List the most recent commits on the repo's default branch.",
+ "parameters": {
+ "type": "object",
+ "properties": {
+ "count": {
+ "type": "integer",
+ "description": "How many commits to return (max 20).",
+ "default": 5,
+ }
+ },
+ },
+ },
+ },
+ {
+ "type": "function",
+ "function": {
+ "name": "get_commit_diff",
+ "description": "Get the file-level diff (patch) for one commit, to inspect what changed.",
+ "parameters": {
+ "type": "object",
+ "properties": {
+ "sha": {"type": "string", "description": "The commit SHA to inspect."}
+ },
+ "required": ["sha"],
+ },
+ },
+ },
+ {
+ "type": "function",
+ "function": {
+ "name": "list_open_issues",
+ "description": "List currently open issues in the repo (pull requests excluded).",
+ "parameters": {
+ "type": "object",
+ "properties": {
+ "count": {
+ "type": "integer",
+ "description": "How many issues to return (max 30).",
+ "default": 10,
+ }
+ },
+ },
+ },
+ },
+ {
+ "type": "function",
+ "function": {
+ "name": "create_issue",
+ "description": (
+ "Open a new GitHub issue. Use this for anything the spoken request asks "
+ "to flag or track, after you've actually inspected the relevant code."
+ ),
+ "parameters": {
+ "type": "object",
+ "properties": {
+ "title": {"type": "string", "description": "Short issue title."},
+ "body": {
+ "type": "string",
+ "description": "Issue description — what was found and why it matters.",
+ },
+ },
+ "required": ["title"],
+ },
+ },
+ },
+ {
+ "type": "function",
+ "function": {
+ "name": "add_comment",
+ "description": "Add a comment to an existing issue or pull request.",
+ "parameters": {
+ "type": "object",
+ "properties": {
+ "issue_number": {
+ "type": "integer",
+ "description": "The issue or PR number to comment on.",
+ },
+ "body": {"type": "string", "description": "Comment text."},
+ },
+ "required": ["issue_number", "body"],
+ },
+ },
+ },
+]
diff --git a/ai_agents/voice-github-agent/prompts/system_prompt.txt b/ai_agents/voice-github-agent/prompts/system_prompt.txt
new file mode 100644
index 0000000..a1c7a48
--- /dev/null
+++ b/ai_agents/voice-github-agent/prompts/system_prompt.txt
@@ -0,0 +1,20 @@
+You are a voice-controlled GitHub agent. You receive a transcript of a spoken
+instruction from a developer and act on it using the tools available to you.
+
+Guidelines:
+- The transcript comes from speech-to-text, so it may contain filler words,
+ false starts, or minor recognition errors. Interpret the developer's intent
+ rather than the literal wording.
+- Decide which tools to use based on what was asked. Only call the tools you
+ actually need.
+- Look before you act: read commits, diffs, or issues before creating an
+ issue or comment, so what you file is grounded in something you actually
+ inspected, not assumed.
+- If asked to flag anything "important" or "obvious," use your judgment —
+ bugs, security concerns, and clearly broken code count; style nitpicks
+ generally don't unless specifically asked for.
+- Do not invent findings. If you didn't find anything worth flagging, say so
+ instead of creating an issue anyway.
+- When you're done, respond with a concise final summary and no more tool
+ calls. Cover: what you checked, what you found, and what actions you took
+ (include links where relevant).
diff --git a/ai_agents/voice-github-agent/requirements.txt b/ai_agents/voice-github-agent/requirements.txt
new file mode 100644
index 0000000..2808e37
--- /dev/null
+++ b/ai_agents/voice-github-agent/requirements.txt
@@ -0,0 +1,4 @@
+flask>=3.0
+requests>=2.31
+openai>=1.40
+python-dotenv>=1.0
diff --git a/ai_agents/voice-github-agent/static/app.js b/ai_agents/voice-github-agent/static/app.js
new file mode 100644
index 0000000..6632a3b
--- /dev/null
+++ b/ai_agents/voice-github-agent/static/app.js
@@ -0,0 +1,245 @@
+const recordBtn = document.getElementById("record-btn");
+const statusLine = document.getElementById("status-line");
+const errorBanner = document.getElementById("error-banner");
+const resultSection = document.getElementById("result");
+const transcriptText = document.getElementById("transcript-text");
+const activityFeed = document.getElementById("activity-feed");
+const summaryText = document.getElementById("summary-text");
+
+let mediaRecorder = null;
+let mediaStream = null;
+let chunks = [];
+let isRecording = false;
+
+recordBtn.addEventListener("click", () => {
+ if (isRecording) {
+ stopRecording();
+ } else {
+ startRecording();
+ }
+});
+
+async function startRecording() {
+ hideError();
+
+ try {
+ mediaStream = await navigator.mediaDevices.getUserMedia({ audio: true });
+ } catch (err) {
+ showError("Microphone access was denied or is unavailable.");
+ return;
+ }
+
+ chunks = [];
+ mediaRecorder = new MediaRecorder(mediaStream);
+ mediaRecorder.ondataavailable = (event) => {
+ if (event.data && event.data.size > 0) chunks.push(event.data);
+ };
+ mediaRecorder.onstop = handleStop;
+ mediaRecorder.start();
+
+ isRecording = true;
+ recordBtn.classList.add("is-recording");
+ recordBtn.setAttribute("aria-pressed", "true");
+ recordBtn.setAttribute("aria-label", "Stop recording");
+ statusLine.textContent = "Recording — click to stop";
+}
+
+function stopRecording() {
+ if (mediaRecorder && mediaRecorder.state !== "inactive") {
+ mediaRecorder.stop();
+ }
+ if (mediaStream) {
+ mediaStream.getTracks().forEach((track) => track.stop());
+ }
+ isRecording = false;
+ recordBtn.classList.remove("is-recording");
+ recordBtn.setAttribute("aria-pressed", "false");
+ recordBtn.setAttribute("aria-label", "Start recording");
+}
+
+async function handleStop() {
+ statusLine.textContent = "Encoding audio...";
+
+ const recordedBlob = new Blob(chunks, { type: mediaRecorder.mimeType || "audio/webm" });
+ const arrayBuffer = await recordedBlob.arrayBuffer();
+
+ const AudioContextCtor = window.AudioContext || window.webkitAudioContext;
+ const audioCtx = new AudioContextCtor();
+
+ let audioBuffer;
+ try {
+ audioBuffer = await audioCtx.decodeAudioData(arrayBuffer);
+ } catch (err) {
+ showError("Could not process that recording — try again.");
+ statusLine.textContent = "Click to speak";
+ return;
+ } finally {
+ audioCtx.close();
+ }
+
+ const wavBlob = encodeWav(audioBuffer);
+ await sendToServer(wavBlob);
+}
+
+/** Encodes an AudioBuffer as a 16-bit PCM mono WAV Blob — AssemblyAI's Sync API needs WAV. */
+function encodeWav(audioBuffer) {
+ const samples = downmixToMono(audioBuffer);
+ const sampleRate = audioBuffer.sampleRate;
+ const bytesPerSample = 2;
+ const dataSize = samples.length * bytesPerSample;
+
+ const buffer = new ArrayBuffer(44 + dataSize);
+ const view = new DataView(buffer);
+
+ writeString(view, 0, "RIFF");
+ view.setUint32(4, 36 + dataSize, true);
+ writeString(view, 8, "WAVE");
+ writeString(view, 12, "fmt ");
+ view.setUint32(16, 16, true); // fmt chunk size
+ view.setUint16(20, 1, true); // PCM
+ view.setUint16(22, 1, true); // mono
+ view.setUint32(24, sampleRate, true);
+ view.setUint32(28, sampleRate * bytesPerSample, true); // byte rate
+ view.setUint16(32, bytesPerSample, true); // block align
+ view.setUint16(34, 16, true); // bits per sample
+ writeString(view, 36, "data");
+ view.setUint32(40, dataSize, true);
+
+ let offset = 44;
+ for (let i = 0; i < samples.length; i++) {
+ const clamped = Math.max(-1, Math.min(1, samples[i]));
+ view.setInt16(offset, clamped < 0 ? clamped * 0x8000 : clamped * 0x7fff, true);
+ offset += 2;
+ }
+
+ return new Blob([buffer], { type: "audio/wav" });
+}
+
+function downmixToMono(audioBuffer) {
+ if (audioBuffer.numberOfChannels === 1) {
+ return audioBuffer.getChannelData(0);
+ }
+ const left = audioBuffer.getChannelData(0);
+ const right = audioBuffer.getChannelData(1);
+ const out = new Float32Array(left.length);
+ for (let i = 0; i < left.length; i++) {
+ out[i] = (left[i] + right[i]) / 2;
+ }
+ return out;
+}
+
+function writeString(view, offset, str) {
+ for (let i = 0; i < str.length; i++) {
+ view.setUint8(offset + i, str.charCodeAt(i));
+ }
+}
+
+async function sendToServer(wavBlob) {
+ statusLine.textContent = "Thinking...";
+ recordBtn.disabled = true;
+ resultSection.hidden = true;
+
+ const formData = new FormData();
+ formData.append("audio", wavBlob, "instruction.wav");
+
+ try {
+ const response = await fetch("/api/run", { method: "POST", body: formData });
+ const data = await response.json();
+
+ if (!response.ok) {
+ showError(data.error || "Something went wrong.");
+ return;
+ }
+
+ renderResult(data);
+ } catch (err) {
+ showError("Could not reach the server.");
+ } finally {
+ statusLine.textContent = "Click to speak";
+ recordBtn.disabled = false;
+ }
+}
+
+function renderResult(data) {
+ resultSection.hidden = false;
+
+ // Transcript, with a briefly blinking caret to sell the "lands at the cursor" feel.
+ transcriptText.textContent = data.transcript || "";
+ const caret = document.createElement("span");
+ caret.className = "caret";
+ transcriptText.appendChild(caret);
+ setTimeout(() => caret.remove(), 2200);
+
+ activityFeed.innerHTML = "";
+ const toolCalls = data.tool_calls || [];
+
+ if (toolCalls.length === 0) {
+ const li = document.createElement("li");
+ li.className = "activity-item activity-item-empty";
+ li.textContent = "No tools were needed for this one.";
+ activityFeed.appendChild(li);
+ } else {
+ toolCalls.forEach((call, index) => {
+ const li = document.createElement("li");
+ li.className = "activity-item";
+ li.style.animationDelay = `${index * 120}ms`;
+ li.innerHTML = renderMarkdownInline(describeToolCall(call));
+ activityFeed.appendChild(li);
+ });
+ }
+
+ summaryText.innerHTML = renderMarkdownBlock(data.summary || "");
+}
+
+/**
+ * The summary and tool-call descriptions come from the model, so before
+ * handing them to marked we escape the characters that would otherwise let
+ * raw HTML (a <script> tag, an onerror= attribute, ...) slip into innerHTML.
+ * Markdown syntax itself (*, `, #, -, etc.) doesn't use < or >, so this
+ * doesn't affect legitimate formatting — it only neutralizes literal HTML.
+ */
+function escapeHtml(text) {
+ return text.replace(/&/g, "&amp;").replace(/</g, "&lt;").replace(/>/g, "&gt;");
+}
+
+/** Full block-level markdown (paragraphs, numbered/bulleted lists, etc.) for the summary panel. */
+function renderMarkdownBlock(text) {
+ return marked.parse(escapeHtml(text));
+}
+
+/** Inline-only markdown (bold, code, links) for single-line activity feed entries. */
+function renderMarkdownInline(text) {
+ return marked.parseInline(escapeHtml(text));
+}
+
+/** Turns a raw tool call into a plain-language activity line for the feed. */
+function describeToolCall(call) {
+ const name = call.name;
+ const args = call.args || {};
+ const result = call.result || {};
+
+ switch (name) {
+ case "list_recent_commits":
+ return `Checked the last ${args.count || 5} commit${(args.count || 5) === 1 ? "" : "s"}`;
+ case "get_commit_diff":
+ return `Looked at the diff for commit ${(args.sha || "").slice(0, 7)}`;
+ case "list_open_issues":
+ return "Checked the open issues";
+ case "create_issue":
+ return result.number ? `Opened issue #${result.number}` : `Opened a new issue: "${args.title || ""}"`;
+ case "add_comment":
+ return `Added a comment on issue #${args.issue_number}`;
+ default:
+ return `Ran ${name}`;
+ }
+}
+
+function showError(message) {
+ errorBanner.textContent = message;
+ errorBanner.hidden = false;
+}
+
+function hideError() {
+ errorBanner.hidden = true;
+ errorBanner.textContent = "";
+}
diff --git a/ai_agents/voice-github-agent/static/style.css b/ai_agents/voice-github-agent/static/style.css
new file mode 100644
index 0000000..e4fd9a8
--- /dev/null
+++ b/ai_agents/voice-github-agent/static/style.css
@@ -0,0 +1,334 @@
+:root {
+ --bg: #f6f3ec;
+ --bg-elevated: #fffdf8;
+ --text: #23201a;
+ --text-muted: #7a7263;
+ --border: #e2dcca;
+ --accent: #b5502f;
+ --accent-ink: #ffffff;
+ --caret: #b5502f;
+ color-scheme: light;
+}
+
+@media (prefers-color-scheme: dark) {
+ :root {
+ --bg: #17150f;
+ --bg-elevated: #201d16;
+ --text: #ece6d8;
+ --text-muted: #948c78;
+ --border: #332e22;
+ --accent: #e0774a;
+ --accent-ink: #17150f;
+ --caret: #e0774a;
+ color-scheme: dark;
+ }
+}
+
+* {
+ box-sizing: border-box;
+}
+
+html, body {
+ margin: 0;
+ padding: 0;
+}
+
+body {
+ background: var(--bg);
+ color: var(--text);
+ font-family: -apple-system, "Segoe UI", system-ui, sans-serif;
+ min-height: 100vh;
+}
+
+.app {
+ max-width: 640px;
+ margin: 0 auto;
+ padding: 56px 24px 96px;
+}
+
+.app-header {
+ text-align: center;
+ margin-bottom: 56px;
+}
+
+.wordmark {
+ font-size: 15px;
+ font-weight: 600;
+ letter-spacing: 0.16em;
+ text-transform: uppercase;
+ color: var(--text);
+ display: inline-flex;
+ align-items: center;
+ gap: 8px;
+}
+
+.wordmark-dot {
+ width: 8px;
+ height: 8px;
+ border-radius: 50%;
+ background: var(--accent);
+ display: inline-block;
+}
+
+.tagline {
+ margin: 10px 0 0;
+ font-family: Georgia, "Iowan Old Style", ui-serif, serif;
+ font-style: italic;
+ color: var(--text-muted);
+ font-size: 15px;
+}
+
+.stage {
+ display: flex;
+ flex-direction: column;
+ align-items: center;
+ gap: 18px;
+ margin-bottom: 48px;
+}
+
+.record-btn {
+ position: relative;
+ width: 96px;
+ height: 96px;
+ border-radius: 50%;
+ border: none;
+ background: transparent;
+ cursor: pointer;
+ padding: 0;
+ display: grid;
+ place-items: center;
+}
+
+.record-btn-ring {
+ position: absolute;
+ inset: 0;
+ border-radius: 50%;
+ border: 1.5px solid var(--border);
+ transition: border-color 160ms ease;
+}
+
+.record-btn-core {
+ width: 56px;
+ height: 56px;
+ border-radius: 50%;
+ background: var(--accent);
+ transition: transform 160ms ease, border-radius 160ms ease;
+}
+
+.record-btn:hover .record-btn-ring {
+ border-color: var(--accent);
+}
+
+.record-btn:active .record-btn-core {
+ transform: scale(0.92);
+}
+
+.record-btn.is-recording .record-btn-core {
+ border-radius: 10px;
+ width: 32px;
+ height: 32px;
+}
+
+.record-btn.is-recording .record-btn-ring {
+ border-color: var(--accent);
+ animation: pulse-ring 1.6s ease-out infinite;
+}
+
+.record-btn:disabled {
+ cursor: default;
+ opacity: 0.6;
+}
+
+@keyframes pulse-ring {
+ 0% { box-shadow: 0 0 0 0 rgba(181, 80, 47, 0.35); }
+ 70% { box-shadow: 0 0 0 14px rgba(181, 80, 47, 0); }
+ 100% { box-shadow: 0 0 0 0 rgba(181, 80, 47, 0); }
+}
+
+.status-line {
+ margin: 0;
+ font-size: 13px;
+ letter-spacing: 0.04em;
+ color: var(--text-muted);
+ text-transform: uppercase;
+}
+
+.error-banner {
+ max-width: 480px;
+ margin: -8px auto 32px;
+ padding: 12px 16px;
+ border: 1px solid var(--accent);
+ border-radius: 8px;
+ color: var(--accent);
+ font-size: 14px;
+ text-align: center;
+}
+
+.result {
+ display: flex;
+ flex-direction: column;
+ gap: 28px;
+}
+
+.panel {
+ background: var(--bg-elevated);
+ border: 1px solid var(--border);
+ border-radius: 10px;
+ padding: 24px 26px;
+}
+
+.panel-label {
+ margin: 0 0 14px;
+ font-size: 12px;
+ font-weight: 600;
+ letter-spacing: 0.14em;
+ text-transform: uppercase;
+ color: var(--text-muted);
+}
+
+.panel-transcript {
+ border-left: 3px solid var(--accent);
+}
+
+.transcript-text {
+ margin: 0;
+ font-family: Georgia, "Iowan Old Style", ui-serif, serif;
+ font-size: 19px;
+ line-height: 1.6;
+ color: var(--text);
+}
+
+.transcript-text .caret {
+ display: inline-block;
+ width: 2px;
+ height: 1.1em;
+ margin-left: 2px;
+ vertical-align: -2px;
+ background: var(--caret);
+ animation: blink 1s step-end infinite;
+}
+
+@keyframes blink {
+ 50% { opacity: 0; }
+}
+
+.activity-feed {
+ list-style: none;
+ margin: 0;
+ padding: 0;
+ display: flex;
+ flex-direction: column;
+ gap: 10px;
+}
+
+.activity-item {
+ display: flex;
+ align-items: baseline;
+ gap: 10px;
+ font-size: 14.5px;
+ color: var(--text);
+ opacity: 0;
+ transform: translateY(4px);
+ animation: rise-in 320ms ease forwards;
+}
+
+.activity-item::before {
+ content: "\2713";
+ color: var(--accent);
+ font-size: 12px;
+ flex: none;
+}
+
+.activity-item-empty {
+ color: var(--text-muted);
+}
+
+.activity-item-empty::before {
+ content: "\2013";
+}
+
+.activity-item code {
+ font-family: ui-monospace, "SFMono-Regular", Consolas, Menlo, monospace;
+ font-size: 0.9em;
+ padding: 0.1em 0.35em;
+ border-radius: 4px;
+ background: var(--bg);
+ border: 1px solid var(--border);
+}
+
+.activity-item strong {
+ font-weight: 700;
+}
+
+@keyframes rise-in {
+ to { opacity: 1; transform: translateY(0); }
+}
+
+.panel-summary {
+ background: var(--accent);
+ color: var(--accent-ink);
+ border-color: var(--accent);
+}
+
+.panel-summary .panel-label {
+ color: var(--accent-ink);
+ opacity: 0.75;
+}
+
+.summary-text {
+ margin: 0;
+ font-size: 16px;
+ line-height: 1.6;
+}
+
+.summary-text p {
+ margin: 0 0 12px;
+}
+
+.summary-text p:last-child {
+ margin-bottom: 0;
+}
+
+.summary-text strong {
+ font-weight: 700;
+}
+
+.summary-text code {
+ font-family: ui-monospace, "SFMono-Regular", Consolas, Menlo, monospace;
+ font-size: 0.9em;
+ padding: 0.15em 0.4em;
+ border-radius: 4px;
+ background: rgba(255, 255, 255, 0.18);
+}
+
+.summary-text ol,
+.summary-text ul {
+ margin: 0 0 12px;
+ padding-left: 1.3em;
+}
+
+.summary-text ol:last-child,
+.summary-text ul:last-child {
+ margin-bottom: 0;
+}
+
+.summary-text li {
+ margin: 4px 0;
+ line-height: 1.6;
+}
+
+.summary-text li::marker {
+ color: var(--accent-ink);
+ opacity: 0.7;
+}
+
+.summary-text a {
+ color: inherit;
+ text-decoration: underline;
+}
+
+@media (max-width: 480px) {
+ .app {
+ padding: 40px 18px 72px;
+ }
+}
diff --git a/ai_agents/voice-github-agent/templates/index.html b/ai_agents/voice-github-agent/templates/index.html
new file mode 100644
index 0000000..78f4497
--- /dev/null
+++ b/ai_agents/voice-github-agent/templates/index.html
@@ -0,0 +1,47 @@
+<!doctype html>
+<html lang="en">
+<head>
+ <meta charset="utf-8">
+ <meta name="viewport" content="width=device-width, initial-scale=1">
+ <title>Dictate — voice GitHub agent</title>
+ <link rel="stylesheet" href="{{ url_for('static', filename='style.css') }}">
+</head>
+<body>
+ <div class="app">
+ <header class="app-header">
+ <div class="wordmark"><span class="wordmark-dot"></span>Dictate</div>
+ <p class="tagline">Speak an instruction. It lands on the page, then on GitHub.</p>
+ </header>
+
+ <main class="stage">
+ <button id="record-btn" class="record-btn" type="button" aria-pressed="false" aria-label="Start recording">
+ <span class="record-btn-ring"></span>
+ <span class="record-btn-core"></span>
+ </button>
+ <p id="status-line" class="status-line">Click to speak</p>
+ </main>
+
+ <p id="error-banner" class="error-banner" hidden></p>
+
+ <section id="result" class="result" hidden>
+ <article class="panel panel-transcript">
+ <h2 class="panel-label">Transcript</h2>
+ <p id="transcript-text" class="transcript-text"></p>
+ </article>
+
+ <article class="panel panel-activity">
+ <h2 class="panel-label">Activity</h2>
+ <ul id="activity-feed" class="activity-feed"></ul>
+ </article>
+
+ <article class="panel panel-summary">
+ <h2 class="panel-label">Summary</h2>
+ <div id="summary-text" class="summary-text"></div>
+ </article>
+ </section>
+ </div>
+
+ <script src="https://cdnjs.cloudflare.com/ajax/libs/marked/12.0.0/marked.min.js"></script>
+ <script src="{{ url_for('static', filename='app.js') }}"></script>
+</body>
+</html>
diff --git a/ai_agents/voice-github-agent/transcribe.py b/ai_agents/voice-github-agent/transcribe.py
new file mode 100644
index 0000000..2ae6047
--- /dev/null
+++ b/ai_agents/voice-github-agent/transcribe.py
@@ -0,0 +1,45 @@
+"""Thin wrapper around AssemblyAI's Sync API.
+
+One HTTP call in, finished transcript back in the same response — no
+polling. Runs on the flagship Universal-3.5 Pro model.
+
+Docs: https://www.assemblyai.com/docs/sync-stt/getting-started/transcribe-a-short-audio-file
+"""
+import os
+
+import requests
+
+SYNC_URL = "https://sync.assemblyai.com/transcribe"
+MODEL = "universal-3-5-pro"
+
+
+def transcribe_audio(file_path: str) -> dict:
+ """Sends a WAV file to the Sync API and returns the parsed JSON result
+ (includes 'text', 'words', and 'session_id')."""
+ api_key = os.environ.get("ASSEMBLYAI_API_KEY")
+ if not api_key:
+ raise RuntimeError("ASSEMBLYAI_API_KEY is not set. Check your .env file.")
+
+ with open(file_path, "rb") as f:
+ audio_bytes = f.read()
+
+ response = requests.post(
+ SYNC_URL,
+ headers={
+ "Authorization": api_key,
+ "X-AAI-Model": MODEL,
+ },
+ files={"audio": (os.path.basename(file_path), audio_bytes, "audio/wav")},
+ timeout=60,
+ )
+
+ if not response.ok:
+ raise RuntimeError(
+ f"Sync API request failed ({response.status_code}): {response.text}"
+ )
+
+ result = response.json()
+ if "session_id" in result:
+ # Worth keeping in logs — it's the first thing AssemblyAI support asks for.
+ print(f"[AssemblyAI session_id: {result['session_id']}]")
+ return result