Voice-to-text dictation app with local (Nvidia Parakeet/Whisper) and cloud models (BYOK). Privacy-first and available cross-platform.
JavaScript aianthropiccross-platformgeminigroqlinuxmacosnvidiaopen-sourceopenaiparakeetspeech-to-texttranscribewhisperwindows
refactor: Add cleanup docs/script, tighten onboarding hotkey/API key UX, fix timestamps, and drop legacy text cleanup
Gabriel Stein363d418e8ade
diff --git a/TROUBLESHOOTING.md b/TROUBLESHOOTING.mdnew file mode 100644index 000000000..ca42d8d89--- /dev/null+++ b/TROUBLESHOOTING.md@@ -0,0 +1,63 @@+# OpenWhispr Troubleshooting++## Quick Diagnostics++| Check | Command |+| --- | --- |+| Host architecture | `uname -m` |+| Node architecture | `node -p "process.arch"` |+| Native module arch | `file node_modules/better-sqlite3/build/Release/better_sqlite3.node` |+| FFmpeg availability | `ffmpeg -version` or `node -e "console.log(require('ffmpeg-static'))"` |+| Python install | `python3 --version && which python3` |+| Whisper cache size | `du -sh ~/Library/Application\ Support/whisper ~/.cache/whisper 2>/dev/null` |++If any of these look wrong (e.g., `x86_64` binaries on an Apple Silicon host), run `scripts/complete-uninstall.sh`, then reinstall dependencies.++## Common Issues++### 1. Architecture mismatch (arm64 vs x86_64)+**Symptoms:** the renderer crashes on launch, or `better_sqlite3.node` throws “wrong architecture” errors.++**Resolution:**+1. Run the diagnostics above. If `uname -m` reports `arm64` but `node -p "process.arch"` or `file …better_sqlite3.node` shows `x86_64`, Node was installed with Rosetta.+2. Uninstall the mismatched Node build (`brew uninstall node` or remove `/usr/local/bin/node`).+3. Reinstall an ARM build: `arch -arm64 brew install node@20` or download the arm64 pkg from nodejs.org.+4. Remove residual artifacts: run `scripts/complete-uninstall.sh` and allow it to delete caches.+5. Recreate dependencies: `rm -rf node_modules package-lock.json && npm ci`.+6. Rebuild (`npm run dev` or `npm run build`) and confirm onboarding starts normally.++### 2. “you” transcription bug+**Symptoms:** every transcription history entry shows the word “you”, even when you speak full sentences.++**Cause:** Whisper produced an empty/near-empty result and the legacy fallback string (`"you"`) is being saved. This usually happens when:+- the microphone permission was revoked mid-session,+- stale local Whisper caches contain zero-byte clips,+- or a hotkey collision triggers dictation without audio input.++**Resolution:**+1. Confirm microphone + accessibility permissions via macOS Settings, then re-run onboarding step 3 to re-test.+2. Clear SQLite history and caches: `scripts/complete-uninstall.sh` (answer “y” to Whisper model removal) or manually delete the folders listed in `CLEANUP_INSTRUCTIONS.md`.+3. Reinstall Whisper models from the Control Panel or rerun onboarding’s model picker.+4. Try a different dictation hotkey—reserved accelerators are now blocked in step 4 and settings.+5. If you rely on cloud transcription, verify your OpenAI key is valid (link in the UI now opens `https://platform.openai.com`).++### 3. FFmpeg missing+**Symptoms:** Local transcription fails immediately with “FFmpeg not found” or `ffmpeg-static` resolves to `null`.++**Resolution:**+1. Run `ffmpeg -version`. If the command is missing, install via `brew install ffmpeg` (Apple Silicon) or add it to your PATH.+2. Ensure npm dependencies pulled the static binary: `rm -rf node_modules package-lock.json && npm ci`.+3. Execute `npm run setup` so the bridge verifies FFmpeg before launching Electron.+4. If the packaged app still cannot find FFmpeg, run `scripts/complete-uninstall.sh` to clear frozen binaries, then rebuild via `npm run build:mac`.++### 4. Python install issues+**Symptoms:** Local Whisper install fails, or `window.electronAPI.installPython` never completes.++**Resolution:**+1. Verify you have a working Python 3: `python3 --version` and `which python3`. Apple Silicon should point to `/usr/bin/python3` or a Homebrew arm64 path.+2. Install Xcode Command Line Tools (`xcode-select --install`) so compilation helpers are available.+3. From the running app, open Settings → Whisper Models and click “Install Python” (uses the IPC bridge) or manually run the helper script `python3 setup.py` if supplied.+4. Delete `~/Library/Application Support/whisper` if partially downloaded models exist, then try again.+5. As a last resort, run `scripts/complete-uninstall.sh`, reboot, and rerun onboarding so the installer can download a clean toolchain.++Need deeper help? Capture the diagnostic command output and attach it to an issue so we can see which stage failed.diff --git a/package-lock.json b/package-lock.jsonindex 22d03c4f2..ceea79b14 100644--- a/package-lock.json+++ b/package-lock.json@@ -1,12 +1,12 @@{"name": "open-whispr",- "version": "1.0.11",+ "version": "1.0.12","lockfileVersion": 3,"requires": true,"packages": {"": {"name": "open-whispr",- "version": "1.0.11",+ "version": "1.0.12","hasInstallScript": true,"license": "MIT","dependencies": {diff --git a/package.json b/package.jsonindex 6d41feb2e..bc5bb5161 100644--- a/package.json+++ b/package.json@@ -1,6 +1,6 @@{"name": "open-whispr",- "version": "1.0.11",+ "version": "1.0.12","description": "A desktop dictation application using OpenAI Whisper for speech-to-text transcription","main": "main.js","private": true,diff --git a/scripts/complete-uninstall.sh b/scripts/complete-uninstall.shnew file mode 100755index 000000000..447068f7b--- /dev/null+++ b/scripts/complete-uninstall.sh@@ -0,0 +1,79 @@+#!/usr/bin/env bash++set -euo pipefail++PROJECT_ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"++echo "⚠️ This script will stop OpenWhispr, remove the installed app, and delete caches, databases, and preferences."+read -r -p "Continue with the full uninstall? [y/N]: " confirm+if [[ ! "$confirm" =~ ^[Yy]$ ]]; then+ echo "Aborted."+ exit 0+fi++remove_target() {+ local target="$1"+ if [[ -e "$target" ]]; then+ echo "Removing $target"+ rm -rf "$target" 2>/dev/null || sudo rm -rf "$target"+ fi+}++echo "Stopping running OpenWhispr/Electron processes..."+pkill -f "OpenWhispr" 2>/dev/null || true+pkill -f "open-whispr" 2>/dev/null || true+pkill -f "Electron Helper.*OpenWhispr" 2>/dev/null || true++echo "Removing /Applications/OpenWhispr.app (requires admin)..."+remove_target "/Applications/OpenWhispr.app"++echo "Purging Application Support data..."+remove_target "$HOME/Library/Application Support/OpenWhispr"+remove_target "$HOME/Library/Application Support/open-whispr"+remove_target "$HOME/Library/Application Support/OpenWhispr-dev"+remove_target "$HOME/Library/Application Support/com.openwhispr"+remove_target "$HOME/Library/Application Support/com.openwhispr.OpenWhispr"++echo "Removing caches, logs, and saved state..."+remove_target "$HOME/Library/Caches/open-whispr"+remove_target "$HOME/Library/Caches/com.openwhispr.OpenWhispr"+remove_target "$HOME/Library/Preferences/com.openwhispr.OpenWhispr.plist"+remove_target "$HOME/Library/Preferences/com.openwhispr.helper.plist"+remove_target "$HOME/Library/Logs/OpenWhispr"+remove_target "$HOME/Library/Saved Application State/com.openwhispr.OpenWhispr.savedState"++echo "Cleaning temporary files..."+shopt -s nullglob+for tmp in /tmp/openwhispr*; do+ remove_target "$tmp"+done+for crash in "$HOME/Library/Application Support/CrashReporter"/OpenWhispr_*; do+ remove_target "$crash"+done+shopt -u nullglob++read -r -p "Remove downloaded Whisper models and caches (~/.cache/whisper, ~/Library/Application Support/whisper)? [y/N]: " wipe_models+if [[ "$wipe_models" =~ ^[Yy]$ ]]; then+ remove_target "$HOME/.cache/whisper"+ remove_target "$HOME/Library/Application Support/whisper"+ remove_target "$HOME/Library/Application Support/OpenWhispr/models"+fi++ENV_FILE="$PROJECT_ROOT/.env"+if [[ -f "$ENV_FILE" ]]; then+ read -r -p "Remove the local environment file at $ENV_FILE? [y/N]: " wipe_env+ if [[ "$wipe_env" =~ ^[Yy]$ ]]; then+ echo "Removing $ENV_FILE"+ rm -f "$ENV_FILE"+ fi+fi++cat <<'EOF'+macOS keeps microphone, screen recording, and accessibility approvals even after files are removed.+Reset them if you want a truly fresh start:+ tccutil reset Microphone com.openwhispr.app+ tccutil reset Accessibility com.openwhispr.app+ tccutil reset ScreenCapture com.openwhispr.app++Full uninstall complete. Reboot if you removed permissions, then reinstall or run npm scripts on a clean tree.+EOFdiff --git a/src/components/AIModelSelectorEnhanced.tsx b/src/components/AIModelSelectorEnhanced.tsxindex f0e4ac7a6..abe8ec185 100644--- a/src/components/AIModelSelectorEnhanced.tsx+++ b/src/components/AIModelSelectorEnhanced.tsx@@ -742,7 +742,19 @@ export default function AIModelSelectorEnhanced({<ApiKeyInputapiKey={openaiApiKey}setApiKey={setOpenaiApiKey}- helpText="Get your API key from platform.openai.com"+ helpText={+ <>+ Need an API key?{" "}+ <a+ href="https://platform.openai.com"+ target="_blank"+ rel="noopener noreferrer"+ className="text-blue-600 underline"+ >+ platform.openai.com+ </a>+ </>+ }/></div>)}@@ -921,4 +933,4 @@ export default function AIModelSelectorEnhanced({)}</div>);-}\ No newline at end of file+}diff --git a/src/components/OnboardingFlow.tsx b/src/components/OnboardingFlow.tsxindex 882a9bccf..7ebaa8e2b 100644--- a/src/components/OnboardingFlow.tsx+++ b/src/components/OnboardingFlow.tsx@@ -296,6 +296,34 @@ export default function OnboardingFlow({ onComplete }: OnboardingFlowProps) {}}, [currentStep]);+ const ensureHotkeyRegistered = useCallback(async () => {+ if (!window.electronAPI?.updateHotkey) {+ return true;+ }++ try {+ const result = await window.electronAPI.updateHotkey(hotkey);+ if (result && !result.success) {+ showAlertDialog({+ title: "Hotkey Not Registered",+ description:+ result.message ||+ "We couldn't register that key. Please choose another hotkey.",+ });+ return false;+ }+ return true;+ } catch (error) {+ console.error("Failed to register onboarding hotkey", error);+ showAlertDialog({+ title: "Hotkey Error",+ description:+ "We couldn't register that key. Please choose another hotkey.",+ });+ return false;+ }+ }, [hotkey, showAlertDialog]);+const saveSettings = useCallback(async () => {const normalizedTranscriptionBase = (transcriptionBaseUrl || '').trim();const normalizedReasoningBaseValue = (reasoningBaseUrl || '').trim();@@ -313,24 +341,11 @@ export default function OnboardingFlow({ onComplete }: OnboardingFlowProps) {reasoningModel,cloudReasoningBaseUrl: normalizedReasoningBaseValue,});- setDictationKey(hotkey);- try {- const result = await window.electronAPI?.updateHotkey?.(hotkey);- if (result && !result.success) {- showAlertDialog({- title: "Hotkey Not Registered",- description:- result.message ||- "We couldn't register that key. Please choose another hotkey.",- });- }- } catch (error) {- console.error("Failed to register onboarding hotkey", error);- showAlertDialog({- title: "Hotkey Error",- description: "We couldn't register that key. Please choose another hotkey.",- });+ const hotkeyRegistered = await ensureHotkeyRegistered();+ if (!hotkeyRegistered) {+ return false;}+ setDictationKey(hotkey);saveAgentName(agentName);localStorage.setItem(@@ -342,11 +357,15 @@ export default function OnboardingFlow({ onComplete }: OnboardingFlowProps) {permissionsHook.accessibilityPermissionGranted.toString());localStorage.setItem("onboardingCompleted", "true");+ const trimmedApiKey = apiKey.trim();+ const skipAuth = trimmedApiKey.length === 0;+ localStorage.setItem("skipAuth", skipAuth.toString());- if (!useLocalWhisper && apiKey.trim()) {- await window.electronAPI.saveOpenAIKey(apiKey);- updateApiKeys({ openaiApiKey: apiKey });+ if (!useLocalWhisper && trimmedApiKey) {+ await window.electronAPI.saveOpenAIKey(trimmedApiKey);+ updateApiKeys({ openaiApiKey: trimmedApiKey });}+ return true;}, [whisperModel,hotkey,@@ -364,21 +383,33 @@ export default function OnboardingFlow({ onComplete }: OnboardingFlowProps) {setCloudTranscriptionBaseUrl,setCloudReasoningBaseUrl,setDictationKey,+ ensureHotkeyRegistered,]);- const nextStep = useCallback(() => {- if (currentStep < steps.length - 1) {- const newStep = currentStep + 1;- setCurrentStep(newStep);+ const nextStep = useCallback(async () => {+ if (currentStep >= steps.length - 1) {+ return;+ }- // Show dictation panel when moving from permissions step (3) to hotkey step (4)- if (currentStep === 3 && newStep === 4) {- if (window.electronAPI?.showDictationPanel) {- window.electronAPI.showDictationPanel();- }+ const newStep = currentStep + 1;++ if (currentStep === 4) {+ const registered = await ensureHotkeyRegistered();+ if (!registered) {+ return;+ }+ setDictationKey(hotkey);+ }++ setCurrentStep(newStep);++ // Show dictation panel when moving from permissions step (3) to hotkey step (4)+ if (currentStep === 3 && newStep === 4) {+ if (window.electronAPI?.showDictationPanel) {+ window.electronAPI.showDictationPanel();}}- }, [currentStep, setCurrentStep, steps.length]);+ }, [currentStep, ensureHotkeyRegistered, hotkey, setCurrentStep, setDictationKey, steps.length]);const prevStep = useCallback(() => {if (currentStep > 0) {@@ -388,7 +419,10 @@ export default function OnboardingFlow({ onComplete }: OnboardingFlowProps) {}, [currentStep, setCurrentStep]);const finishOnboarding = useCallback(async () => {- await saveSettings();+ const saved = await saveSettings();+ if (!saved) {+ return;+ }// Clear the onboarding step since we're doneremoveCurrentStep();onComplete();@@ -606,7 +640,19 @@ export default function OnboardingFlow({ onComplete }: OnboardingFlowProps) {apiKey={apiKey}setApiKey={setApiKey}label="OpenAI API Key"- helpText="Get your API key from platform.openai.com"+ helpText={+ <>+ Need an API key?{" "}+ <a+ href="https://platform.openai.com"+ target="_blank"+ rel="noopener noreferrer"+ className="text-blue-600 underline"+ >+ platform.openai.com+ </a>+ </>+ }/><div className="space-y-2">diff --git a/src/components/SettingsPage.tsx b/src/components/SettingsPage.tsxindex ea51affa7..8103ba4db 100644--- a/src/components/SettingsPage.tsx+++ b/src/components/SettingsPage.tsx@@ -916,7 +916,20 @@ export default function SettingsPage({<ApiKeyInputapiKey={openaiApiKey}setApiKey={setOpenaiApiKey}- helpText="Supports OpenAI or compatible endpoints"+ helpText={+ <>+ Supports OpenAI or compatible endpoints.{" "}+ <a+ href="https://platform.openai.com"+ target="_blank"+ rel="noopener noreferrer"+ className="text-blue-600 underline"+ >+ Get an API key+ </a>+ .+ </>+ }/><div className="space-y-2"><label className="block text-sm font-medium text-blue-900">diff --git a/src/components/ui/ApiKeyInput.tsx b/src/components/ui/ApiKeyInput.tsxindex 1ec670a34..c217966b7 100644--- a/src/components/ui/ApiKeyInput.tsx+++ b/src/components/ui/ApiKeyInput.tsx@@ -9,7 +9,7 @@ interface ApiKeyInputProps {className?: string;placeholder?: string;label?: string;- helpText?: string;+ helpText?: React.ReactNode;variant?: "default" | "purple";}@@ -53,7 +53,9 @@ export default function ApiKeyInput({Paste</Button></div>- <p className="text-xs text-neutral-600 mt-2">{helpText}</p>+ {helpText && (+ <p className="text-xs text-neutral-600 mt-2">{helpText}</p>+ )}</div>);}diff --git a/src/components/ui/TranscriptionItem.tsx b/src/components/ui/TranscriptionItem.tsxindex 4db48f1a4..cc6d39bea 100644--- a/src/components/ui/TranscriptionItem.tsx+++ b/src/components/ui/TranscriptionItem.tsx@@ -18,6 +18,19 @@ export default function TranscriptionItem({onCopy,onDelete,}: TranscriptionItemProps) {+ const timestampSource = item.timestamp.endsWith("Z")+ ? item.timestamp+ : `${item.timestamp}Z`;+ const timestampDate = new Date(timestampSource);+ const formattedTimestamp = Number.isNaN(timestampDate.getTime())+ ? item.timestamp+ : timestampDate.toLocaleString("en-US", {+ month: "short",+ day: "numeric",+ hour: "2-digit",+ minute: "2-digit",+ });+return (<div className="relative bg-gradient-to-b from-blue-50/30 to-white rounded-xl shadow-sm hover:shadow-md transition-shadow"><div className="p-6 pl-16" style={{ paddingTop: "8px" }}>@@ -32,12 +45,7 @@ export default function TranscriptionItem({</span><div className="w-px h-3 bg-neutral-300" /><span className="text-xs text-neutral-500">- {new Date(item.timestamp).toLocaleString("en-US", {- month: "short",- day: "numeric",- hour: "2-digit",- minute: "2-digit",- })}+ {formattedTimestamp}</span></div><pdiff --git a/src/helpers/audioManager.js b/src/helpers/audioManager.jsindex ff375b480..29e96abc1 100644--- a/src/helpers/audioManager.js+++ b/src/helpers/audioManager.js@@ -1,4 +1,3 @@-import TextCleanup from "../utils/textCleanup";import ReasoningService from "../services/ReasoningService";import { API_ENDPOINTS, buildApiUrl, normalizeBaseUrl } from "../config/constants";@@ -138,24 +137,6 @@ class AudioManager {}}- static cleanTranscription(text, options = {}) {- return TextCleanup.cleanTranscription(text, {- removeArtifacts: true,- normalizeSpaces: true,- fixPunctuation: true,- removeFillers: true,- removeRepetitions: true,- capitalizeFirst: true,- addPeriod: false,- ...options,- });- }-- static cleanTranscriptionForAPI(text) {- // Minimal cleanup - only normalize spaces for API processing- return TextCleanup.normalizeSpaces(text);- }-async processWithLocalWhisper(audioBlob, model = "base") {// Analyze audio levels first@@ -457,12 +438,13 @@ class AudioManager {}async processTranscription(text, source) {+ const normalizedText = typeof text === "string" ? text.trim() : "";// Log incoming transcriptiondebugLogger.logReasoning("TRANSCRIPTION_RECEIVED", {source,- textLength: text.length,- textPreview: text.substring(0, 100) + (text.length > 100 ? "..." : ""),+ textLength: normalizedText.length,+ textPreview: normalizedText.substring(0, 100) + (normalizedText.length > 100 ? "..." : ""),timestamp: new Date().toISOString()});@@ -490,7 +472,7 @@ class AudioManager {if (useReasoning) {try {// Minimal cleanup for reasoning models- const preparedText = AudioManager.cleanTranscriptionForAPI(text);+ const preparedText = normalizedText;debugLogger.logReasoning("SENDING_TO_REASONING", {preparedTextLength: preparedText.length,@@ -523,7 +505,7 @@ class AudioManager {});// Standard cleanup when reasoning is unavailable or fails- return AudioManager.cleanTranscription(text);+ return normalizedText;}async processWithOpenAIAPI(audioBlob) {diff --git a/src/types/electron.ts b/src/types/electron.tsindex 354ae981e..a9866f339 100644--- a/src/types/electron.ts+++ b/src/types/electron.ts@@ -250,7 +250,7 @@ declare global {removeAllListeners: (channel: string) => void;// Hotkey management- updateHotkey: (key: string) => Promise<void>;+ updateHotkey: (key: string) => Promise<{ success: boolean; message: string }>;// Gemini API key managementgetGeminiKey: () => Promise<string | null>;diff --git a/src/utils/textCleanup.js b/src/utils/textCleanup.jsdeleted file mode 100644index 1af4053ce..000000000--- a/src/utils/textCleanup.js+++ /dev/null@@ -1,462 +0,0 @@-/**- * Industry-standard text cleanup utility for transcription processing- * Optimized for performance with batch processing and smart filtering- */--class TextCleanup {- // Optimized patterns for transcription cleanup- static PATTERNS = {- // Non-verbal artifacts in brackets, parentheses, or asterisks- nonVerbal: [- /\[(?:laughter|music|noise|crosstalk|applause|coughing|inaudible|unclear|pause|silence|background|static|beep|mumbling|whispering|breathing|sighing|yawning|typing|footsteps|door|phone|rain|wind|traffic|crowd)\]/gi,- /\((?:laughter|music|noise|crosstalk|applause|coughing|inaudible|unclear|pause|silence|background|static|beep|mumbling|whispering|breathing|sighing|yawning|typing|footsteps|door|phone|rain|wind|traffic|crowd)\)/gi,- /\*(?:laughter|music|noise|crosstalk|applause|coughing|inaudible|unclear|pause|silence|background|static|beep|mumbling|whispering|breathing|sighing|yawning|typing|footsteps|door|phone|rain|wind|traffic|crowd)\*/gi,- ],-- // Speaker labels and metadata- speakers: [- /^Speaker\s*\d+:\s*/gmi,- /^\w+:\s*/gm,- /\[\d{1,2}:\d{2}(:\d{2})?\]/g, // Timestamps- /\<\d+\.\d+\>/g, // Confidence scores- /\[SPEAKER_\d+\]/gi,- /\[MUSIC\]/gi,- /\[SOUND\]/gi,- ],-- // Enhanced filler words - industry standard- fillers: [- /\b(?:um|uh|er|ah|mm|hmm|mhm)\b\s*/gi, // Hesitation sounds- /\b(?:like|so|well|right|okay|ok|yeah|yep|yup|sure|alright)\b\s*/gi, // Common discourse markers- /\b(?:you know|I mean|I guess|sort of|kind of)\b\s*/gi, // Meaningless phrases and uncertainty markers- /\b(?:maybe|perhaps)\b(?=\s)/gi, // Uncertainty markers (when not essential)- /\b(?:basically|literally|totally|honestly|obviously|definitely|probably|actually)\b\s*/gi, // Overused intensifiers- /\bOh,?\s+(?=\w)/gi, // "Oh" when followed by another word (often just filler)- ],-- // Repetition patterns- repetitions: [- /\b(\w+)\s+\1\b/gi, // Doubled words- /\b(\w+)\s+\1\s+\1\b/gi, // Tripled words- /\b(\w{2,})\s+\1\s+\1\s+\1\b/gi, // Quadrupled words- /\b(\w+[-']\w+)\s+\1\b/gi, // Repeated hyphenated/contracted- /\b([a-z]{1,3})\s+\1\s+\1\b/gi, // Stuttering (short words)- ],-- // Smart quotes and punctuation normalization- quotes: [- /[“”]/g, // Smart quotes to regular quotes- /[‘’]/g, // Smart apostrophes to regular- /…/g, // Ellipsis to three periods- ],-- // Excessive punctuation- punctuation: [- /\.{4,}/g, // 4+ periods to ellipsis- /\.{3}\s*\.+/g, // Ellipsis + more periods- /\?{2,}/g, // Multiple question marks- /!{2,}/g, // Multiple exclamation marks- /,{2,}/g, // Multiple commas- /;{2,}/g, // Multiple semicolons- /:{2,}/g, // Multiple colons- /-{3,}/g, // Long dashes to em-dash- ],- };-- // Common contractions and their expansions- static CONTRACTIONS = {- "won't": "will not",- "can't": "cannot",- "n't": " not",- "'re": " are",- "'ve": " have",- "'ll": " will",- "'d": " would",- "'m": " am",- "'s": " is", // Note: This is ambiguous ('s could be 'is' or possessive)- };-- // Sentence-ending punctuation- static SENTENCE_ENDERS = /[.!?]/;-- // Common sentence starters (capitalized words that typically start sentences)- static SENTENCE_STARTERS = /\b(?:The|This|That|These|Those|A|An|I|We|You|They|He|She|It|There|Here|What|When|Where|Why|How|Who|Which|My|Our|Your|Their|His|Her|Its|Some|Many|Most|All|Every|Each|One|Two|First|Second|Next|Then|Now|Today|Yesterday|Tomorrow|After|Before|During|While|Since|Because|Although|However|Therefore|Furthermore|Moreover|Nevertheless|Meanwhile|Finally|Initially|Subsequently|Consequently|Additionally|Specifically|Particularly|Essentially|Actually|Obviously|Clearly|Certainly|Definitely|Probably|Perhaps|Maybe|Sometimes|Often|Usually|Always|Never|Once|Again|Still|Yet|Already|Just|Only|Even|Also|Too|Very|Really|Quite|Rather|Extremely|Completely|Absolutely|Totally|Entirely|Exactly|Precisely|Approximately|About|Around|Nearly|Almost|Especially|Particularly|Generally|Typically|Normally|Usually|Frequently|Occasionally|Rarely|Seldom)\b/;-- /**- * Main cleanup function - processes text with all cleanup rules- * @param {string} text - Raw transcription text- * @param {Object} options - Cleanup options- * @returns {string} - Cleaned text- */- static cleanTranscription(text, options = {}) {- const {- removeArtifacts = true,- normalizeSpaces = true,- fixPunctuation = true,- expandContractions = false,- removeFillers = true,- removeRepetitions = true,- capitalizeFirst = true,- addPeriod = true,- customFilters = [],- preserveEmphasis = false,- } = options;-- if (!text || typeof text !== 'string') {- return '';- }-- // Start with trimming- let cleanedText = text.trim();-- // Early return for empty text- if (!cleanedText) {- return '';- }-- // Apply cleanup steps in order of importance- if (removeArtifacts) {- cleanedText = this.removeNonVerbalArtifacts(cleanedText);- }-- if (removeFillers) {- cleanedText = this.removeFillerWords(cleanedText);- }-- if (removeRepetitions) {- cleanedText = this.removeRepeatedWords(cleanedText);- }-- if (expandContractions) {- cleanedText = this.expandContractions(cleanedText);- }-- // Smart sentence reconstruction- cleanedText = this.reconstructSentences(cleanedText);-- if (fixPunctuation) {- cleanedText = this.fixPunctuation(cleanedText);- }-- if (normalizeSpaces) {- cleanedText = this.normalizeSpaces(cleanedText);- }-- // Apply custom filters- if (customFilters.length > 0) {- cleanedText = this.applyCustomFilters(cleanedText, customFilters);- }-- // Final formatting- if (capitalizeFirst) {- cleanedText = this.capitalizeFirstLetter(cleanedText);- }-- if (addPeriod) {- cleanedText = this.ensurePeriod(cleanedText);- }-- return cleanedText.trim();- }-- /**- * Remove non-verbal artifacts and speaker metadata- * @param {string} text- * @returns {string}- */- static removeNonVerbalArtifacts(text) {- let cleaned = text;-- // Remove non-verbal artifacts- this.PATTERNS.nonVerbal.forEach(pattern => {- cleaned = cleaned.replace(pattern, ' ');- });-- // Remove speaker artifacts- this.PATTERNS.speakers.forEach(pattern => {- cleaned = cleaned.replace(pattern, '');- });-- return cleaned;- }-- /**- * Remove common filler words - enhanced industry standard approach- * @param {string} text- * @returns {string}- */- static removeFillerWords(text) {- let cleaned = text;-- // Apply enhanced filler removal patterns- this.PATTERNS.fillers.forEach(pattern => {- cleaned = cleaned.replace(pattern, ' ');- });-- return cleaned;- }-- /**- * Remove repeated words - enhanced pattern detection- * @param {string} text- * @returns {string}- */- static removeRepeatedWords(text) {- let cleaned = text;-- // Apply all repetition patterns- this.PATTERNS.repetitions.forEach(pattern => {- cleaned = cleaned.replace(pattern, '$1');- });-- return cleaned;- }-- /**- * Expand common contractions- * @param {string} text- * @returns {string}- */- static expandContractions(text) {- let expanded = text;-- Object.entries(this.CONTRACTIONS).forEach(([contraction, expansion]) => {- const regex = new RegExp(contraction.replace(/'/g, "'?"), 'gi');- expanded = expanded.replace(regex, expansion);- });-- return expanded;- }-- /**- * Fix punctuation issues - enhanced industry standard approach- * @param {string} text- * @returns {string}- */- static fixPunctuation(text) {- let fixed = text;-- // Normalize smart quotes first- this.PATTERNS.quotes.forEach((pattern, index) => {- if (index === 0) {- fixed = fixed.replace(pattern, '"'); // Smart quotes to regular- } else if (index === 1) {- fixed = fixed.replace(pattern, "'"); // Smart apostrophes to regular- } else if (index === 2) {- fixed = fixed.replace(pattern, '...'); // Ellipsis to three periods- }- });-- // Fix excessive punctuation using patterns- this.PATTERNS.punctuation.forEach((pattern, index) => {- if (pattern.toString().includes('{4,}')) {- fixed = fixed.replace(pattern, '...'); // 4+ periods to ellipsis- } else if (pattern.toString().includes('\\?{2,}')) {- fixed = fixed.replace(pattern, '?'); // Multiple ? to single- } else if (pattern.toString().includes('!{2,}')) {- fixed = fixed.replace(pattern, '!'); // Multiple ! to single- } else if (pattern.toString().includes(',{2,}')) {- fixed = fixed.replace(pattern, ','); // Multiple , to single- } else if (pattern.toString().includes('-{3,}')) {- fixed = fixed.replace(pattern, '—'); // Long dashes to em-dash- } else {- fixed = fixed.replace(pattern, (match) => match.charAt(0)); // Other multiples to single- }- });-- // Fix spacing around punctuation (industry standard)- fixed = fixed.replace(/\s+([,.!?;:])/g, '$1'); // Remove space before punctuation- fixed = fixed.replace(/([,.!?;:])\s{2,}/g, '$1 '); // Single space after punctuation- fixed = fixed.replace(/([.!?])([A-Z])/g, '$1 $2'); // Ensure space after sentence endings-- return fixed;- }-- /**- * Smart sentence boundary detection and reconstruction- * @param {string} text- * @returns {string}- */- static reconstructSentences(text) {- let cleaned = text;-- // Remove orphaned punctuation and fix spacing issues- cleaned = this.cleanOrphanedPunctuation(cleaned);-- // Add periods before clear sentence starters (safe approach)- // Look for: lowercase letter + space + capitalized sentence starter- cleaned = cleaned.replace(- /([a-z])\s+(The|This|That|These|Those|I|We|You|They|He|She|It|There|Here|What|When|Where|Why|How|Who|Which|My|Our|Your|Their|His|Her|Its|Then|Now|Today|Yesterday|Tomorrow|After|Before|However|Therefore|Furthermore|Moreover|Nevertheless|Meanwhile|Finally|Initially|Subsequently|Consequently|Additionally)\b/g,- '$1. $2'- );-- // Add periods after common sentence endings that don't have punctuation- // Look for: statement-ending words + space + capitalized word- cleaned = cleaned.replace(- /\b(said|asked|replied|answered|explained|stated|mentioned|noted|added|continued|concluded|decided|agreed|disagreed|thought|believed|knew|felt|saw|heard|found|got|went|came|left|started|finished|ended|began|stopped|worked|played|studied|learned|taught|helped|tried|wanted|needed|liked|loved|hated|enjoyed|preferred|expected|hoped|wished|planned|intended|suggested|recommended|proposed|offered|promised|refused|accepted|rejected|approved|denied|confirmed|admitted|confessed|revealed|discovered|realized|understood|remembered|forgot|ignored|avoided|prevented|caused|created|built|made|did|took|gave|received|bought|sold|paid|cost|saved|spent|lost|won|failed|succeeded|improved|changed|increased|decreased|grew|shrank|expanded|reduced|opened|closed|started|stopped|turned|moved|walked|ran|drove|flew|traveled|visited|met|called|wrote|read|watched|listened|talked|spoke|discussed|argued|fought|competed|cooperated|collaborated|participated|contributed|donated|volunteered|organized|managed|led|followed|obeyed|disobeyed|respected|admired|praised|criticized|blamed|thanked|apologized|forgiven|punished|rewarded|celebrated|mourned|worried|relaxed|rested|slept|woke|ate|drank|cooked|cleaned|washed|dressed|exercised|practiced|performed|presented|demonstrated|showed|proved|tested|examined|investigated|researched|studied|analyzed|compared|contrasted|evaluated|judged|rated|scored|measured|counted|calculated|estimated|predicted|forecasted|planned|scheduled|arranged|organized|prepared|completed|accomplished|achieved|reached|attained|obtained|acquired|gained|earned|deserved|qualified|graduated|retired|resigned|quit|joined|entered|exited|arrived|departed|returned|stayed|remained|continued|persisted|persevered|survived|recovered|healed|improved|deteriorated|worsened|died|lived|existed|appeared|disappeared|emerged|vanished|occurred|happened|began|ended|lasted|continued|repeated|resumed)\s+([A-Z])/g,- '$1. $2'- );-- // Add periods before obvious new thoughts starting with conjunctions- cleaned = cleaned.replace(- /([a-z])\s+(But|And|Or|So|Yet|Still|Also|However|Therefore|Furthermore|Moreover|Nevertheless|Meanwhile|Additionally|Consequently|Subsequently|Similarly|Likewise|Otherwise|Instead|Rather|Indeed|Actually|Obviously|Clearly|Certainly|Definitely|Probably|Perhaps|Maybe|Hopefully|Unfortunately|Fortunately|Interestingly|Surprisingly|Remarkably)\b/g,- '$1. $2'- );-- return cleaned;- }-- /**- * Clean orphaned punctuation and fix spacing issues- * @param {string} text- * @returns {string}- */- static cleanOrphanedPunctuation(text) {- return text- // Remove multiple consecutive punctuation marks- .replace(/,{2,}/g, ',') // Multiple commas to single- .replace(/\.{2,}/g, '.') // Multiple periods to single- .replace(/;{2,}/g, ';') // Multiple semicolons to single- .replace(/:{2,}/g, ':') // Multiple colons to single- .replace(/!{2,}/g, '!') // Multiple exclamations to single- .replace(/\?{2,}/g, '?') // Multiple questions to single-- // Remove orphaned punctuation combinations- .replace(/\s*,\s*,/g, ',') // Double commas with spaces- .replace(/\s*\.\s*\./g, '.') // Double periods with spaces- .replace(/\s*\.\s*,/g, '.') // Period comma- .replace(/\s*,\s*\./g, '.') // Comma period- .replace(/\s*;\s*,/g, ';') // Semicolon comma- .replace(/\s*,\s*;/g, ';') // Comma semicolon- .replace(/\s*!\s*,/g, '!') // Exclamation comma- .replace(/\s*,\s*!/g, '!') // Comma exclamation- .replace(/\s*\?\s*,/g, '?') // Question comma- .replace(/\s*,\s*\?/g, '?') // Comma question-- // Remove standalone punctuation with spaces- .replace(/\s+[.,;:!?]\s*$/g, '') // Trailing orphaned punctuation- .replace(/^\s*[.,;:!?]\s+/g, '') // Leading orphaned punctuation-- // Fix spacing issues around punctuation- .replace(/\s+([.,;:!?])/g, '$1') // Remove spaces before punctuation- .replace(/([.,;:!?])\s{2,}/g, '$1 '); // Single space after punctuation- }-- /**- * Normalize whitespace - public method for minimal cleanup- * @param {string} text- * @returns {string}- */- static normalizeSpaces(text) {- if (!text) return '';- return text- .replace(/\s+/g, ' ') // Multiple spaces to single space- .replace(/[ ]+(?=[.,!?;:])/g, '') // Remove spaces before punctuation- .replace(/\n+/g, ' ') // Newlines to spaces- .replace(/\t+/g, ' ') // Tabs to spaces- .trim();- }-- /**- * Apply custom user-defined filters- * @param {string} text- * @param {Array} filters - Array of {pattern, replacement} objects- * @returns {string}- */- static applyCustomFilters(text, filters) {- let filtered = text;-- filters.forEach(filter => {- if (filter.pattern && typeof filter.replacement !== 'undefined') {- const regex = new RegExp(filter.pattern, filter.flags || 'gi');- filtered = filtered.replace(regex, filter.replacement);- }- });-- return filtered;- }-- /**- * Capitalize first letter of text- * @param {string} text- * @returns {string}- */- static capitalizeFirstLetter(text) {- if (!text) return text;- return text.charAt(0).toUpperCase() + text.slice(1);- }-- /**- * Ensure text ends with appropriate punctuation- * @param {string} text- * @returns {string}- */- static ensurePeriod(text) {- if (!text) return text;-- const trimmed = text.trim();- const lastChar = trimmed.slice(-1);-- // Don't add period if already has sentence-ending punctuation- if (this.SENTENCE_ENDERS.test(lastChar)) {- return trimmed;- }-- return trimmed + '.';- }-- /**- * Batch process multiple texts for better performance- * @param {Array<string>} texts- * @param {Object} options- * @returns {Array<string>}- */- static batchClean(texts, options = {}) {- if (!Array.isArray(texts)) {- return [];- }-- // Process in chunks for better memory management- const chunkSize = options.chunkSize || 100;- const results = [];-- for (let i = 0; i < texts.length; i += chunkSize) {- const chunk = texts.slice(i, i + chunkSize);- const cleanedChunk = chunk.map(text => this.cleanTranscription(text, options));- results.push(...cleanedChunk);- }-- return results;- }-- /**- * Get performance metrics for cleanup operation- * @param {string} text- * @param {Object} options- * @returns {Object}- */- static cleanWithMetrics(text, options = {}) {- const start = performance.now();- const originalLength = text.length;-- const cleaned = this.cleanTranscription(text, options);-- const end = performance.now();- const processingTime = end - start;- const cleanedLength = cleaned.length;- const compressionRatio = originalLength > 0 ? (cleanedLength / originalLength) : 0;-- return {- cleaned,- metrics: {- originalLength,- cleanedLength,- compressionRatio,- processingTime,- charactersRemoved: originalLength - cleanedLength,- }- };- }-}--export default TextCleanup;