Skip to content

Voice-to-text dictation app with local (Nvidia Parakeet/Whisper) and cloud models (BYOK). Privacy-first and available cross-platform.

refactor: Add cleanup docs/script, tighten onboarding hotkey/API key UX, fix timestamps, and drop legacy text cleanup

Gabriel Stein363d418e8ade
diff --git a/TROUBLESHOOTING.md b/TROUBLESHOOTING.md
new file mode 100644
index 000000000..ca42d8d89
--- /dev/null
+++ b/TROUBLESHOOTING.md
@@ -0,0 +1,63 @@
+# OpenWhispr Troubleshooting
+
+## Quick Diagnostics
+
+| Check | Command |
+| --- | --- |
+| Host architecture | `uname -m` |
+| Node architecture | `node -p "process.arch"` |
+| Native module arch | `file node_modules/better-sqlite3/build/Release/better_sqlite3.node` |
+| FFmpeg availability | `ffmpeg -version` or `node -e "console.log(require('ffmpeg-static'))"` |
+| Python install | `python3 --version && which python3` |
+| Whisper cache size | `du -sh ~/Library/Application\ Support/whisper ~/.cache/whisper 2>/dev/null` |
+
+If any of these look wrong (e.g., `x86_64` binaries on an Apple Silicon host), run `scripts/complete-uninstall.sh`, then reinstall dependencies.
+
+## Common Issues
+
+### 1. Architecture mismatch (arm64 vs x86_64)
+**Symptoms:** the renderer crashes on launch, or `better_sqlite3.node` throws “wrong architecture” errors.
+
+**Resolution:**
+1. Run the diagnostics above. If `uname -m` reports `arm64` but `node -p "process.arch"` or `file …better_sqlite3.node` shows `x86_64`, Node was installed with Rosetta.
+2. Uninstall the mismatched Node build (`brew uninstall node` or remove `/usr/local/bin/node`).
+3. Reinstall an ARM build: `arch -arm64 brew install node@20` or download the arm64 pkg from nodejs.org.
+4. Remove residual artifacts: run `scripts/complete-uninstall.sh` and allow it to delete caches.
+5. Recreate dependencies: `rm -rf node_modules package-lock.json && npm ci`.
+6. Rebuild (`npm run dev` or `npm run build`) and confirm onboarding starts normally.
+
+### 2. “you” transcription bug
+**Symptoms:** every transcription history entry shows the word “you”, even when you speak full sentences.
+
+**Cause:** Whisper produced an empty/near-empty result and the legacy fallback string (`"you"`) is being saved. This usually happens when:
+- the microphone permission was revoked mid-session,
+- stale local Whisper caches contain zero-byte clips,
+- or a hotkey collision triggers dictation without audio input.
+
+**Resolution:**
+1. Confirm microphone + accessibility permissions via macOS Settings, then re-run onboarding step 3 to re-test.
+2. Clear SQLite history and caches: `scripts/complete-uninstall.sh` (answer “y” to Whisper model removal) or manually delete the folders listed in `CLEANUP_INSTRUCTIONS.md`.
+3. Reinstall Whisper models from the Control Panel or rerun onboarding’s model picker.
+4. Try a different dictation hotkey—reserved accelerators are now blocked in step 4 and settings.
+5. If you rely on cloud transcription, verify your OpenAI key is valid (link in the UI now opens `https://platform.openai.com`).
+
+### 3. FFmpeg missing
+**Symptoms:** Local transcription fails immediately with “FFmpeg not found” or `ffmpeg-static` resolves to `null`.
+
+**Resolution:**
+1. Run `ffmpeg -version`. If the command is missing, install via `brew install ffmpeg` (Apple Silicon) or add it to your PATH.
+2. Ensure npm dependencies pulled the static binary: `rm -rf node_modules package-lock.json && npm ci`.
+3. Execute `npm run setup` so the bridge verifies FFmpeg before launching Electron.
+4. If the packaged app still cannot find FFmpeg, run `scripts/complete-uninstall.sh` to clear frozen binaries, then rebuild via `npm run build:mac`.
+
+### 4. Python install issues
+**Symptoms:** Local Whisper install fails, or `window.electronAPI.installPython` never completes.
+
+**Resolution:**
+1. Verify you have a working Python 3: `python3 --version` and `which python3`. Apple Silicon should point to `/usr/bin/python3` or a Homebrew arm64 path.
+2. Install Xcode Command Line Tools (`xcode-select --install`) so compilation helpers are available.
+3. From the running app, open Settings → Whisper Models and click “Install Python” (uses the IPC bridge) or manually run the helper script `python3 setup.py` if supplied.
+4. Delete `~/Library/Application Support/whisper` if partially downloaded models exist, then try again.
+5. As a last resort, run `scripts/complete-uninstall.sh`, reboot, and rerun onboarding so the installer can download a clean toolchain.
+
+Need deeper help? Capture the diagnostic command output and attach it to an issue so we can see which stage failed.
diff --git a/package-lock.json b/package-lock.json
index 22d03c4f2..ceea79b14 100644
--- a/package-lock.json
+++ b/package-lock.json
@@ -1,12 +1,12 @@
{
"name": "open-whispr",
- "version": "1.0.11",
+ "version": "1.0.12",
"lockfileVersion": 3,
"requires": true,
"packages": {
"": {
"name": "open-whispr",
- "version": "1.0.11",
+ "version": "1.0.12",
"hasInstallScript": true,
"license": "MIT",
"dependencies": {
diff --git a/package.json b/package.json
index 6d41feb2e..bc5bb5161 100644
--- a/package.json
+++ b/package.json
@@ -1,6 +1,6 @@
{
"name": "open-whispr",
- "version": "1.0.11",
+ "version": "1.0.12",
"description": "A desktop dictation application using OpenAI Whisper for speech-to-text transcription",
"main": "main.js",
"private": true,
diff --git a/scripts/complete-uninstall.sh b/scripts/complete-uninstall.sh
new file mode 100755
index 000000000..447068f7b
--- /dev/null
+++ b/scripts/complete-uninstall.sh
@@ -0,0 +1,79 @@
+#!/usr/bin/env bash
+
+set -euo pipefail
+
+PROJECT_ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
+
+echo "⚠️ This script will stop OpenWhispr, remove the installed app, and delete caches, databases, and preferences."
+read -r -p "Continue with the full uninstall? [y/N]: " confirm
+if [[ ! "$confirm" =~ ^[Yy]$ ]]; then
+ echo "Aborted."
+ exit 0
+fi
+
+remove_target() {
+ local target="$1"
+ if [[ -e "$target" ]]; then
+ echo "Removing $target"
+ rm -rf "$target" 2>/dev/null || sudo rm -rf "$target"
+ fi
+}
+
+echo "Stopping running OpenWhispr/Electron processes..."
+pkill -f "OpenWhispr" 2>/dev/null || true
+pkill -f "open-whispr" 2>/dev/null || true
+pkill -f "Electron Helper.*OpenWhispr" 2>/dev/null || true
+
+echo "Removing /Applications/OpenWhispr.app (requires admin)..."
+remove_target "/Applications/OpenWhispr.app"
+
+echo "Purging Application Support data..."
+remove_target "$HOME/Library/Application Support/OpenWhispr"
+remove_target "$HOME/Library/Application Support/open-whispr"
+remove_target "$HOME/Library/Application Support/OpenWhispr-dev"
+remove_target "$HOME/Library/Application Support/com.openwhispr"
+remove_target "$HOME/Library/Application Support/com.openwhispr.OpenWhispr"
+
+echo "Removing caches, logs, and saved state..."
+remove_target "$HOME/Library/Caches/open-whispr"
+remove_target "$HOME/Library/Caches/com.openwhispr.OpenWhispr"
+remove_target "$HOME/Library/Preferences/com.openwhispr.OpenWhispr.plist"
+remove_target "$HOME/Library/Preferences/com.openwhispr.helper.plist"
+remove_target "$HOME/Library/Logs/OpenWhispr"
+remove_target "$HOME/Library/Saved Application State/com.openwhispr.OpenWhispr.savedState"
+
+echo "Cleaning temporary files..."
+shopt -s nullglob
+for tmp in /tmp/openwhispr*; do
+ remove_target "$tmp"
+done
+for crash in "$HOME/Library/Application Support/CrashReporter"/OpenWhispr_*; do
+ remove_target "$crash"
+done
+shopt -u nullglob
+
+read -r -p "Remove downloaded Whisper models and caches (~/.cache/whisper, ~/Library/Application Support/whisper)? [y/N]: " wipe_models
+if [[ "$wipe_models" =~ ^[Yy]$ ]]; then
+ remove_target "$HOME/.cache/whisper"
+ remove_target "$HOME/Library/Application Support/whisper"
+ remove_target "$HOME/Library/Application Support/OpenWhispr/models"
+fi
+
+ENV_FILE="$PROJECT_ROOT/.env"
+if [[ -f "$ENV_FILE" ]]; then
+ read -r -p "Remove the local environment file at $ENV_FILE? [y/N]: " wipe_env
+ if [[ "$wipe_env" =~ ^[Yy]$ ]]; then
+ echo "Removing $ENV_FILE"
+ rm -f "$ENV_FILE"
+ fi
+fi
+
+cat <<'EOF'
+macOS keeps microphone, screen recording, and accessibility approvals even after files are removed.
+Reset them if you want a truly fresh start:
+ tccutil reset Microphone com.openwhispr.app
+ tccutil reset Accessibility com.openwhispr.app
+ tccutil reset ScreenCapture com.openwhispr.app
+
+Full uninstall complete. Reboot if you removed permissions, then reinstall or run npm scripts on a clean tree.
+EOF
diff --git a/src/components/AIModelSelectorEnhanced.tsx b/src/components/AIModelSelectorEnhanced.tsx
index f0e4ac7a6..abe8ec185 100644
--- a/src/components/AIModelSelectorEnhanced.tsx
+++ b/src/components/AIModelSelectorEnhanced.tsx
@@ -742,7 +742,19 @@ export default function AIModelSelectorEnhanced({
<ApiKeyInput
apiKey={openaiApiKey}
setApiKey={setOpenaiApiKey}
- helpText="Get your API key from platform.openai.com"
+ helpText={
+ <>
+ Need an API key?{" "}
+ <a
+ href="https://platform.openai.com"
+ target="_blank"
+ rel="noopener noreferrer"
+ className="text-blue-600 underline"
+ >
+ platform.openai.com
+ </a>
+ </>
+ }
/>
</div>
)}
@@ -921,4 +933,4 @@ export default function AIModelSelectorEnhanced({
)}
</div>
);
-}
\ No newline at end of file
+}
diff --git a/src/components/OnboardingFlow.tsx b/src/components/OnboardingFlow.tsx
index 882a9bccf..7ebaa8e2b 100644
--- a/src/components/OnboardingFlow.tsx
+++ b/src/components/OnboardingFlow.tsx
@@ -296,6 +296,34 @@ export default function OnboardingFlow({ onComplete }: OnboardingFlowProps) {
}
}, [currentStep]);
+ const ensureHotkeyRegistered = useCallback(async () => {
+ if (!window.electronAPI?.updateHotkey) {
+ return true;
+ }
+
+ try {
+ const result = await window.electronAPI.updateHotkey(hotkey);
+ if (result && !result.success) {
+ showAlertDialog({
+ title: "Hotkey Not Registered",
+ description:
+ result.message ||
+ "We couldn't register that key. Please choose another hotkey.",
+ });
+ return false;
+ }
+ return true;
+ } catch (error) {
+ console.error("Failed to register onboarding hotkey", error);
+ showAlertDialog({
+ title: "Hotkey Error",
+ description:
+ "We couldn't register that key. Please choose another hotkey.",
+ });
+ return false;
+ }
+ }, [hotkey, showAlertDialog]);
+
const saveSettings = useCallback(async () => {
const normalizedTranscriptionBase = (transcriptionBaseUrl || '').trim();
const normalizedReasoningBaseValue = (reasoningBaseUrl || '').trim();
@@ -313,24 +341,11 @@ export default function OnboardingFlow({ onComplete }: OnboardingFlowProps) {
reasoningModel,
cloudReasoningBaseUrl: normalizedReasoningBaseValue,
});
- setDictationKey(hotkey);
- try {
- const result = await window.electronAPI?.updateHotkey?.(hotkey);
- if (result && !result.success) {
- showAlertDialog({
- title: "Hotkey Not Registered",
- description:
- result.message ||
- "We couldn't register that key. Please choose another hotkey.",
- });
- }
- } catch (error) {
- console.error("Failed to register onboarding hotkey", error);
- showAlertDialog({
- title: "Hotkey Error",
- description: "We couldn't register that key. Please choose another hotkey.",
- });
+ const hotkeyRegistered = await ensureHotkeyRegistered();
+ if (!hotkeyRegistered) {
+ return false;
}
+ setDictationKey(hotkey);
saveAgentName(agentName);
localStorage.setItem(
@@ -342,11 +357,15 @@ export default function OnboardingFlow({ onComplete }: OnboardingFlowProps) {
permissionsHook.accessibilityPermissionGranted.toString()
);
localStorage.setItem("onboardingCompleted", "true");
+ const trimmedApiKey = apiKey.trim();
+ const skipAuth = trimmedApiKey.length === 0;
+ localStorage.setItem("skipAuth", skipAuth.toString());
- if (!useLocalWhisper && apiKey.trim()) {
- await window.electronAPI.saveOpenAIKey(apiKey);
- updateApiKeys({ openaiApiKey: apiKey });
+ if (!useLocalWhisper && trimmedApiKey) {
+ await window.electronAPI.saveOpenAIKey(trimmedApiKey);
+ updateApiKeys({ openaiApiKey: trimmedApiKey });
}
+ return true;
}, [
whisperModel,
hotkey,
@@ -364,21 +383,33 @@ export default function OnboardingFlow({ onComplete }: OnboardingFlowProps) {
setCloudTranscriptionBaseUrl,
setCloudReasoningBaseUrl,
setDictationKey,
+ ensureHotkeyRegistered,
]);
- const nextStep = useCallback(() => {
- if (currentStep < steps.length - 1) {
- const newStep = currentStep + 1;
- setCurrentStep(newStep);
+ const nextStep = useCallback(async () => {
+ if (currentStep >= steps.length - 1) {
+ return;
+ }
- // Show dictation panel when moving from permissions step (3) to hotkey step (4)
- if (currentStep === 3 && newStep === 4) {
- if (window.electronAPI?.showDictationPanel) {
- window.electronAPI.showDictationPanel();
- }
+ const newStep = currentStep + 1;
+
+ if (currentStep === 4) {
+ const registered = await ensureHotkeyRegistered();
+ if (!registered) {
+ return;
+ }
+ setDictationKey(hotkey);
+ }
+
+ setCurrentStep(newStep);
+
+ // Show dictation panel when moving from permissions step (3) to hotkey step (4)
+ if (currentStep === 3 && newStep === 4) {
+ if (window.electronAPI?.showDictationPanel) {
+ window.electronAPI.showDictationPanel();
}
}
- }, [currentStep, setCurrentStep, steps.length]);
+ }, [currentStep, ensureHotkeyRegistered, hotkey, setCurrentStep, setDictationKey, steps.length]);
const prevStep = useCallback(() => {
if (currentStep > 0) {
@@ -388,7 +419,10 @@ export default function OnboardingFlow({ onComplete }: OnboardingFlowProps) {
}, [currentStep, setCurrentStep]);
const finishOnboarding = useCallback(async () => {
- await saveSettings();
+ const saved = await saveSettings();
+ if (!saved) {
+ return;
+ }
// Clear the onboarding step since we're done
removeCurrentStep();
onComplete();
@@ -606,7 +640,19 @@ export default function OnboardingFlow({ onComplete }: OnboardingFlowProps) {
apiKey={apiKey}
setApiKey={setApiKey}
label="OpenAI API Key"
- helpText="Get your API key from platform.openai.com"
+ helpText={
+ <>
+ Need an API key?{" "}
+ <a
+ href="https://platform.openai.com"
+ target="_blank"
+ rel="noopener noreferrer"
+ className="text-blue-600 underline"
+ >
+ platform.openai.com
+ </a>
+ </>
+ }
/>
<div className="space-y-2">
diff --git a/src/components/SettingsPage.tsx b/src/components/SettingsPage.tsx
index ea51affa7..8103ba4db 100644
--- a/src/components/SettingsPage.tsx
+++ b/src/components/SettingsPage.tsx
@@ -916,7 +916,20 @@ export default function SettingsPage({
<ApiKeyInput
apiKey={openaiApiKey}
setApiKey={setOpenaiApiKey}
- helpText="Supports OpenAI or compatible endpoints"
+ helpText={
+ <>
+ Supports OpenAI or compatible endpoints.{" "}
+ <a
+ href="https://platform.openai.com"
+ target="_blank"
+ rel="noopener noreferrer"
+ className="text-blue-600 underline"
+ >
+ Get an API key
+ </a>
+ .
+ </>
+ }
/>
<div className="space-y-2">
<label className="block text-sm font-medium text-blue-900">
diff --git a/src/components/ui/ApiKeyInput.tsx b/src/components/ui/ApiKeyInput.tsx
index 1ec670a34..c217966b7 100644
--- a/src/components/ui/ApiKeyInput.tsx
+++ b/src/components/ui/ApiKeyInput.tsx
@@ -9,7 +9,7 @@ interface ApiKeyInputProps {
className?: string;
placeholder?: string;
label?: string;
- helpText?: string;
+ helpText?: React.ReactNode;
variant?: "default" | "purple";
}
@@ -53,7 +53,9 @@ export default function ApiKeyInput({
Paste
</Button>
</div>
- <p className="text-xs text-neutral-600 mt-2">{helpText}</p>
+ {helpText && (
+ <p className="text-xs text-neutral-600 mt-2">{helpText}</p>
+ )}
</div>
);
}
diff --git a/src/components/ui/TranscriptionItem.tsx b/src/components/ui/TranscriptionItem.tsx
index 4db48f1a4..cc6d39bea 100644
--- a/src/components/ui/TranscriptionItem.tsx
+++ b/src/components/ui/TranscriptionItem.tsx
@@ -18,6 +18,19 @@ export default function TranscriptionItem({
onCopy,
onDelete,
}: TranscriptionItemProps) {
+ const timestampSource = item.timestamp.endsWith("Z")
+ ? item.timestamp
+ : `${item.timestamp}Z`;
+ const timestampDate = new Date(timestampSource);
+ const formattedTimestamp = Number.isNaN(timestampDate.getTime())
+ ? item.timestamp
+ : timestampDate.toLocaleString("en-US", {
+ month: "short",
+ day: "numeric",
+ hour: "2-digit",
+ minute: "2-digit",
+ });
+
return (
<div className="relative bg-gradient-to-b from-blue-50/30 to-white rounded-xl shadow-sm hover:shadow-md transition-shadow">
<div className="p-6 pl-16" style={{ paddingTop: "8px" }}>
@@ -32,12 +45,7 @@ export default function TranscriptionItem({
</span>
<div className="w-px h-3 bg-neutral-300" />
<span className="text-xs text-neutral-500">
- {new Date(item.timestamp).toLocaleString("en-US", {
- month: "short",
- day: "numeric",
- hour: "2-digit",
- minute: "2-digit",
- })}
+ {formattedTimestamp}
</span>
</div>
<p
diff --git a/src/helpers/audioManager.js b/src/helpers/audioManager.js
index ff375b480..29e96abc1 100644
--- a/src/helpers/audioManager.js
+++ b/src/helpers/audioManager.js
@@ -1,4 +1,3 @@
-import TextCleanup from "../utils/textCleanup";
import ReasoningService from "../services/ReasoningService";
import { API_ENDPOINTS, buildApiUrl, normalizeBaseUrl } from "../config/constants";
@@ -138,24 +137,6 @@ class AudioManager {
}
}
- static cleanTranscription(text, options = {}) {
- return TextCleanup.cleanTranscription(text, {
- removeArtifacts: true,
- normalizeSpaces: true,
- fixPunctuation: true,
- removeFillers: true,
- removeRepetitions: true,
- capitalizeFirst: true,
- addPeriod: false,
- ...options,
- });
- }
-
- static cleanTranscriptionForAPI(text) {
- // Minimal cleanup - only normalize spaces for API processing
- return TextCleanup.normalizeSpaces(text);
- }
-
async processWithLocalWhisper(audioBlob, model = "base") {
// Analyze audio levels first
@@ -457,12 +438,13 @@ class AudioManager {
}
async processTranscription(text, source) {
+ const normalizedText = typeof text === "string" ? text.trim() : "";
// Log incoming transcription
debugLogger.logReasoning("TRANSCRIPTION_RECEIVED", {
source,
- textLength: text.length,
- textPreview: text.substring(0, 100) + (text.length > 100 ? "..." : ""),
+ textLength: normalizedText.length,
+ textPreview: normalizedText.substring(0, 100) + (normalizedText.length > 100 ? "..." : ""),
timestamp: new Date().toISOString()
});
@@ -490,7 +472,7 @@ class AudioManager {
if (useReasoning) {
try {
// Minimal cleanup for reasoning models
- const preparedText = AudioManager.cleanTranscriptionForAPI(text);
+ const preparedText = normalizedText;
debugLogger.logReasoning("SENDING_TO_REASONING", {
preparedTextLength: preparedText.length,
@@ -523,7 +505,7 @@ class AudioManager {
});
// Standard cleanup when reasoning is unavailable or fails
- return AudioManager.cleanTranscription(text);
+ return normalizedText;
}
async processWithOpenAIAPI(audioBlob) {
diff --git a/src/types/electron.ts b/src/types/electron.ts
index 354ae981e..a9866f339 100644
--- a/src/types/electron.ts
+++ b/src/types/electron.ts
@@ -250,7 +250,7 @@ declare global {
removeAllListeners: (channel: string) => void;
// Hotkey management
- updateHotkey: (key: string) => Promise<void>;
+ updateHotkey: (key: string) => Promise<{ success: boolean; message: string }>;
// Gemini API key management
getGeminiKey: () => Promise<string | null>;
diff --git a/src/utils/textCleanup.js b/src/utils/textCleanup.js
deleted file mode 100644
index 1af4053ce..000000000
--- a/src/utils/textCleanup.js
+++ /dev/null
@@ -1,462 +0,0 @@
-/**
- * Industry-standard text cleanup utility for transcription processing
- * Optimized for performance with batch processing and smart filtering
- */
-
-class TextCleanup {
- // Optimized patterns for transcription cleanup
- static PATTERNS = {
- // Non-verbal artifacts in brackets, parentheses, or asterisks
- nonVerbal: [
- /\[(?:laughter|music|noise|crosstalk|applause|coughing|inaudible|unclear|pause|silence|background|static|beep|mumbling|whispering|breathing|sighing|yawning|typing|footsteps|door|phone|rain|wind|traffic|crowd)\]/gi,
- /\((?:laughter|music|noise|crosstalk|applause|coughing|inaudible|unclear|pause|silence|background|static|beep|mumbling|whispering|breathing|sighing|yawning|typing|footsteps|door|phone|rain|wind|traffic|crowd)\)/gi,
- /\*(?:laughter|music|noise|crosstalk|applause|coughing|inaudible|unclear|pause|silence|background|static|beep|mumbling|whispering|breathing|sighing|yawning|typing|footsteps|door|phone|rain|wind|traffic|crowd)\*/gi,
- ],
-
- // Speaker labels and metadata
- speakers: [
- /^Speaker\s*\d+:\s*/gmi,
- /^\w+:\s*/gm,
- /\[\d{1,2}:\d{2}(:\d{2})?\]/g, // Timestamps
- /\<\d+\.\d+\>/g, // Confidence scores
- /\[SPEAKER_\d+\]/gi,
- /\[MUSIC\]/gi,
- /\[SOUND\]/gi,
- ],
-
- // Enhanced filler words - industry standard
- fillers: [
- /\b(?:um|uh|er|ah|mm|hmm|mhm)\b\s*/gi, // Hesitation sounds
- /\b(?:like|so|well|right|okay|ok|yeah|yep|yup|sure|alright)\b\s*/gi, // Common discourse markers
- /\b(?:you know|I mean|I guess|sort of|kind of)\b\s*/gi, // Meaningless phrases and uncertainty markers
- /\b(?:maybe|perhaps)\b(?=\s)/gi, // Uncertainty markers (when not essential)
- /\b(?:basically|literally|totally|honestly|obviously|definitely|probably|actually)\b\s*/gi, // Overused intensifiers
- /\bOh,?\s+(?=\w)/gi, // "Oh" when followed by another word (often just filler)
- ],
-
- // Repetition patterns
- repetitions: [
- /\b(\w+)\s+\1\b/gi, // Doubled words
- /\b(\w+)\s+\1\s+\1\b/gi, // Tripled words
- /\b(\w{2,})\s+\1\s+\1\s+\1\b/gi, // Quadrupled words
- /\b(\w+[-']\w+)\s+\1\b/gi, // Repeated hyphenated/contracted
- /\b([a-z]{1,3})\s+\1\s+\1\b/gi, // Stuttering (short words)
- ],
-
- // Smart quotes and punctuation normalization
- quotes: [
- /[“”]/g, // Smart quotes to regular quotes
- /[‘’]/g, // Smart apostrophes to regular
- /…/g, // Ellipsis to three periods
- ],
-
- // Excessive punctuation
- punctuation: [
- /\.{4,}/g, // 4+ periods to ellipsis
- /\.{3}\s*\.+/g, // Ellipsis + more periods
- /\?{2,}/g, // Multiple question marks
- /!{2,}/g, // Multiple exclamation marks
- /,{2,}/g, // Multiple commas
- /;{2,}/g, // Multiple semicolons
- /:{2,}/g, // Multiple colons
- /-{3,}/g, // Long dashes to em-dash
- ],
- };
-
- // Common contractions and their expansions
- static CONTRACTIONS = {
- "won't": "will not",
- "can't": "cannot",
- "n't": " not",
- "'re": " are",
- "'ve": " have",
- "'ll": " will",
- "'d": " would",
- "'m": " am",
- "'s": " is", // Note: This is ambiguous ('s could be 'is' or possessive)
- };
-
- // Sentence-ending punctuation
- static SENTENCE_ENDERS = /[.!?]/;
-
- // Common sentence starters (capitalized words that typically start sentences)
- static SENTENCE_STARTERS = /\b(?:The|This|That|These|Those|A|An|I|We|You|They|He|She|It|There|Here|What|When|Where|Why|How|Who|Which|My|Our|Your|Their|His|Her|Its|Some|Many|Most|All|Every|Each|One|Two|First|Second|Next|Then|Now|Today|Yesterday|Tomorrow|After|Before|During|While|Since|Because|Although|However|Therefore|Furthermore|Moreover|Nevertheless|Meanwhile|Finally|Initially|Subsequently|Consequently|Additionally|Specifically|Particularly|Essentially|Actually|Obviously|Clearly|Certainly|Definitely|Probably|Perhaps|Maybe|Sometimes|Often|Usually|Always|Never|Once|Again|Still|Yet|Already|Just|Only|Even|Also|Too|Very|Really|Quite|Rather|Extremely|Completely|Absolutely|Totally|Entirely|Exactly|Precisely|Approximately|About|Around|Nearly|Almost|Especially|Particularly|Generally|Typically|Normally|Usually|Frequently|Occasionally|Rarely|Seldom)\b/;
-
- /**
- * Main cleanup function - processes text with all cleanup rules
- * @param {string} text - Raw transcription text
- * @param {Object} options - Cleanup options
- * @returns {string} - Cleaned text
- */
- static cleanTranscription(text, options = {}) {
- const {
- removeArtifacts = true,
- normalizeSpaces = true,
- fixPunctuation = true,
- expandContractions = false,
- removeFillers = true,
- removeRepetitions = true,
- capitalizeFirst = true,
- addPeriod = true,
- customFilters = [],
- preserveEmphasis = false,
- } = options;
-
- if (!text || typeof text !== 'string') {
- return '';
- }
-
- // Start with trimming
- let cleanedText = text.trim();
-
- // Early return for empty text
- if (!cleanedText) {
- return '';
- }
-
- // Apply cleanup steps in order of importance
- if (removeArtifacts) {
- cleanedText = this.removeNonVerbalArtifacts(cleanedText);
- }
-
- if (removeFillers) {
- cleanedText = this.removeFillerWords(cleanedText);
- }
-
- if (removeRepetitions) {
- cleanedText = this.removeRepeatedWords(cleanedText);
- }
-
- if (expandContractions) {
- cleanedText = this.expandContractions(cleanedText);
- }
-
- // Smart sentence reconstruction
- cleanedText = this.reconstructSentences(cleanedText);
-
- if (fixPunctuation) {
- cleanedText = this.fixPunctuation(cleanedText);
- }
-
- if (normalizeSpaces) {
- cleanedText = this.normalizeSpaces(cleanedText);
- }
-
- // Apply custom filters
- if (customFilters.length > 0) {
- cleanedText = this.applyCustomFilters(cleanedText, customFilters);
- }
-
- // Final formatting
- if (capitalizeFirst) {
- cleanedText = this.capitalizeFirstLetter(cleanedText);
- }
-
- if (addPeriod) {
- cleanedText = this.ensurePeriod(cleanedText);
- }
-
- return cleanedText.trim();
- }
-
- /**
- * Remove non-verbal artifacts and speaker metadata
- * @param {string} text
- * @returns {string}
- */
- static removeNonVerbalArtifacts(text) {
- let cleaned = text;
-
- // Remove non-verbal artifacts
- this.PATTERNS.nonVerbal.forEach(pattern => {
- cleaned = cleaned.replace(pattern, ' ');
- });
-
- // Remove speaker artifacts
- this.PATTERNS.speakers.forEach(pattern => {
- cleaned = cleaned.replace(pattern, '');
- });
-
- return cleaned;
- }
-
- /**
- * Remove common filler words - enhanced industry standard approach
- * @param {string} text
- * @returns {string}
- */
- static removeFillerWords(text) {
- let cleaned = text;
-
- // Apply enhanced filler removal patterns
- this.PATTERNS.fillers.forEach(pattern => {
- cleaned = cleaned.replace(pattern, ' ');
- });
-
- return cleaned;
- }
-
- /**
- * Remove repeated words - enhanced pattern detection
- * @param {string} text
- * @returns {string}
- */
- static removeRepeatedWords(text) {
- let cleaned = text;
-
- // Apply all repetition patterns
- this.PATTERNS.repetitions.forEach(pattern => {
- cleaned = cleaned.replace(pattern, '$1');
- });
-
- return cleaned;
- }
-
- /**
- * Expand common contractions
- * @param {string} text
- * @returns {string}
- */
- static expandContractions(text) {
- let expanded = text;
-
- Object.entries(this.CONTRACTIONS).forEach(([contraction, expansion]) => {
- const regex = new RegExp(contraction.replace(/'/g, "'?"), 'gi');
- expanded = expanded.replace(regex, expansion);
- });
-
- return expanded;
- }
-
- /**
- * Fix punctuation issues - enhanced industry standard approach
- * @param {string} text
- * @returns {string}
- */
- static fixPunctuation(text) {
- let fixed = text;
-
- // Normalize smart quotes first
- this.PATTERNS.quotes.forEach((pattern, index) => {
- if (index === 0) {
- fixed = fixed.replace(pattern, '"'); // Smart quotes to regular
- } else if (index === 1) {
- fixed = fixed.replace(pattern, "'"); // Smart apostrophes to regular
- } else if (index === 2) {
- fixed = fixed.replace(pattern, '...'); // Ellipsis to three periods
- }
- });
-
- // Fix excessive punctuation using patterns
- this.PATTERNS.punctuation.forEach((pattern, index) => {
- if (pattern.toString().includes('{4,}')) {
- fixed = fixed.replace(pattern, '...'); // 4+ periods to ellipsis
- } else if (pattern.toString().includes('\\?{2,}')) {
- fixed = fixed.replace(pattern, '?'); // Multiple ? to single
- } else if (pattern.toString().includes('!{2,}')) {
- fixed = fixed.replace(pattern, '!'); // Multiple ! to single
- } else if (pattern.toString().includes(',{2,}')) {
- fixed = fixed.replace(pattern, ','); // Multiple , to single
- } else if (pattern.toString().includes('-{3,}')) {
- fixed = fixed.replace(pattern, '—'); // Long dashes to em-dash
- } else {
- fixed = fixed.replace(pattern, (match) => match.charAt(0)); // Other multiples to single
- }
- });
-
- // Fix spacing around punctuation (industry standard)
- fixed = fixed.replace(/\s+([,.!?;:])/g, '$1'); // Remove space before punctuation
- fixed = fixed.replace(/([,.!?;:])\s{2,}/g, '$1 '); // Single space after punctuation
- fixed = fixed.replace(/([.!?])([A-Z])/g, '$1 $2'); // Ensure space after sentence endings
-
- return fixed;
- }
-
- /**
- * Smart sentence boundary detection and reconstruction
- * @param {string} text
- * @returns {string}
- */
- static reconstructSentences(text) {
- let cleaned = text;
-
- // Remove orphaned punctuation and fix spacing issues
- cleaned = this.cleanOrphanedPunctuation(cleaned);
-
- // Add periods before clear sentence starters (safe approach)
- // Look for: lowercase letter + space + capitalized sentence starter
- cleaned = cleaned.replace(
- /([a-z])\s+(The|This|That|These|Those|I|We|You|They|He|She|It|There|Here|What|When|Where|Why|How|Who|Which|My|Our|Your|Their|His|Her|Its|Then|Now|Today|Yesterday|Tomorrow|After|Before|However|Therefore|Furthermore|Moreover|Nevertheless|Meanwhile|Finally|Initially|Subsequently|Consequently|Additionally)\b/g,
- '$1. $2'
- );
-
- // Add periods after common sentence endings that don't have punctuation
- // Look for: statement-ending words + space + capitalized word
- cleaned = cleaned.replace(
- /\b(said|asked|replied|answered|explained|stated|mentioned|noted|added|continued|concluded|decided|agreed|disagreed|thought|believed|knew|felt|saw|heard|found|got|went|came|left|started|finished|ended|began|stopped|worked|played|studied|learned|taught|helped|tried|wanted|needed|liked|loved|hated|enjoyed|preferred|expected|hoped|wished|planned|intended|suggested|recommended|proposed|offered|promised|refused|accepted|rejected|approved|denied|confirmed|admitted|confessed|revealed|discovered|realized|understood|remembered|forgot|ignored|avoided|prevented|caused|created|built|made|did|took|gave|received|bought|sold|paid|cost|saved|spent|lost|won|failed|succeeded|improved|changed|increased|decreased|grew|shrank|expanded|reduced|opened|closed|started|stopped|turned|moved|walked|ran|drove|flew|traveled|visited|met|called|wrote|read|watched|listened|talked|spoke|discussed|argued|fought|competed|cooperated|collaborated|participated|contributed|donated|volunteered|organized|managed|led|followed|obeyed|disobeyed|respected|admired|praised|criticized|blamed|thanked|apologized|forgiven|punished|rewarded|celebrated|mourned|worried|relaxed|rested|slept|woke|ate|drank|cooked|cleaned|washed|dressed|exercised|practiced|performed|presented|demonstrated|showed|proved|tested|examined|investigated|researched|studied|analyzed|compared|contrasted|evaluated|judged|rated|scored|measured|counted|calculated|estimated|predicted|forecasted|planned|scheduled|arranged|organized|prepared|completed|accomplished|achieved|reached|attained|obtained|acquired|gained|earned|deserved|qualified|graduated|retired|resigned|quit|joined|entered|exited|arrived|departed|returned|stayed|remained|continued|persisted|persevered|survived|recovered|healed|improved|deteriorated|worsened|died|lived|existed|appeared|disappeared|emerged|vanished|occurred|happened|began|ended|lasted|continued|repeated|resumed)\s+([A-Z])/g,
- '$1. $2'
- );
-
- // Add periods before obvious new thoughts starting with conjunctions
- cleaned = cleaned.replace(
- /([a-z])\s+(But|And|Or|So|Yet|Still|Also|However|Therefore|Furthermore|Moreover|Nevertheless|Meanwhile|Additionally|Consequently|Subsequently|Similarly|Likewise|Otherwise|Instead|Rather|Indeed|Actually|Obviously|Clearly|Certainly|Definitely|Probably|Perhaps|Maybe|Hopefully|Unfortunately|Fortunately|Interestingly|Surprisingly|Remarkably)\b/g,
- '$1. $2'
- );
-
- return cleaned;
- }
-
- /**
- * Clean orphaned punctuation and fix spacing issues
- * @param {string} text
- * @returns {string}
- */
- static cleanOrphanedPunctuation(text) {
- return text
- // Remove multiple consecutive punctuation marks
- .replace(/,{2,}/g, ',') // Multiple commas to single
- .replace(/\.{2,}/g, '.') // Multiple periods to single
- .replace(/;{2,}/g, ';') // Multiple semicolons to single
- .replace(/:{2,}/g, ':') // Multiple colons to single
- .replace(/!{2,}/g, '!') // Multiple exclamations to single
- .replace(/\?{2,}/g, '?') // Multiple questions to single
-
- // Remove orphaned punctuation combinations
- .replace(/\s*,\s*,/g, ',') // Double commas with spaces
- .replace(/\s*\.\s*\./g, '.') // Double periods with spaces
- .replace(/\s*\.\s*,/g, '.') // Period comma
- .replace(/\s*,\s*\./g, '.') // Comma period
- .replace(/\s*;\s*,/g, ';') // Semicolon comma
- .replace(/\s*,\s*;/g, ';') // Comma semicolon
- .replace(/\s*!\s*,/g, '!') // Exclamation comma
- .replace(/\s*,\s*!/g, '!') // Comma exclamation
- .replace(/\s*\?\s*,/g, '?') // Question comma
- .replace(/\s*,\s*\?/g, '?') // Comma question
-
- // Remove standalone punctuation with spaces
- .replace(/\s+[.,;:!?]\s*$/g, '') // Trailing orphaned punctuation
- .replace(/^\s*[.,;:!?]\s+/g, '') // Leading orphaned punctuation
-
- // Fix spacing issues around punctuation
- .replace(/\s+([.,;:!?])/g, '$1') // Remove spaces before punctuation
- .replace(/([.,;:!?])\s{2,}/g, '$1 '); // Single space after punctuation
- }
-
- /**
- * Normalize whitespace - public method for minimal cleanup
- * @param {string} text
- * @returns {string}
- */
- static normalizeSpaces(text) {
- if (!text) return '';
- return text
- .replace(/\s+/g, ' ') // Multiple spaces to single space
- .replace(/[ ]+(?=[.,!?;:])/g, '') // Remove spaces before punctuation
- .replace(/\n+/g, ' ') // Newlines to spaces
- .replace(/\t+/g, ' ') // Tabs to spaces
- .trim();
- }
-
- /**
- * Apply custom user-defined filters
- * @param {string} text
- * @param {Array} filters - Array of {pattern, replacement} objects
- * @returns {string}
- */
- static applyCustomFilters(text, filters) {
- let filtered = text;
-
- filters.forEach(filter => {
- if (filter.pattern && typeof filter.replacement !== 'undefined') {
- const regex = new RegExp(filter.pattern, filter.flags || 'gi');
- filtered = filtered.replace(regex, filter.replacement);
- }
- });
-
- return filtered;
- }
-
- /**
- * Capitalize first letter of text
- * @param {string} text
- * @returns {string}
- */
- static capitalizeFirstLetter(text) {
- if (!text) return text;
- return text.charAt(0).toUpperCase() + text.slice(1);
- }
-
- /**
- * Ensure text ends with appropriate punctuation
- * @param {string} text
- * @returns {string}
- */
- static ensurePeriod(text) {
- if (!text) return text;
-
- const trimmed = text.trim();
- const lastChar = trimmed.slice(-1);
-
- // Don't add period if already has sentence-ending punctuation
- if (this.SENTENCE_ENDERS.test(lastChar)) {
- return trimmed;
- }
-
- return trimmed + '.';
- }
-
- /**
- * Batch process multiple texts for better performance
- * @param {Array<string>} texts
- * @param {Object} options
- * @returns {Array<string>}
- */
- static batchClean(texts, options = {}) {
- if (!Array.isArray(texts)) {
- return [];
- }
-
- // Process in chunks for better memory management
- const chunkSize = options.chunkSize || 100;
- const results = [];
-
- for (let i = 0; i < texts.length; i += chunkSize) {
- const chunk = texts.slice(i, i + chunkSize);
- const cleanedChunk = chunk.map(text => this.cleanTranscription(text, options));
- results.push(...cleanedChunk);
- }
-
- return results;
- }
-
- /**
- * Get performance metrics for cleanup operation
- * @param {string} text
- * @param {Object} options
- * @returns {Object}
- */
- static cleanWithMetrics(text, options = {}) {
- const start = performance.now();
- const originalLength = text.length;
-
- const cleaned = this.cleanTranscription(text, options);
-
- const end = performance.now();
- const processingTime = end - start;
- const cleanedLength = cleaned.length;
- const compressionRatio = originalLength > 0 ? (cleanedLength / originalLength) : 0;
-
- return {
- cleaned,
- metrics: {
- originalLength,
- cleanedLength,
- compressionRatio,
- processingTime,
- charactersRemoved: originalLength - cleanedLength,
- }
- };
- }
-}
-
-export default TextCleanup;