Skip to content

Speakr is a personal, self-hosted web application designed for transcribing audio recordings

Python

Preserve original audio and recover Tencent COS transcription tasks

ZOLO67aef54ce0c8
diff --git a/config/env.transcription.example b/config/env.transcription.example
index a289dd2b..2045b05a 100644
--- a/config/env.transcription.example
+++ b/config/env.transcription.example
@@ -9,6 +9,8 @@
# TENCENT_ASR_TIMEOUT=1800
# TENCENT_ASR_POLL_INTERVAL=5
# TENCENT_ASR_HOTWORD_ID=your-hotword-table-id
+# TENCENT_ASR_STAGING_PREFIX=tencent-asr
+# TENCENT_ASR_STATE_DIR=/data/instance/tencent-tasks
# Files over 5 MiB require Speakr S3 storage; see docs/advanced/tencent-asr.md.
#
# Speakr supports multiple transcription providers through a connector-based
diff --git a/docs/advanced/tencent-asr.md b/docs/advanced/tencent-asr.md
index b6d42108..94a837f5 100644
--- a/docs/advanced/tencent-asr.md
+++ b/docs/advanced/tencent-asr.md
@@ -36,7 +36,11 @@ in seconds (maximum timeout 10800).
## Large files and storage
-Files up to 5 MiB are sent directly. Larger files use the existing Speakr S3
+The connector declares original-audio passthrough, so Speakr skips audio
+preprocessing (including forced WebM conversion) and the MP3 error-retry path
+for it. This applies to audio recordings; video extraction is a separate app
+feature. The connector never compresses or splits audio. Files up to 5 MiB are sent
+directly. Larger files use the existing Speakr S3
backend: configure `S3_BUCKET_NAME`, `S3_REGION`, `S3_ENDPOINT_URL`,
`S3_ACCESS_KEY_ID` and `S3_SECRET_ACCESS_KEY`. The endpoint and signed download
URL must be reachable by Tencent over the public internet; a NAS-only address
@@ -45,20 +49,73 @@ S3-compatible client, not a separate COS SDK; verify your chosen storage
provider's compatibility before relying on it. Local recording storage can
remain selected (`FILE_STORAGE_BACKEND=local`).
-Staging objects use the `tencent-asr/` prefix and 24-hour signed URLs. They are
+Tencent COS example (use your own bucket, credentials and region):
+
+```env
+FILE_STORAGE_BACKEND=local
+S3_BUCKET_NAME=your-bucket-1234567890
+S3_REGION=ap-hongkong
+S3_ENDPOINT_URL=https://cos.ap-hongkong.myqcloud.com
+S3_ACCESS_KEY_ID=your-cos-secret-id
+S3_SECRET_ACCESS_KEY=your-cos-secret-key
+S3_USE_PATH_STYLE=false
+S3_VERIFY_SSL=true
+TENCENT_ASR_STAGING_PREFIX=recordings/speakr
+```
+
+These storage credentials need object upload, read and delete permissions.
+They may differ from the ASR credentials. COS uses virtual-host addressing;
+the backend disables optional AWS checksum trailers for COS compatibility.
+Staging uploads explicitly set a private object ACL, even for an existing
+image bucket. Check that no bucket policy independently grants public access.
+There is no requirement to store audio with Alibaba or AWS.
+
+Staging objects use the configurable `TENCENT_ASR_STAGING_PREFIX` (default
+`tencent-asr/`) and 24-hour signed URLs. They are
deleted after a terminal task result. Configure a bucket lifecycle rule to
expire this prefix after at least two days: timed-out tasks, interrupted
workers, and uncertain submissions retain their objects so Tencent can finish
downloading them. These objects contain audio; storage charges may apply.
-The API limits URL jobs to 1 GiB and five hours.
+The API limits URL jobs to 1 GiB and five hours. The operator should configure
+expiration and incomplete-multipart-upload cleanup on this prefix only;
+Speakr does not change bucket lifecycle policies automatically.
+
+Live validation used an unchanged 9.64 MB WebM recording (598.74 seconds) in
+a private COS object: downloaded bytes matched SHA-256, anonymous access was
+denied, Tencent returned 176 timestamped segments, and deletion was verified.
+This is a functional test, not an accuracy or performance benchmark.
## Failures and retries
-Transient polling failures retry the same task. **Submission is never retried
-by the connector.** Task IDs are logged and included in timeout errors. There
-is no automatic resume across worker restarts: check the Tencent ASR console
-before manually retrying a failed/timeout recording, since a new transcription
-can create another billable task. A network timeout during submission can mean
-Tencent accepted it even though no task ID reached Speakr.
+Checkpoints are stored in `tencent-tasks/` under the Flask instance directory,
+or in `TENCENT_ASR_STATE_DIR` when set. Keep that directory on persistent storage.
+It contains task IDs, object locators and completed transcription results, but
+not credentials or signed URLs. Treat it as private application data. Every
+worker serving the same instance must share it; per-file OS locks prevent
+concurrent submission. Distributed deployments need filesystem lock semantics
+and are not validated by this connector.
+
+An identical audio file, ASR account and recognition parameters reuse the saved
+task. After a polling failure or worker restart, retrying the recording resumes
+polling rather than submitting again. Completed results are reused; results are
+saved before deleting staging audio. Failed cleanup retains its locator and is
+retried when that recording is processed again, with bucket lifecycle cleanup
+as a fallback. There is no new background polling thread: recovery occurs when
+Speakr's job queue or the user invokes transcription again.
+
+**Ambiguous submissions never retry automatically.** If the worker dies after
+starting submission but before saving the returned task ID, the checkpoint
+blocks further submission until an operator reconciles it with Tencent's
+console. Tasks older than 23 hours also stop instead of querying a potentially
+reused task ID (Tencent retains results for 24 hours). A known provider rejection
+can be retried after fixing configuration or quota. Failed remote tasks require
+operator review. Checkpoints are not automatically expired to avoid silently
+charging for the same recording again.
+
+To intentionally request fresh recognition with unchanged settings, stop work
+on that recording, confirm the old remote task is finished and clean up any
+staging locator, then back up/remove only its matching checkpoint JSON. This
+explicit operation can create another billable task. A user-facing reset action
+is not included yet.
API reference: [CreateRecTask](https://cloud.tencent.com/document/api/1093/37823).
diff --git a/src/services/storage/s3.py b/src/services/storage/s3.py
index e5f0c3d0..a60a0383 100644
--- a/src/services/storage/s3.py
+++ b/src/services/storage/s3.py
@@ -19,6 +19,11 @@ def _is_aliyun_oss_endpoint(endpoint_url: Optional[str]) -> bool:
return hostname.endswith('.aliyuncs.com')
+def _is_tencent_cos_endpoint(endpoint_url: Optional[str]) -> bool:
+ hostname = urlparse(endpoint_url or '').hostname or ''
+ return hostname.startswith('cos.') and hostname.endswith('.myqcloud.com')
+
+
def _botocore_supports_request_checksum() -> bool:
"""True if the installed botocore accepts request_checksum_calculation.
@@ -78,17 +83,19 @@ class S3StorageBackend:
'signature_version': 's3v4',
's3': {
'addressing_style': (
- 'virtual' if _is_aliyun_oss_endpoint(resolved_endpoint)
+ 'virtual' if (_is_aliyun_oss_endpoint(resolved_endpoint) or _is_tencent_cos_endpoint(resolved_endpoint))
else 'path' if self.use_path_style else 'auto'
),
},
}
- # Aliyun OSS rejects the aws-chunked transfer encoding triggered by
- # request checksums, so opt out for OSS endpoints. AWS/MinIO keep the
+ # OSS/COS need compatibility with the older S3 upload wire format;
+ # disable optional checksum trailers. AWS/MinIO keep the
# botocore default. Guarded by the version that introduced the option.
- if (_is_aliyun_oss_endpoint(resolved_endpoint)
+ if ((_is_aliyun_oss_endpoint(resolved_endpoint) or _is_tencent_cos_endpoint(resolved_endpoint))
and _botocore_supports_request_checksum()):
config_kwargs['request_checksum_calculation'] = 'when_required'
+ if _is_tencent_cos_endpoint(resolved_endpoint):
+ config_kwargs['response_checksum_validation'] = 'when_required'
client_kwargs['config'] = Config(**config_kwargs)
return boto3.client(**client_kwargs)
@@ -126,9 +133,11 @@ class S3StorageBackend:
stat = self.stat(StorageLocator(scheme='s3', raw=self.build_locator(key), bucket=self.bucket, key=key))
return StoredObject(locator=self.build_locator(key), key=key, size=stat.size, content_type=stat.content_type, etag=stat.etag)
- def upload_local_file(self, local_path: str, key: str, content_type: Optional[str] = None, metadata: Optional[dict] = None, delete_source: bool = False) -> StoredObject:
+ def upload_local_file(self, local_path: str, key: str, content_type: Optional[str] = None, metadata: Optional[dict] = None, delete_source: bool = False, acl: Optional[str] = None) -> StoredObject:
client = self._get_client()
extra = {}
+ if acl:
+ extra['ACL'] = acl
if content_type:
extra['ContentType'] = content_type
if metadata:
diff --git a/src/services/transcription/base.py b/src/services/transcription/base.py
index 287eb5d4..da06ddec 100644
--- a/src/services/transcription/base.py
+++ b/src/services/transcription/base.py
@@ -51,6 +51,8 @@ class ConnectorSpecifications:
# Codecs this connector doesn't support (removed from defaults)
# Merged with AUDIO_UNSUPPORTED_CODECS env var
unsupported_codecs: Optional[FrozenSet[str]] = None
+ # Provider accepts original audio; do not transcode or repair its container.
+ preserve_original_audio: bool = False
# Default specifications for connectors that don't define their own
diff --git a/src/services/transcription/connectors/tencent_asr.py b/src/services/transcription/connectors/tencent_asr.py
index c08e99d6..57fe19be 100644
--- a/src/services/transcription/connectors/tencent_asr.py
+++ b/src/services/transcription/connectors/tencent_asr.py
@@ -6,6 +6,7 @@ import hmac
import io
import json
import logging
+import os
import tempfile
import time
from datetime import datetime, timezone
@@ -17,6 +18,7 @@ import httpx
from ..base import (BaseTranscriptionConnector, ConnectorSpecifications,
TranscriptionCapability, TranscriptionResponse, TranscriptionSegment)
from ..exceptions import ConfigurationError, ProviderError, TranscriptionError
+from ..tencent_state import locked_checkpoint
logger = logging.getLogger(__name__)
HOST = 'asr.tencentcloudapi.com'
@@ -31,7 +33,7 @@ class TencentASRConnector(BaseTranscriptionConnector):
TranscriptionCapability.TIMESTAMPS, TranscriptionCapability.HOTWORDS}
SPECIFICATIONS = ConnectorSpecifications(
max_file_size_bytes=FILE_LIMIT, max_duration_seconds=18000,
- handles_chunking_internally=True)
+ handles_chunking_internally=True, preserve_original_audio=True)
def _validate_config(self):
self.secret_id = self.config.get('secret_id', '').strip()
@@ -50,6 +52,10 @@ class TencentASRConnector(BaseTranscriptionConnector):
raise ConfigurationError('Tencent requires 0 < poll_interval <= timeout <= 10800 seconds')
self.hotword_id = self.config.get('hotword_id', '').strip()
self.default_diarize = self.config.get('diarize', True)
+ self.state_dir = self.config.get('state_dir')
+ self.staging_prefix = (self.config.get('staging_prefix') or 'tencent-asr').strip('/')
+ if not self.staging_prefix or '..' in self.staging_prefix.split('/'):
+ raise ConfigurationError('Tencent staging prefix must be a nonempty object prefix without ..')
def _headers(self, action, body, timestamp):
date = datetime.fromtimestamp(timestamp, timezone.utc).strftime('%Y-%m-%d')
@@ -92,7 +98,8 @@ class TencentASRConnector(BaseTranscriptionConnector):
elif code.startswith(('AuthFailure', 'UnauthorizedOperation')):
hint = ' Check credentials, CAM permissions and system clock.'
raise ProviderError(f'Tencent {action}: {code}.{hint}', provider=self.PROVIDER_NAME,
- status_code=429 if code.startswith('RequestLimitExceeded') else 400)
+ status_code=(429 if code.startswith('RequestLimitExceeded')
+ else 500 if code.startswith('InternalError') else 400))
if not isinstance(data['Data'], dict):
raise ValueError('Invalid task data')
return data['Data']
@@ -115,15 +122,19 @@ class TencentASRConnector(BaseTranscriptionConnector):
params['HotwordId'] = self.hotword_id
return params
- def _stage(self, path, request):
+ def _stage(self, path, request, state=None, save=None):
from src.services.storage import get_storage_service, parse_locator
storage = get_storage_service()
if not storage.s3:
raise ConfigurationError('Tencent audio over 5 MiB requires Speakr S3 storage with a publicly reachable endpoint')
- key = 'tencent-asr/' + uuid4().hex + Path(request.filename).suffix
+ key = self.staging_prefix + '/' + uuid4().hex + Path(request.filename).suffix
locator = parse_locator(storage.s3.build_locator(key))
+ if state is not None:
+ state['staging_locator'] = storage.s3.build_locator(key)
+ state['staging_endpoint'] = storage.s3.endpoint_url
+ save() # Remember the key even if the worker dies during upload.
try:
- storage.s3.upload_local_file(path, key, content_type=request.mime_type or 'application/octet-stream')
+ storage.s3.upload_local_file(path, key, content_type=request.mime_type or 'application/octet-stream', acl='private')
url = storage.s3.presign_get_url(locator, expires_seconds=86400)
return storage.s3, locator, url
except Exception:
@@ -134,8 +145,10 @@ class TencentASRConnector(BaseTranscriptionConnector):
def _cleanup(storage, locator):
try:
storage.delete(locator, missing_ok=True)
+ return True
except Exception:
logger.warning('Tencent ASR staging cleanup failed; configure a tencent-asr/ bucket lifecycle rule')
+ return False
def _poll(self, client, task_id):
deadline = time.monotonic() + self.poll_timeout
@@ -154,63 +167,105 @@ class TencentASRConnector(BaseTranscriptionConnector):
raise
time.sleep(min(self.poll_interval, max(0, deadline - time.monotonic())))
raise TranscriptionError(f'Tencent task {task_id}: polling timed out. The remote task may still be running; '
- 'a new transcription submits a new billable task.')
+ 'retry the same recording to resume polling the saved task.')
+
+ def _state_directory(self):
+ if self.state_dir:
+ return self.state_dir
+ from flask import current_app
+ return os.path.join(current_app.instance_path, 'tencent-tasks')
+
+ def _cleanup_checkpoint(self, state, save):
+ if not state.get('staging_locator'):
+ return
+ from src.services.storage import get_storage_service, parse_locator
+ storage = get_storage_service().s3
+ if not storage or storage.endpoint_url != state.get('staging_endpoint'):
+ logger.warning('Tencent staging cleanup deferred: original storage endpoint is unavailable')
+ return
+ if self._cleanup(storage, parse_locator(state['staging_locator'])):
+ state.pop('staging_locator', None)
+ state.pop('staging_endpoint', None)
+ save()
def transcribe(self, request):
params = self._parameters(request)
- storage = locator = None
- submitted = terminal = False
try:
- try:
- request.audio_file.seek(0)
- except (AttributeError, OSError, io.UnsupportedOperation):
- pass
- data = request.audio_file.read(INLINE_LIMIT + 1)
- if not data:
+ request.audio_file.seek(0)
+ except (AttributeError, OSError, io.UnsupportedOperation):
+ pass
+ # Spool without transcoding; hash original bytes to identify safe retries.
+ with tempfile.TemporaryDirectory(prefix='speakr_tencent_') as directory:
+ path = Path(directory) / 'audio'
+ digest = hashlib.sha256()
+ size = 0
+ with path.open('wb') as output:
+ while True:
+ chunk = request.audio_file.read(1024 * 1024)
+ if not chunk:
+ break
+ size += len(chunk)
+ if size > FILE_LIMIT:
+ raise TranscriptionError('Tencent ASR: audio exceeds 1 GiB URL limit')
+ output.write(chunk)
+ digest.update(chunk)
+ if not size:
raise TranscriptionError('Tencent ASR: empty audio file')
- if len(data) <= INLINE_LIMIT:
- params.update(SourceType=1, Data=base64.b64encode(data).decode(), DataLen=len(data))
- else:
- with tempfile.TemporaryDirectory(prefix='speakr_tencent_') as directory:
- path = str(Path(directory) / 'audio')
- size = len(data)
- with open(path, 'wb') as output:
- output.write(data)
- while True:
- chunk = request.audio_file.read(1024 * 1024)
- if not chunk:
- break
- size += len(chunk)
- if size > FILE_LIMIT:
- raise TranscriptionError('Tencent ASR: audio exceeds 1 GiB URL limit')
- output.write(chunk)
- storage, locator, url = self._stage(path, request)
- params.update(SourceType=0, Url=url)
- with httpx.Client(timeout=httpx.Timeout(60, connect=15), follow_redirects=False) as client:
- # Mark before POST: a transport failure can occur after acceptance.
- submitted = True
- try:
- result = self._call(client, 'CreateRecTask', params)
- except ProviderError as exc:
- # A definite rejection cannot be downloading the staged object.
- terminal = bool(exc.status_code and 400 <= exc.status_code < 500)
- raise
- task_id = result.get('TaskId')
- if not isinstance(task_id, int) or isinstance(task_id, bool):
- raise ProviderError('Tencent submit returned no task ID', provider=self.PROVIDER_NAME)
- logger.info('Tencent ASR task submitted: %s', task_id)
- result = self._poll(client, task_id)
- terminal = True
- if result['Status'] == 3:
- raise TranscriptionError(f'Tencent task {task_id} failed; check the ASR console for details')
- return self._parse_result(result, params['EngineModelType'], request.diarize)
- except httpx.HTTPError:
- raise TranscriptionError('Tencent ASR network error. Submission was not retried: check the ASR console '
- 'before retrying to avoid duplicate charges.') from None
- finally:
- # On timeout or ambiguous POST failure, Tencent may still need the URL.
- if storage and (terminal or not submitted):
- self._cleanup(storage, locator)
+ identity = json.dumps([self.secret_id, digest.hexdigest(), params], sort_keys=True)
+ fingerprint = hashlib.sha256(identity.encode()).hexdigest()
+ with locked_checkpoint(self._state_directory(), fingerprint) as (state, save):
+ state.update(source_sha256=digest.hexdigest(), source_bytes=size)
+ save()
+ if state.get('status') in ('completed', 'failed'):
+ self._cleanup_checkpoint(state, save)
+ if state['status'] == 'failed':
+ raise TranscriptionError('Saved Tencent task failed; inspect the checkpoint before explicitly starting another billable task')
+ return self._parse_result(state['result'], params['EngineModelType'], request.diarize)
+ if state.get('status') == 'submitting':
+ raise TranscriptionError('Tencent submission outcome is uncertain. Check the ASR console and saved checkpoint; automatic resubmission is blocked to prevent duplicate charges')
+ with httpx.Client(timeout=httpx.Timeout(60, connect=15), follow_redirects=False) as client:
+ task_id = state.get('task_id')
+ if task_id:
+ if time.time() - state['submitted_at'] >= 23 * 3600:
+ raise TranscriptionError('Saved Tencent task is nearing its 24-hour expiry; automatic resubmission is blocked. Check the ASR console')
+ logger.info('Resuming Tencent ASR task: %s', task_id)
+ else:
+ # A previous interrupted upload has no ASR task yet.
+ self._cleanup_checkpoint(state, save)
+ if state.get('staging_locator'):
+ raise TranscriptionError('Previous Tencent staging cleanup is pending; check object storage before retrying')
+ state.update(status='preparing')
+ save()
+ if size <= INLINE_LIMIT:
+ params.update(SourceType=1, Data=base64.b64encode(path.read_bytes()).decode(), DataLen=size)
+ else:
+ _, _, url = self._stage(str(path), request, state, save)
+ params.update(SourceType=0, Url=url)
+ state.update(status='submitting', submitted_at=time.time())
+ save() # Crash after this point must not silently submit twice.
+ try:
+ result = self._call(client, 'CreateRecTask', params)
+ except ProviderError as exc:
+ if exc.status_code and 400 <= exc.status_code < 500:
+ state['status'] = 'rejected'
+ save()
+ self._cleanup_checkpoint(state, save)
+ raise
+ except httpx.HTTPError:
+ raise TranscriptionError('Tencent ASR network error. Submission was not retried; its outcome is uncertain. Check the ASR console') from None
+ task_id = result.get('TaskId')
+ if not isinstance(task_id, int) or isinstance(task_id, bool):
+ raise ProviderError('Tencent submit returned no task ID; automatic resubmission is blocked', provider=self.PROVIDER_NAME)
+ state.update(status='pending', task_id=task_id)
+ save()
+ logger.info('Tencent ASR task submitted: %s', task_id)
+ result = self._poll(client, task_id)
+ state.update(status='completed' if result['Status'] == 2 else 'failed', result=result)
+ save() # Keep result before deleting the only cloud audio copy.
+ self._cleanup_checkpoint(state, save)
+ if result['Status'] == 3:
+ raise TranscriptionError(f'Tencent task {task_id} failed; check the ASR console for details')
+ return self._parse_result(result, params['EngineModelType'], request.diarize)
def _parse_result(self, result, model, diarize):
segments = []
diff --git a/src/services/transcription/registry.py b/src/services/transcription/registry.py
index 8c9a1397..e0b01193 100644
--- a/src/services/transcription/registry.py
+++ b/src/services/transcription/registry.py
@@ -362,6 +362,8 @@ class ConnectorRegistry:
'timeout': os.environ.get('TENCENT_ASR_TIMEOUT', '1800'),
'poll_interval': os.environ.get('TENCENT_ASR_POLL_INTERVAL', '5'),
'hotword_id': os.environ.get('TENCENT_ASR_HOTWORD_ID', ''),
+ 'state_dir': os.environ.get('TENCENT_ASR_STATE_DIR') or None,
+ 'staging_prefix': os.environ.get('TENCENT_ASR_STAGING_PREFIX', 'tencent-asr'),
'diarize': os.environ.get('ASR_DIARIZE', 'true').lower() == 'true',
}
diff --git a/src/services/transcription/tencent_state.py b/src/services/transcription/tencent_state.py
new file mode 100644
index 00000000..353650f3
--- /dev/null
+++ b/src/services/transcription/tencent_state.py
@@ -0,0 +1,57 @@
+"""Durable Tencent task checkpoints; one OS lock per audio/options fingerprint."""
+import json
+import os
+from contextlib import contextmanager
+from pathlib import Path
+from uuid import uuid4
+
+from .exceptions import TranscriptionError
+
+
+@contextmanager
+def locked_checkpoint(directory, fingerprint):
+ root = Path(directory)
+ root.mkdir(parents=True, exist_ok=True, mode=0o700)
+ path = root / (fingerprint + '.json')
+ with (root / (fingerprint + '.lock')).open('a+b') as lock:
+ lock.seek(0)
+ if os.name == 'nt':
+ import msvcrt
+ if not lock.read(1):
+ lock.write(b'0')
+ lock.flush()
+ lock.seek(0)
+ try:
+ msvcrt.locking(lock.fileno(), msvcrt.LK_NBLCK, 1)
+ except OSError:
+ raise TranscriptionError('This Tencent recording is already being processed; wait before retrying') from None
+ else:
+ import fcntl
+ try:
+ fcntl.flock(lock, fcntl.LOCK_EX | fcntl.LOCK_NB)
+ except BlockingIOError:
+ raise TranscriptionError('This Tencent recording is already being processed; wait before retrying') from None
+ try:
+ try:
+ state = json.loads(path.read_text()) if path.exists() else {}
+ except (ValueError, OSError):
+ raise TranscriptionError('Tencent checkpoint cannot be read; refusing a potentially duplicate submission') from None
+
+ def save():
+ temporary = root / (fingerprint + '.' + uuid4().hex + '.tmp')
+ try:
+ fd = os.open(temporary, os.O_WRONLY | os.O_CREAT | os.O_EXCL, 0o600)
+ with os.fdopen(fd, 'w', encoding='utf-8') as output:
+ json.dump(state, output, ensure_ascii=False)
+ output.flush()
+ os.fsync(output.fileno())
+ os.replace(temporary, path)
+ finally:
+ temporary.unlink(missing_ok=True)
+ yield state, save
+ finally:
+ if os.name == 'nt':
+ lock.seek(0)
+ msvcrt.locking(lock.fileno(), msvcrt.LK_UNLCK, 1)
+ else:
+ fcntl.flock(lock, fcntl.LOCK_UN)
diff --git a/src/tasks/processing.py b/src/tasks/processing.py
index 3100913c..134f1c59 100644
--- a/src/tasks/processing.py
+++ b/src/tasks/processing.py
@@ -2153,7 +2153,7 @@ def transcribe_with_connector(app_context, recording_id, filepath, original_file
])
# Only retry with MP3 conversion on first attempt for format errors
- if attempt == 0 and is_format_error and not converted_filepath:
+ if attempt == 0 and is_format_error and not converted_filepath and not getattr(connector_specs, 'preserve_original_audio', False):
current_app.logger.warning(f"Transcription failed with possible format error: {e}")
current_app.logger.info(f"Attempting MP3 conversion and retry...")
diff --git a/src/utils/audio_conversion.py b/src/utils/audio_conversion.py
index 8b92616b..77202fba 100644
--- a/src/utils/audio_conversion.py
+++ b/src/utils/audio_conversion.py
@@ -151,6 +151,13 @@ def convert_if_needed(
# Get original file size
original_size = os.path.getsize(filepath)
+ if getattr(connector_specs, 'preserve_original_audio', False):
+ import mimetypes
+ return ConversionResult(
+ output_path=filepath,
+ mime_type=mimetypes.guess_type(original_filename)[0] or 'application/octet-stream',
+ was_converted=False, was_compressed=False,
+ original_size=original_size, final_size=original_size)
# Probe if codec info not provided
if codec_info is None:
diff --git a/tests/test_tencent_asr.py b/tests/test_tencent_asr.py
index 27cb61ab..f3f84317 100644
--- a/tests/test_tencent_asr.py
+++ b/tests/test_tencent_asr.py
@@ -13,9 +13,9 @@ from src.services.transcription.exceptions import ConfigurationError, ProviderEr
@pytest.fixture
-def connector():
+def connector(tmp_path):
return mod.TencentASRConnector({'secret_id': 'test-id', 'secret_key': 'test-secret',
- 'poll_interval': .001, 'timeout': .05})
+ 'poll_interval': .001, 'timeout': .5, 'state_dir': str(tmp_path)})
def request(data=b'audio', **kwargs):
@@ -113,10 +113,11 @@ def test_large_file_uses_private_staging_and_cleans_terminal(connector, monkeypa
monkeypatch.setattr(mod, 'INLINE_LIMIT', 5)
storage = Mock()
locator = object()
- def stage(path, req):
+ def stage(path, req, *args):
assert mod.Path(path).read_bytes() == b'123456'
return storage, locator, 'https://bucket/file?signed=PRIVATE'
monkeypatch.setattr(connector, '_stage', stage)
+ monkeypatch.setattr(connector, '_cleanup_checkpoint', lambda state, save: storage.delete(locator, missing_ok=True) if state.get('status') in ('completed','failed') else None)
calls = transport(monkeypatch, [{'Data': {'TaskId': 42}}, {'Data': {'Status': status, 'Result': 'text'}}])
if status == 2:
assert connector.transcribe(request(b'123456')).text == 'text'
@@ -199,6 +200,7 @@ def test_submission_rejection_removes_staging(connector, monkeypatch):
monkeypatch.setattr(mod, 'INLINE_LIMIT', 5)
storage = Mock()
monkeypatch.setattr(connector, '_stage', lambda *args: (storage, 'locator', 'https://bucket/file'))
+ monkeypatch.setattr(connector, '_cleanup_checkpoint', lambda state, save: storage.delete('locator', missing_ok=True) if state.get('status') == 'rejected' else None)
transport(monkeypatch, [{'Error': {'Code': 'AuthFailure.SecretIdNotFound'}}])
with pytest.raises(ProviderError):
connector.transcribe(request(b'123456'))
diff --git a/tests/test_tencent_recovery.py b/tests/test_tencent_recovery.py
new file mode 100644
index 00000000..19cc6153
--- /dev/null
+++ b/tests/test_tencent_recovery.py
@@ -0,0 +1,141 @@
+"""Recovery and cross-process duplication tests for paid Tencent ASR tasks."""
+import io
+import json
+import subprocess
+import sys
+from unittest.mock import Mock
+
+import httpx
+import pytest
+
+from src.services.transcription.base import TranscriptionRequest
+from src.services.transcription.connectors.tencent_asr import TencentASRConnector
+from src.services.transcription.exceptions import TranscriptionError
+from src.services.transcription.tencent_state import locked_checkpoint
+
+
+def connector(tmp_path):
+ return TencentASRConnector({'secret_id':'id','secret_key':'secret','state_dir':str(tmp_path)})
+
+
+def audio():
+ return TranscriptionRequest(audio_file=io.BytesIO(b'original bytes'),filename='meeting.webm',diarize=True)
+
+
+def test_new_worker_resumes_saved_task(tmp_path):
+ first=connector(tmp_path)
+ first._call=Mock(return_value={'TaskId':123})
+ first._poll=Mock(side_effect=TranscriptionError('network interrupted'))
+ with pytest.raises(TranscriptionError):
+ first.transcribe(audio())
+ second=connector(tmp_path)
+ second._call=Mock(side_effect=AssertionError('must not resubmit'))
+ second._poll=Mock(return_value={'Status':2,'Result':'recovered'})
+ assert second.transcribe(audio()).text=='recovered'
+ assert second._poll.call_args.args[1]==123
+ third=connector(tmp_path)
+ third._call=Mock(side_effect=AssertionError('must use saved result'))
+ third._poll=Mock(side_effect=AssertionError('must use saved result'))
+ assert third.transcribe(audio()).text=='recovered'
+
+
+def test_uncertain_post_blocks_new_worker(tmp_path):
+ first=connector(tmp_path)
+ first._call=Mock(side_effect=httpx.ReadTimeout('sensitive URL'))
+ with pytest.raises(TranscriptionError,match='uncertain'):
+ first.transcribe(audio())
+ second=connector(tmp_path)
+ second._call=Mock(side_effect=AssertionError('duplicate submission'))
+ with pytest.raises(TranscriptionError,match='automatic resubmission is blocked'):
+ second.transcribe(audio())
+
+
+def test_expired_task_never_reused_or_resubmitted(tmp_path):
+ first=connector(tmp_path)
+ first._call=Mock(return_value={'TaskId':123})
+ first._poll=Mock(side_effect=TranscriptionError('interrupted'))
+ with pytest.raises(TranscriptionError): first.transcribe(audio())
+ path=next(tmp_path.glob('*.json'))
+ state=json.loads(path.read_text())
+ state['submitted_at']=0
+ path.write_text(json.dumps(state))
+ with pytest.raises(TranscriptionError,match='expiry'):
+ connector(tmp_path).transcribe(audio())
+
+
+def test_corrupt_checkpoint_does_not_submit(tmp_path):
+ path=tmp_path/'fingerprint.json'
+ path.write_text('{')
+ with pytest.raises(TranscriptionError,match='cannot be read'):
+ with locked_checkpoint(tmp_path,'fingerprint'): pass
+
+
+def test_real_process_lock_and_crash_release(tmp_path):
+ program='''import sys,time
+from src.services.transcription.tencent_state import locked_checkpoint
+with locked_checkpoint(sys.argv[1], 'same-audio') as (state, save):
+ state.update(status='submitting')
+ save()
+ print('LOCKED',flush=True)
+ time.sleep(120)
+'''
+ child=subprocess.Popen([sys.executable,'-c',program,str(tmp_path)],stdout=subprocess.PIPE,text=True)
+ try:
+ assert child.stdout.readline().strip()=='LOCKED'
+ with pytest.raises(TranscriptionError,match='already being processed'):
+ with locked_checkpoint(tmp_path,'same-audio'): pass
+ finally:
+ child.kill()
+ child.wait(timeout=10)
+ with locked_checkpoint(tmp_path,'same-audio') as (state,save):
+ assert state['status']=='submitting'
+
+
+def test_failed_cleanup_preserves_locator_for_retry(tmp_path,monkeypatch):
+ import src.services.storage as storage_module
+ from types import SimpleNamespace
+ storage=Mock(endpoint_url='https://cos.ap-hongkong.myqcloud.com')
+ monkeypatch.setattr(storage_module,'get_storage_service',lambda:SimpleNamespace(s3=storage))
+ c=connector(tmp_path)
+ state={'status':'completed','staging_locator':'s3://test/recordings/audio.webm',
+ 'staging_endpoint':storage.endpoint_url}
+ save=Mock()
+ storage.delete.side_effect=RuntimeError('network')
+ c._cleanup_checkpoint(state,save)
+ assert 'staging_locator' in state
+ storage.delete.side_effect=None
+ c._cleanup_checkpoint(state,save)
+ assert 'staging_locator' not in state
+ save.assert_called_once()
+
+
+def test_cos_client_configuration_and_private_upload(tmp_path):
+ from unittest.mock import patch
+ from src.services.storage.s3 import S3StorageBackend
+ backend=S3StorageBackend(bucket='test-123',region='ap-hongkong',
+ endpoint_url='https://cos.ap-hongkong.myqcloud.com')
+ with patch('boto3.client') as factory:
+ backend._create_client()
+ config=factory.call_args.kwargs['config']
+ assert config.s3['addressing_style']=='virtual'
+ assert config.request_checksum_calculation=='when_required'
+ assert config.response_checksum_validation=='when_required'
+ path=tmp_path/'audio.webm'
+ path.write_bytes(b'original audio unchanged')
+ fake=Mock()
+ with patch.object(backend,'_get_client',return_value=fake), patch.object(backend,'stat'):
+ backend.upload_local_file(str(path),'recordings/audio.webm',acl='private')
+ fake.upload_file.assert_called_once_with(str(path),'test-123','recordings/audio.webm',ExtraArgs={'ACL':'private'})
+ assert path.read_bytes()==b'original audio unchanged'
+
+
+def test_webm_pipeline_preserves_original_even_when_compression_enabled(tmp_path,monkeypatch):
+ import src.utils.audio_conversion as conversion
+ p=tmp_path/'meeting.webm'
+ p.write_bytes(b'original webm bytes')
+ monkeypatch.setattr(conversion,'AUDIO_COMPRESS_UPLOADS',True)
+ monkeypatch.setattr(conversion,'get_codec_info',Mock(side_effect=AssertionError('must not inspect or convert')))
+ result=conversion.convert_if_needed(str(p),connector_specs=TencentASRConnector.SPECIFICATIONS)
+ assert result.output_path==str(p)
+ assert not result.was_compressed and not result.was_converted
+ assert p.read_bytes()==b'original webm bytes'