Skip to content

Ministack: Free, open-source local AWS emulator - 60+ services, Terraform compatible, real databases. Free forever. MIT licensed.

feat(transcribe): emulate Amazon Transcribe batch transcription jobs (#1593)

ppettitau315a13bae1cf
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 35ebec4d..46027612 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -7,6 +7,9 @@ Versioning follows [Semantic Versioning](https://semver.org/).
## [Unreleased]
+### Added
+- **Amazon Transcribe (`transcribe`)** is a new service emulator covering batch transcription jobs: `StartTranscriptionJob`, `GetTranscriptionJob`, `ListTranscriptionJobs` and `DeleteTranscriptionJob`, verified against botocore `transcribe-2017-10-26`. A job walks `QUEUED` → `IN_PROGRESS` → `COMPLETED` on a background task paced by `TRANSCRIBE_JOB_RUN_SECONDS` (default 2, `0` completes immediately), reads its media object from MiniStack's S3 store through an `s3://`, path-style or virtual-hosted HTTPS URI, and writes a transcript document back to S3 in the real Transcribe result format: `results.transcripts`, word-level `results.items`, `results.audio_segments`, and `results.segments` / `results.speaker_labels` when `ShowAlternatives` and `ShowSpeakerLabels` are set. `IdentifyLanguage` fills in `LanguageCode` and `IdentifiedLanguageScore` on completion while `IdentifyMultipleLanguages` reports `LanguageCodes` and leaves `LanguageCode` unset, as AWS does. `ContentRedaction` decides whether the `redacted-` copy replaces the transcript or sits beside it, and `Subtitles` writes WebVTT and SubRip files reported as `SubtitleFileUris`, redacted and renamed to match when redaction is on. Output lands at `OutputBucketName` / `OutputKey`, or in an auto-created `ministack-transcribe-output` bucket when neither is given, so the reported URIs are always fetchable. Media that cannot be read fails the job asynchronously with a `FailureReason` rather than erroring at `Start`, matching AWS, and a job left mid-flight by a restart fails on restore instead of stranding pollers. Reaching a terminal state publishes a `Transcribe Job State Change` event on the default bus, so an S3 → Lambda → Transcribe → EventBridge pipeline runs end to end locally. There is **no speech recognition**: the transcript is deterministic canned text keyed on a digest of the media URI, the same approach Bedrock Runtime takes for model output, which keeps integration tests reproducible. Streaming transcription, the call-analytics and medical job families, custom vocabularies and the tagging operations are not implemented. State is account- and region-scoped and persisted. Contributed by @ppettitau.
+
## [1.5.5] — 2026-09-01
### Added
diff --git a/README.md b/README.md
index c700df44..cb252482 100644
--- a/README.md
+++ b/README.md
@@ -427,6 +427,7 @@ subnet = ec2.create_subnet(
| **Bedrock Agent** | 72 operations — agents (versions, aliases, action groups, collaborators, agent knowledge bases, PrepareAgent), knowledge bases (data sources, ingestion jobs — `StartIngestionJob` reads the S3 data source and reports real statistics, `FAILED` with `failureReasons` when the source is unreadable — documents), flows (aliases, versions, ValidateFlowDefinition, PrepareFlow), prompts with versions, tagging | All 72 operations verified against botocore `bedrock-agent-2023-06-05` |
| **Bedrock Agent Runtime** | InvokeAgent, InvokeInlineAgent, GetAgentMemory, DeleteAgentMemory, Retrieve, RetrieveAndGenerate, RetrieveAndGenerateStream, Rerank, CreateSession, GetSession, UpdateSession, DeleteSession, EndSession, ListSessions, CreateInvocation, ListInvocations, PutInvocationStep, GetInvocationStep, ListInvocationSteps, InvokeFlow, StartFlowExecution, StopFlowExecution, GetFlowExecution, ListFlowExecutions, ListFlowExecutionEvents, GetExecutionFlowSnapshot, OptimizePrompt | All 31 operations verified against botocore `bedrock-agent-runtime-2023-07-26`; eventstream responses on streaming operations; `Retrieve` searches the documents Knowledge Base ingestion stored (lexical scoring — no embeddings) and returns content, `s3Location` and score; `InvokeAgent` answers a canned reply without orchestration and emits no trace events — it never fabricates an `ORCHESTRATE` trace |
| **Bedrock AgentCore** | Control plane (`bedrock-agentcore-control`): CreateAgentRuntime, GetAgentRuntime, ListAgentRuntimes, UpdateAgentRuntime, DeleteAgentRuntime, ListAgentRuntimeVersions, CreateAgentRuntimeEndpoint, GetAgentRuntimeEndpoint, ListAgentRuntimeEndpoints, UpdateAgentRuntimeEndpoint, DeleteAgentRuntimeEndpoint. Data plane (`bedrock-agentcore`): InvokeAgentRuntime | Agent-runtime lifecycle with `CREATING`/`UPDATING`→`READY` transitions, versioned ARNs (`agent/{uuid}:{version}`, `agentEndpoint/{uuid}`); request/response shapes verified against botocore `bedrock-agentcore-control` / `bedrock-agentcore`; account- and region-scoped state; `InvokeAgentRuntime` is a deterministic echo (no model inference), matching the requested emulation scope |
+| **Transcribe** | StartTranscriptionJob, GetTranscriptionJob, ListTranscriptionJobs, DeleteTranscriptionJob | Batch transcription jobs only; streaming is a separate endpoint (`transcribestreaming`). All four operations verified against botocore `transcribe-2017-10-26`. Jobs walk `QUEUED` → `IN_PROGRESS` → `COMPLETED` over `TRANSCRIBE_JOB_RUN_SECONDS` (default 2, `0` completes immediately); the media object is read from MiniStack S3 via an `s3://`, path-style or virtual-hosted HTTPS URI, and an unreadable URI fails the job with a `FailureReason` rather than erroring at `Start`, as on AWS; the transcript is written back to S3 in the real result format (`results.transcripts`, word-level `results.items`, `results.audio_segments`, plus `results.segments` and `results.speaker_labels` when the matching `Settings` are set) at `OutputBucketName`/`OutputKey`, or to an auto-created `ministack-transcribe-output` bucket. `IdentifyLanguage` and `IdentifyMultipleLanguages` resolve a language on completion, `ContentRedaction` controls whether a `redacted-` copy replaces or accompanies the transcript, and `Subtitles` writes WebVTT and SubRip files, redacted alongside the transcript when redaction is on. Reaching a terminal state publishes a `Transcribe Job State Change` event, so EventBridge rules downstream of a job fire. There is no speech recognition, so the transcript is deterministic canned text keyed on a digest of the media URI |
| **MSK** | CreateCluster, ListClusters, DescribeCluster, DeleteCluster, GetBootstrapBrokers, ListNodes, CreateConfiguration, ListConfigurations, DescribeConfiguration, ListConfigurationRevisions, DescribeConfigurationRevision, BatchAssociateScramSecret, BatchDisassociateScramSecret, ListScramSecrets, TagResource, UntagResource, ListTagsForResource | Kafka control plane; `GetBootstrapBrokers` honors `MINISTACK_MSK_BOOTSTRAP` so clients route to a real broker you bring (Redpanda, Kafka, KRaft) — the Kafka wire protocol itself is not emulated |
| **WAF v2** | CreateWebACL, GetWebACL, UpdateWebACL, DeleteWebACL, ListWebACLs, AssociateWebACL, DisassociateWebACL, GetWebACLForResource, ListResourcesForWebACL, CreateIPSet, GetIPSet, UpdateIPSet, DeleteIPSet, ListIPSets, CreateRuleGroup, GetRuleGroup, UpdateRuleGroup, DeleteRuleGroup, ListRuleGroups, TagResource, UntagResource, ListTagsForResource, CheckCapacity, DescribeManagedRuleGroup | LockToken enforced on Update/Delete; resource associations tracked |
| **Step Functions** | CreateStateMachine, DeleteStateMachine, DescribeStateMachine, UpdateStateMachine, ListStateMachines, StartExecution, StartSyncExecution, StopExecution, DescribeExecution, DescribeStateMachineForExecution, ListExecutions, GetExecutionHistory, SendTaskSuccess, SendTaskFailure, SendTaskHeartbeat, CreateActivity, DeleteActivity, DescribeActivity, ListActivities, GetActivityTask, TestState, TagResource, UntagResource, ListTagsForResource | Full ASL interpreter; Retry/Catch; waitForTaskToken; Activities (worker pattern); Pass/Task/Choice/Wait/Succeed/Fail/Map/Parallel; TestState API with mock and inspectionLevel support; SFN_MOCK_CONFIG for AWS SFN Local compatible mock testing; intrinsic functions (States.StringToJson, States.JsonToString, States.JsonMerge, States.Format); nested startExecution.sync |
@@ -850,6 +851,7 @@ end-to-end without any client config.
| `MINISTACK_MSK_BOOTSTRAP_TLS` | _(unset)_ | TLS bootstrap string (`BootstrapBrokerStringTls`) |
| `MINISTACK_MSK_BOOTSTRAP_SASL_SCRAM` | _(unset)_ | SASL/SCRAM bootstrap string (`BootstrapBrokerStringSaslScram`) |
| `MINISTACK_MSK_BOOTSTRAP_SASL_IAM` | _(unset)_ | SASL/IAM bootstrap string (`BootstrapBrokerStringSaslIam`) |
+| `TRANSCRIBE_JOB_RUN_SECONDS` | `2` | Wall-clock time a Transcribe job spends between `QUEUED` and `COMPLETED`. Set `0` to complete immediately, or raise it when a test needs to observe `IN_PROGRESS` |
### API Gateway HTTP proxy execution model
diff --git a/ministack/app.py b/ministack/app.py
index 1f89ade4..136d0a27 100644
--- a/ministack/app.py
+++ b/ministack/app.py
@@ -366,6 +366,7 @@ SERVICE_REGISTRY = {
"states": {"module": "stepfunctions", "aliases": ("step-functions", "stepfunctions")},
"sts": {"module": "sts"},
"tagging": {"module": "tagging"},
+ "transcribe": {"module": "transcribe"},
"transfer": {"module": "transfer"},
"waf": {"module": "waf_v1"},
"waf-regional": {"module": "waf_v1"},
@@ -430,6 +431,7 @@ _state_map = {
"bedrock_agent_runtime": "bedrock_agent_runtime",
"bedrock_agentcore": "bedrock_agentcore",
"msk": "msk",
+ "transcribe": "transcribe",
}
SERVICE_NAME_ALIASES = {
diff --git a/ministack/core/iam_actions.py b/ministack/core/iam_actions.py
index e1a5509a..2556d1c6 100644
--- a/ministack/core/iam_actions.py
+++ b/ministack/core/iam_actions.py
@@ -97,6 +97,7 @@ SERVICE_TO_IAM_NAMESPACE: dict[str, str] = {
"states": "states",
"sts": "sts",
"tagging": "tag",
+ "transcribe": "transcribe",
"transfer": "transfer",
"waf": "waf",
"waf-regional": "waf-regional",
diff --git a/ministack/core/router.py b/ministack/core/router.py
index d22657f4..9aee28bb 100644
--- a/ministack/core/router.py
+++ b/ministack/core/router.py
@@ -143,6 +143,10 @@ SERVICE_PATTERNS = {
"target_prefixes": ["AmazonAthena"],
"host_patterns": [r"athena\."],
},
+ "transcribe": {
+ "target_prefixes": ["Transcribe."],
+ "host_patterns": [r"transcribe\."],
+ },
"airflow": {
"host_patterns": [r"airflow\."],
"path_patterns": [
diff --git a/ministack/services/transcribe.py b/ministack/services/transcribe.py
new file mode 100644
index 00000000..c1006a9a
--- /dev/null
+++ b/ministack/services/transcribe.py
@@ -0,0 +1,973 @@
+"""
+Amazon Transcribe emulator.
+
+JSON 1.1 protocol with X-Amz-Target prefix ``Transcribe``
+(``transcribe-2017-10-26``).
+
+Implemented:
+ StartTranscriptionJob, GetTranscriptionJob, ListTranscriptionJobs,
+ DeleteTranscriptionJob.
+
+Jobs run as a background task that walks QUEUED -> IN_PROGRESS -> COMPLETED
+over ``TRANSCRIBE_JOB_RUN_SECONDS`` (0 completes immediately), reads the media
+object out of MiniStack's S3 store, and writes a transcript document back to
+S3 in the real Transcribe result format. Entering a terminal state publishes a
+``Transcribe Job State Change`` event so EventBridge rules downstream of a
+transcription job fire the way they do on AWS.
+
+The transcript document carries the structures the real service produces:
+``results.transcripts``, word-level ``results.items``, ``results.segments``
+when ``Settings.ShowAlternatives`` is set, ``results.audio_segments``, and
+``results.speaker_labels`` when ``Settings.ShowSpeakerLabels`` is set.
+
+Language identification is resolved rather than ignored. ``IdentifyLanguage``
+sets ``LanguageCode`` and ``IdentifiedLanguageScore``; ``IdentifyMultipleLanguages``
+sets ``LanguageCodes``, which is the only place the resolved code appears, as
+on AWS. Both pick deterministically from ``LanguageOptions``.
+
+``ContentRedaction`` drives the output layout the way AWS does: ``redacted``
+writes only ``redacted-<key>``, ``redacted_and_unredacted`` writes both.
+``Subtitles`` writes real WebVTT and SubRip files alongside the transcript and
+reports them as ``SubtitleFileUris``. Subtitles carry the transcript text, so a
+redacted job's subtitle files are redacted and named to match.
+
+There is no speech recognition here. The transcript text is deterministic
+canned output derived from the media URI, following the same approach as
+``bedrock_runtime._mock_reply``: identical input gives identical output, and
+different input gives distinguishable output. Speaker labels and alternatives
+are synthesised from that text, so they are well-formed and stable rather
+than meaningful.
+
+Deferred:
+ Streaming transcription (a separate endpoint), call analytics and medical
+ job families, custom vocabularies / vocabulary filters / language models,
+ and TagResource / UntagResource / ListTagsForResource. ``ModelSettings``,
+ ``JobExecutionSettings``, ``LanguageIdSettings``, ``ToxicityDetection`` and
+ ``Tags`` round-trip on the job record without changing the output.
+"""
+
+import asyncio
+import base64
+import contextvars
+import copy
+import hashlib
+import json
+import logging
+import os
+import re
+import time
+from urllib.parse import urlparse
+
+from ministack.core.persistence import load_state
+from ministack.core.responses import (
+ AccountRegionScopedDict,
+ error_response_json,
+ get_account_id,
+ get_region,
+ json_response,
+ new_uuid,
+ now_iso,
+ request_scope,
+)
+
+logger = logging.getLogger("transcribe")
+
+# How long a job spends between QUEUED and COMPLETED. Same knob shape as
+# GLUE_CRAWLER_RUN_SECONDS: tests that assert on IN_PROGRESS need a non-zero
+# value, tests that just want a result set it to 0.
+_JOB_RUN_SECONDS = float(os.environ.get("TRANSCRIBE_JOB_RUN_SECONDS", "2"))
+
+# Where transcripts land when the caller supplies no OutputBucketName. Real
+# Transcribe uses a service-managed bucket and hands back a presigned URL; the
+# local equivalent is a bucket MiniStack owns, so the URI is always fetchable.
+_SERVICE_BUCKET = "ministack-transcribe-output"
+
+_TERMINAL_STATUSES = ("COMPLETED", "FAILED")
+
+_MEDIA_UNREADABLE = (
+ "The S3 URI that you provided can't be accessed. Make sure that you have "
+ "read permission and try your request again."
+)
+
+# AWS publishes the pattern as ^[0-9a-zA-Z._-]+$ with no bound; the 1-200
+# length limit comes from the shape's min/max rather than the pattern itself.
+_JOB_NAME_RE = re.compile(r"^[0-9a-zA-Z._-]{1,200}$")
+
+_HOST_SANITY_RE = re.compile(r"^[A-Za-z0-9.\-\[\]:]{1,255}$")
+
+# Virtual-hosted-style S3 host, e.g. "my-bucket.s3.ap-southeast-2.amazonaws.com".
+# Path-style hosts ("s3-ap-southeast-2.amazonaws.com") deliberately do not
+# match: they have no bucket component before the ".s3" label.
+_S3_VIRTUAL_HOST_RE = re.compile(
+ r"^(?P<bucket>[a-z0-9][a-z0-9.\-]*?)\.s3(?:[.-][a-z0-9-]+)*\.(?:amazonaws\.com|localhost)$"
+)
+
+# The pattern AWS reports when Media.MediaFileUri is rejected.
+_MEDIA_URI_PATTERN = "(s3://|http(s*)://).+"
+
+_DEFAULT_HOST = os.environ.get("MINISTACK_HOST", "localhost")
+_DEFAULT_PORT = os.environ.get("GATEWAY_PORT", os.environ.get("EDGE_PORT", "4566"))
+
+# The caller's Host header, so TranscriptFileUri points at an address the
+# caller can actually reach. A Lambda container calling in over the compose
+# network reaches MiniStack by service name, not by localhost. Same approach
+# as sqs._request_host.
+_request_host: contextvars.ContextVar[str] = contextvars.ContextVar(
+ "_transcribe_request_host", default=""
+)
+
+_jobs = AccountRegionScopedDict() # job_name -> job record
+
+# Public members of the TranscriptionJob shape, in the order botocore models
+# them. Internal bookkeeping lives under keys prefixed with "_" and never
+# reaches the wire.
+_JOB_MEMBERS = (
+ "TranscriptionJobName",
+ "TranscriptionJobStatus",
+ "LanguageCode",
+ "MediaSampleRateHertz",
+ "MediaFormat",
+ "Media",
+ "Transcript",
+ "StartTime",
+ "CreationTime",
+ "CompletionTime",
+ "FailureReason",
+ "Settings",
+ "ModelSettings",
+ "JobExecutionSettings",
+ "ContentRedaction",
+ "IdentifyLanguage",
+ "IdentifyMultipleLanguages",
+ "LanguageOptions",
+ "IdentifiedLanguageScore",
+ "LanguageCodes",
+ "Tags",
+ "Subtitles",
+ "LanguageIdSettings",
+ "ToxicityDetection",
+)
+
+# Members echoed straight back from the request without changing behaviour.
+_ECHOED_MEMBERS = (
+ "ModelSettings",
+ "JobExecutionSettings",
+ "LanguageIdSettings",
+ "ToxicityDetection",
+ "Tags",
+ "LanguageOptions",
+ "ContentRedaction",
+)
+
+# Fallback when a language-identification job supplies no LanguageOptions to
+# choose from. AWS would infer it from the audio; there is no audio here.
+_DEFAULT_LANGUAGE = "en-US"
+
+# Fixed score for an identified language. Real Transcribe reports its
+# confidence; a mock has none, so it reports certainty rather than inventing a
+# number that varies.
+_IDENTIFIED_LANGUAGE_SCORE = 1.0
+
+_SUMMARY_MEMBERS = (
+ "TranscriptionJobName",
+ "CreationTime",
+ "StartTime",
+ "CompletionTime",
+ "LanguageCode",
+ "TranscriptionJobStatus",
+ "FailureReason",
+ "OutputLocationType",
+)
+
+
+# ---------------------------------------------------------------------------
+# State
+# ---------------------------------------------------------------------------
+
+
+def reset():
+ _jobs.clear()
+
+
+def get_state():
+ return copy.deepcopy({"jobs": _jobs})
+
+
+def restore_state(data):
+ if not data:
+ return
+ _jobs.clear()
+ jobs = data.get("jobs")
+ # Not `or {}`: AccountRegionScopedDict.__bool__ is scope-relative, and
+ # restore runs in the default account/region. A snapshot holding jobs only
+ # in other scopes would read as falsy and every one of them would be lost.
+ if jobs is not None:
+ _jobs.update(jobs)
+ _fail_orphaned_jobs()
+
+
+def load_persisted_state(data):
+ restore_state(data)
+
+
+def _fail_orphaned_jobs():
+ """A job that was mid-flight when the process stopped has no worker any
+ more. Leaving it QUEUED or IN_PROGRESS strands every caller polling
+ GetTranscriptionJob forever, so it is failed the way AWS fails a job it
+ cannot finish."""
+ for job in _jobs.all_values():
+ if job.get("TranscriptionJobStatus") not in _TERMINAL_STATUSES:
+ job["TranscriptionJobStatus"] = "FAILED"
+ job["FailureReason"] = "Internal Failure. The job did not survive a MiniStack restart."
+ job["CompletionTime"] = time.time()
+
+try:
+ _restored = load_state("transcribe")
+ if _restored:
+ restore_state(_restored)
+except Exception:
+ logger.exception("Failed to restore persisted Transcribe state; continuing with fresh store")
+
+
+# ---------------------------------------------------------------------------
+# Helpers
+# ---------------------------------------------------------------------------
+
+
+def _bad_request(message):
+ return error_response_json("BadRequestException", message, 400)
+
+
+def _public_job(job):
+ """Project a stored job onto the TranscriptionJob wire shape, dropping
+ members AWS omits when they have no value."""
+ return {k: job[k] for k in _JOB_MEMBERS if job.get(k) is not None}
+
+
+def _public_summary(job):
+ return {k: job[k] for k in _SUMMARY_MEMBERS if job.get(k) is not None}
+
+
+def _external_netloc():
+ host = _request_host.get()
+ if host and _HOST_SANITY_RE.match(host):
+ return host
+ return f"{_DEFAULT_HOST}:{_DEFAULT_PORT}"
+
+
+def _parse_s3_uri(uri):
+ """Return (bucket, key) for an S3 location, or (None, None).
+
+ AWS models Media.MediaFileUri as ``(s3://|http(s*)://).+``, so an HTTPS S3
+ URL is as valid as an ``s3://`` one and SDK callers do emit both. Both
+ path-style (``s3.<region>.amazonaws.com/<bucket>/<key>``) and
+ virtual-hosted style (``<bucket>.s3.<region>.amazonaws.com/<key>``) are
+ accepted; an unrecognised host falls back to path-style, which is how a
+ MiniStack gateway URL resolves."""
+ if not isinstance(uri, str):
+ return None, None
+
+ parsed = urlparse(uri)
+ scheme = (parsed.scheme or "").lower()
+
+ if scheme == "s3":
+ bucket = parsed.netloc
+ key = parsed.path.lstrip("/")
+ elif scheme in ("http", "https"):
+ path = parsed.path.lstrip("/")
+ match = _S3_VIRTUAL_HOST_RE.match(parsed.hostname or "")
+ if match:
+ bucket, key = match.group("bucket"), path
+ else:
+ bucket, _, key = path.partition("/")
+ else:
+ return None, None
+
+ if not bucket or not key:
+ return None, None
+ return bucket, key
+
+
+_SEGMENT_WORDS = 6
+_WORD_SECONDS = 0.5
+_MAX_SPEAKERS = 10
+
+
+def _mock_transcript(media_uri, redacted=False):
+ """Deterministic canned transcript. The digest of the media URI keeps
+ output for different media distinguishable while staying stable across
+ runs, which is the whole point of using fixtures instead of real ASR.
+
+ Long enough to split into several segments, so speaker labels, alternatives
+ and multi-language spans have something to distribute across rather than
+ collapsing onto one segment.
+
+ The redacted variant masks the digest, so it is the identifying part of
+ the text that disappears, the way real redaction removes PII."""
+ if redacted:
+ subject = "[PII]"
+ else:
+ digest = hashlib.sha256(media_uri.encode("utf-8")).hexdigest()[:8]
+ subject = f"media#{digest}"
+ return (
+ f"[ministack mock transcript] deterministic placeholder speech for {subject} "
+ "generated locally by ministack with no speech recognition performed on the audio"
+ )
+
+
+def _fmt_time(seconds):
+ return f"{seconds:.1f}"
+
+
+def _word_items(text):
+ """Word-level items in Transcribe's result format, with synthetic timings
+ so consumers that walk `results.items` have something well-formed to walk.
+ Any `language_code` tag is stamped on by the caller, which knows which
+ segment each item belongs to."""
+ items = []
+ for index, word in enumerate(text.split()):
+ start = index * _WORD_SECONDS
+ item = {
+ "type": "pronunciation",
+ "start_time": _fmt_time(start),
+ "end_time": _fmt_time(start + _WORD_SECONDS),
+ "alternatives": [{"confidence": "1.0", "content": word}],
+ }
+ items.append(item)
+ return items
+
+
+def _alternatives(transcript, items, max_alternatives):
+ """The primary reading plus synthetic runners-up. Real Transcribe returns
+ competing hypotheses when ShowAlternatives is set; there is no recogniser
+ here to disagree with itself, so the extras are labelled as what they are
+ rather than dressed up as genuine alternatives."""
+ alternatives = [{"transcript": transcript, "items": items}]
+ for rank in range(1, max(1, min(max_alternatives, 10))):
+ alternatives.append({"transcript": f"{transcript} [alternative {rank}]", "items": items})
+ return alternatives
+
+
+def _transcript_document(job_name, text, settings, language_codes):
+ """Build the transcript JSON in the shape real Transcribe writes to S3.
+
+ `segments` appears only under ShowAlternatives and `speaker_labels` only
+ under ShowSpeakerLabels, matching the service: a consumer that never asked
+ for diarisation should not find it there."""
+ settings = settings or {}
+ show_speakers = bool(settings.get("ShowSpeakerLabels"))
+ show_alternatives = bool(settings.get("ShowAlternatives"))
+ max_alternatives = settings.get("MaxAlternatives") or 2
+ max_speakers = max(1, min(settings.get("MaxSpeakerLabels") or 2, _MAX_SPEAKERS))
+
+ # Only tag items with a language when more than one was resolved; a
+ # single-language job carries the code on the job record instead. The tag
+ # is applied per segment below, so an item always agrees with the audio
+ # segment that owns it.
+ multi_language = len(language_codes) > 1
+ items = _word_items(text)
+
+ words = text.split()
+ groups = [
+ list(range(start, min(start + _SEGMENT_WORDS, len(items))))
+ for start in range(0, len(items), _SEGMENT_WORDS)
+ ]
+
+ segments = []
+ audio_segments = []
+ speaker_segments = []
+
+ for position, indices in enumerate(groups):
+ segment_text = " ".join(words[indices[0] : indices[-1] + 1])
+ segment_items = [items[i] for i in indices]
+ start_time = segment_items[0]["start_time"]
+ end_time = segment_items[-1]["end_time"]
+ speaker_label = f"spk_{position % max_speakers}"
+ segment_language = language_codes[position % len(language_codes)] if language_codes else None
+ if multi_language:
+ for item in segment_items:
+ item["language_code"] = segment_language
+
+ segments.append(
+ {
+ "start_time": start_time,
+ "end_time": end_time,
+ "alternatives": _alternatives(segment_text, segment_items, max_alternatives),
+ }
+ )
+
+ audio_segment = {
+ "id": position,
+ "transcript": segment_text,
+ "start_time": start_time,
+ "end_time": end_time,
+ "items": indices,
+ }
+ if language_codes:
+ audio_segment["language_code"] = segment_language
+ if show_speakers:
+ audio_segment["speaker_label"] = speaker_label
+ audio_segments.append(audio_segment)
+
+ if show_speakers:
+ speaker_segments.append(
+ {
+ "start_time": start_time,
+ "end_time": end_time,
+ "speaker_label": speaker_label,
+ "items": [
+ {
+ "start_time": item["start_time"],
+ "end_time": item["end_time"],
+ "speaker_label": speaker_label,
+ }
+ for item in segment_items
+ ],
+ }
+ )
+
+ results = {
+ "transcripts": [{"transcript": text}],
+ "items": items,
+ "audio_segments": audio_segments,
+ }
+ if show_alternatives:
+ results["segments"] = segments
+ if show_speakers:
+ results["speaker_labels"] = {
+ "speakers": min(len(groups), max_speakers),
+ "segments": speaker_segments,
+ }
+
+ return {
+ "jobName": job_name,
+ "accountId": get_account_id(),
+ "status": "COMPLETED",
+ "results": results,
+ }
+
+
+# ---------------------------------------------------------------------------
+# Subtitles
+# ---------------------------------------------------------------------------
+
+
+def _subtitle_cue_time(seconds, separator):
+ hours, remainder = divmod(float(seconds), 3600)
+ minutes, secs = divmod(remainder, 60)
+ whole, fraction = divmod(secs, 1)
+ return f"{int(hours):02d}:{int(minutes):02d}:{int(whole):02d}{separator}{int(fraction * 1000):03d}"
+
+
+def _render_subtitles(document, fmt, start_index):
+ """Render the transcript's audio segments as SubRip or WebVTT."""
+ srt = fmt == "srt"
+ separator = "," if srt else "."
+ blocks = []
+ for offset, segment in enumerate(document["results"]["audio_segments"]):
+ start = _subtitle_cue_time(segment["start_time"], separator)
+ end = _subtitle_cue_time(segment["end_time"], separator)
+ blocks.append(
+ f"{offset + start_index}\n{start} --> {end}\n{segment['transcript']}\n"
+ )
+ body = "\n".join(blocks)
+ return body if srt else f"WEBVTT\n\n{body}"
+
+
+def _resolve_languages(job):
+ """The language codes a language-identification job would have detected.
+
+ There is no audio to analyse, so the choice comes from LanguageOptions in
+ request order, which keeps it deterministic and lets a test pin the
+ outcome by listing the language it wants first."""
+ if job.get("LanguageCode"):
+ return [job["LanguageCode"]]
+ # Duplicates would otherwise report the same language twice, each with the
+ # full duration, so the spans sum to more audio than the job contains.
+ options = []
+ for code in job.get("LanguageOptions") or []:
+ if isinstance(code, str) and code not in options:
+ options.append(code)
+ if job.get("IdentifyMultipleLanguages"):
+ return options or [_DEFAULT_LANGUAGE]
+ return [options[0]] if options else [_DEFAULT_LANGUAGE]
+
+
+def _apply_languages(job, codes, document):
+ """Record the resolved language on the job the way AWS does.
+
+ IdentifyLanguage fills LanguageCode and IdentifiedLanguageScore.
+ IdentifyMultipleLanguages fills LanguageCodes and leaves LanguageCode
+ unset, so a caller that wants the source language has to check both. This
+ only happens once the job completes; AWS cannot know the language at
+ submission time, so neither does this."""
+ if job.get("LanguageCode"):
+ return
+ if job.get("IdentifyMultipleLanguages"):
+ durations = {}
+ for segment in document["results"]["audio_segments"]:
+ code = segment.get("language_code")
+ span = float(segment["end_time"]) - float(segment["start_time"])
+ durations[code] = round(durations.get(code, 0.0) + span, 1)
+ job["LanguageCodes"] = [
+ {"LanguageCode": code, "DurationInSeconds": durations.get(code, 0.0)}
+ for code in codes
+ ]
+ else:
+ job["LanguageCode"] = codes[0]
+ job["IdentifiedLanguageScore"] = _IDENTIFIED_LANGUAGE_SCORE
+
+
+def _redacted_key(key):
+ """Redacted output sits beside the transcript with `redacted-` prefixed to
+ the file name, not to the whole key, so it stays in the same prefix."""
+ prefix, separator, name = key.rpartition("/")
+ return f"{prefix}{separator}redacted-{name}"
+
+
+def _subtitle_key(key, fmt):
+ base = key[: -len(".json")] if key.endswith(".json") else key
+ return f"{base}.{fmt}"
+
+
+def _emit_state_change(job_name, status):
+ """Publish `Transcribe Job State Change`, the event AWS emits when a job
+ reaches a terminal state. Pipelines that chain off a completed
+ transcription depend on it, so a failure to deliver is logged rather than
+ swallowed silently, and never propagates into the job itself."""
+ try:
+ from ministack.services import eventbridge as _eb
+
+ _eb._dispatch_event(
+ {
+ "EventId": new_uuid(),
+ "Source": "aws.transcribe",
+ "DetailType": "Transcribe Job State Change",
+ "Detail": json.dumps(
+ {
+ "TranscriptionJobName": job_name,
+ "TranscriptionJobStatus": status,
+ }
+ ),
+ "EventBusName": "default",
+ "Time": now_iso(),
+ "Resources": [],
+ "Account": get_account_id(),
+ "Region": get_region(),
+ }
+ )
+ except Exception:
+ logger.exception("Transcribe: failed to publish state change for job %s", job_name)
+
+
+def _sort_key(job):
+ """Creation time with the job name breaking ties, so the ordering is total
+ and stable across calls. Callers reverse it to get newest first."""
+ return (job.get("CreationTime") or 0.0, job.get("TranscriptionJobName") or "")
+
+
+def _encode_token(job):
+ """Tokens carry the sort position of the last item returned, not an index.
+ An offset would skip an item whenever a job was deleted between pages and
+ repeat one whenever a job was created."""
+ created, name = _sort_key(job)
+ raw = json.dumps({"ct": created, "n": name}).encode("utf-8")
+ return base64.urlsafe_b64encode(raw).decode("ascii")
+
+
+def _decode_token(token):
+ try:
+ raw = json.loads(base64.urlsafe_b64decode(token.encode("ascii")).decode("utf-8"))
+ return (float(raw["ct"]), str(raw["n"]))
+ except Exception:
+ return None
+
+
+# ---------------------------------------------------------------------------
+# Job execution
+# ---------------------------------------------------------------------------
+
+
+def _live_job(job_name, run_id):
+ """The job this worker was started for, or None if it has been deleted or
+ replaced. DeleteTranscriptionJob does not cancel a running worker, so a
+ job re-created under the same name would otherwise be driven by both its
+ own worker and the orphaned one: completing early, writing the transcript
+ twice and emitting the state change event twice."""
+ job = _jobs.get(job_name)
+ if job is None or job.get("_run_id") != run_id:
+ return None
+ return job
+
+
+async def _run_job(job_name, run_id, account_id, region, netloc):
+ """Walk a job to a terminal state. Runs as a background task, so it pins
+ the account and region it was started for rather than inheriting whatever
+ request happens to be in flight."""
+ with request_scope(account_id, region):
+ try:
+ await _run_job_inner(job_name, run_id, netloc)
+ except Exception:
+ logger.exception("Transcribe: job %s crashed", job_name)
+ job = _live_job(job_name, run_id)
+ if job is not None and job.get("TranscriptionJobStatus") not in _TERMINAL_STATUSES:
+ job["TranscriptionJobStatus"] = "FAILED"
+ job["FailureReason"] = "Internal Failure. Please try your request again."
+ job["CompletionTime"] = time.time()
+ _emit_state_change(job_name, "FAILED")
+
+
+async def _run_job_inner(job_name, run_id, netloc):
+ half = _JOB_RUN_SECONDS / 2 if _JOB_RUN_SECONDS > 0 else 0
+
+ if half:
+ await asyncio.sleep(half)
+
+ job = _live_job(job_name, run_id)
+ if job is None:
+ return
+
+ job["TranscriptionJobStatus"] = "IN_PROGRESS"
+ job["StartTime"] = time.time()
+
+ if half:
+ await asyncio.sleep(half)
+
+ job = _live_job(job_name, run_id)
+ if job is None:
+ return
+
+ def _fail(reason):
+ if _live_job(job_name, run_id) is None:
+ return
+ job["TranscriptionJobStatus"] = "FAILED"
+ job["FailureReason"] = reason
+ job["CompletionTime"] = time.time()
+ logger.info("Transcribe: job %s failed: %s", job_name, reason)
+ _emit_state_change(job_name, "FAILED")
+
+ from ministack.services import s3 as s3_svc
+
+ media_uri = job["Media"]["MediaFileUri"]
+ media_bucket, media_key = _parse_s3_uri(media_uri)
+ if s3_svc._get_object_data(media_bucket, media_key) is None:
+ _fail(_MEDIA_UNREADABLE)
+ return
+
+ text = _mock_transcript(media_uri)
+ codes = _resolve_languages(job)
+ document = _transcript_document(job_name, text, job.get("Settings"), codes)
+
+ bucket = job["_output_bucket"]
+ key = job["_output_key"]
+
+ if job["OutputLocationType"] == "SERVICE_BUCKET" and s3_svc._ensure_bucket(bucket) is None:
+ s3_svc._create_bucket(bucket, b"")
+
+ def _write(target_key, payload, content_type):
+ body = payload.encode("utf-8")
+ resp = s3_svc._put_object(
+ bucket,
+ target_key,
+ body,
+ {"content-type": content_type, "content-length": str(len(body))},
+ )
+ if isinstance(resp, tuple) and resp[0] >= 300:
+ return False
+ return True
+
+ # RedactionOutput decides the layout: "redacted" replaces the transcript
+ # with the masked one, "redacted_and_unredacted" writes both. Transcript
+ # then carries whichever URIs actually exist.
+ redaction = job.get("ContentRedaction") or {}
+ redaction_output = str(redaction.get("RedactionOutput") or "").lower()
+ transcript_uris = {}
+ writes = []
+ subtitle_document = document
+ subtitle_base_key = key
+
+ if redaction:
+ redacted_document = _transcript_document(
+ job_name, _mock_transcript(media_uri, redacted=True), job.get("Settings"), codes
+ )
+ redacted_key = _redacted_key(key)
+ writes.append((redacted_key, json.dumps(redacted_document), "application/json"))
+ transcript_uris["RedactedTranscriptFileUri"] = f"http://{netloc}/{bucket}/{redacted_key}"
+ # Subtitles carry the transcript text, so a redacted job's subtitle
+ # files have to be redacted too, and named for what they contain.
+ # Otherwise the one layout whose point is that the raw text never
+ # reaches the bucket writes the raw text to the bucket.
+ subtitle_document = redacted_document
+ subtitle_base_key = redacted_key
+ if redaction_output == "redacted_and_unredacted":
+ writes.append((key, json.dumps(document), "application/json"))
+ transcript_uris["TranscriptFileUri"] = f"http://{netloc}/{bucket}/{key}"
+ else:
+ writes.append((key, json.dumps(document), "application/json"))
+ transcript_uris["TranscriptFileUri"] = f"http://{netloc}/{bucket}/{key}"
+
+ subtitles = job.get("_subtitles_request") or {}
+ subtitle_formats = [f for f in (subtitles.get("Formats") or []) if f in ("vtt", "srt")]
+ subtitle_start = subtitles.get("OutputStartIndex")
+ subtitle_start = 0 if subtitle_start is None else int(subtitle_start)
+ subtitle_uris = []
+ for fmt in subtitle_formats:
+ subtitle_key = _subtitle_key(subtitle_base_key, fmt)
+ writes.append(
+ (subtitle_key, _render_subtitles(subtitle_document, fmt, subtitle_start), "text/plain")
+ )
+ subtitle_uris.append(f"http://{netloc}/{bucket}/{subtitle_key}")
+
+ for target_key, payload, content_type in writes:
+ if not _write(target_key, payload, content_type):
+ _fail(
+ f"The output bucket that you provided can't be written to: "
+ f"s3://{bucket}/{target_key}. "
+ "Make sure that the bucket exists and try your request again."
+ )
+ return
+
+ _apply_languages(job, codes, document)
+ if subtitle_formats:
+ job["Subtitles"] = {
+ "Formats": subtitle_formats,
+ "SubtitleFileUris": subtitle_uris,
+ "OutputStartIndex": subtitle_start,
+ }
+ job["Transcript"] = transcript_uris
+ job["TranscriptionJobStatus"] = "COMPLETED"
+ job["CompletionTime"] = time.time()
+ logger.info(
+ "Transcribe: job %s completed, %d object(s) written to s3://%s",
+ job_name,
+ len(writes),
+ bucket,
+ )
+ _emit_state_change(job_name, "COMPLETED")
+
+
+# ---------------------------------------------------------------------------
+# Operations
+# ---------------------------------------------------------------------------
+
+
+def _start_transcription_job(data):
+ name = data.get("TranscriptionJobName")
+ if not isinstance(name, str) or not name.strip():
+ return _bad_request(
+ "1 validation error detected: Value null at 'transcriptionJobName' "
+ "failed to satisfy constraint: Member must not be null"
+ )
+ if not _JOB_NAME_RE.match(name):
+ return _bad_request(
+ f"1 validation error detected: Value '{name}' at 'transcriptionJobName' failed to "
+ "satisfy constraint: Member must satisfy regular expression pattern: ^[0-9a-zA-Z._-]+$"
+ )
+
+ media = data.get("Media")
+ if not isinstance(media, dict) or not media.get("MediaFileUri"):
+ return _bad_request(
+ "1 validation error detected: Value null at 'media' failed to satisfy "
+ "constraint: Member must not be null"
+ )
+
+ media_uri = media["MediaFileUri"]
+ if _parse_s3_uri(media_uri) == (None, None):
+ return _bad_request(
+ f"1 validation error detected: Value '{media_uri}' at 'media.mediaFileUri' failed "
+ "to satisfy constraint: Member must satisfy regular expression pattern: "
+ f"{_MEDIA_URI_PATTERN}"
+ )
+
+ language_code = data.get("LanguageCode")
+ identify = data.get("IdentifyLanguage") or data.get("IdentifyMultipleLanguages")
+ if not language_code and not identify:
+ return _bad_request(
+ "You must include either LanguageCode or IdentifyLanguage or "
+ "IdentifyMultipleLanguages in your request. Please try your request again."
+ )
+
+ if name in _jobs:
+ return error_response_json(
+ "ConflictException",
+ "The requested job name already exists. Use a different job name.",
+ 400,
+ )
+
+ output_bucket = data.get("OutputBucketName")
+ if output_bucket:
+ output_key = data.get("OutputKey") or f"{name}.json"
+ # AWS treats a key ending in "/" as a prefix and appends the job name.
+ if output_key.endswith("/"):
+ output_key = f"{output_key}{name}.json"
+ location_type = "CUSTOMER_BUCKET"
+ else:
+ output_bucket = _SERVICE_BUCKET
+ # Region-qualified: S3 buckets are account-scoped but jobs are
+ # region-scoped, so same-named jobs in two regions are two jobs and
+ # must not resolve to one transcript object.
+ output_key = f"{get_account_id()}/{get_region()}/{name}.json"
+ location_type = "SERVICE_BUCKET"
+
+ run_id = new_uuid()
+ job = {
+ "TranscriptionJobName": name,
+ "TranscriptionJobStatus": "QUEUED",
+ "LanguageCode": language_code,
+ "MediaSampleRateHertz": data.get("MediaSampleRateHertz"),
+ "MediaFormat": data.get("MediaFormat"),
+ "Media": copy.deepcopy(media),
+ "Transcript": None,
+ "StartTime": None,
+ "CreationTime": time.time(),
+ "CompletionTime": None,
+ "FailureReason": None,
+ "Settings": copy.deepcopy(data.get("Settings")) if data.get("Settings") else None,
+ # The identification flags are echoed as sent; the codes they resolve
+ # to are filled in by the worker, because AWS cannot know the language
+ # until the job has run.
+ "IdentifyLanguage": data.get("IdentifyLanguage"),
+ "IdentifyMultipleLanguages": data.get("IdentifyMultipleLanguages"),
+ "IdentifiedLanguageScore": None,
+ "LanguageCodes": None,
+ "Subtitles": None,
+ "OutputLocationType": location_type,
+ "_run_id": run_id,
+ "_output_bucket": output_bucket,
+ "_output_key": output_key,
+ }
+ for member in _ECHOED_MEMBERS:
+ value = data.get(member)
+ job[member] = copy.deepcopy(value) if value is not None else None
+ # Subtitles is requested with Formats/OutputStartIndex but reported as
+ # Formats/SubtitleFileUris/OutputStartIndex, so it is held aside until the
+ # worker has written the files and can report where they landed.
+ job["_subtitles_request"] = copy.deepcopy(data.get("Subtitles")) or None
+ _jobs[name] = job
+
+ asyncio.create_task(
+ _run_job(name, run_id, get_account_id(), get_region(), _external_netloc())
+ )
+
+ return json_response({"TranscriptionJob": _public_job(job)})
+
+
+def _get_transcription_job(data):
+ name = data.get("TranscriptionJobName")
+ if not isinstance(name, str) or not name.strip():
+ return _bad_request(
+ "1 validation error detected: Value null at 'transcriptionJobName' "
+ "failed to satisfy constraint: Member must not be null"
+ )
+
+ job = _jobs.get(name)
+ if job is None:
+ return error_response_json(
+ "NotFoundException",
+ "The requested job couldn't be found. Check the job name and try your request again.",
+ 400,
+ )
+
+ return json_response({"TranscriptionJob": _public_job(job)})
+
+
+def _list_transcription_jobs(data):
+ status = data.get("Status")
+ if status is not None and status not in ("QUEUED", "IN_PROGRESS", "FAILED", "COMPLETED"):
+ return _bad_request(
+ f"1 validation error detected: Value '{status}' at 'status' failed to satisfy "
+ "constraint: Member must satisfy enum value set: "
+ "[COMPLETED, IN_PROGRESS, QUEUED, FAILED]"
+ )
+
+ contains = data.get("JobNameContains")
+
+ max_results = data.get("MaxResults")
+ if max_results is None:
+ max_results = 100
+ else:
+ try:
+ max_results = int(max_results)
+ except (TypeError, ValueError):
+ return _bad_request("MaxResults must be an integer")
+ if max_results < 1 or max_results > 100:
+ return _bad_request(
+ f"1 validation error detected: Value '{max_results}' at 'maxResults' failed to "
+ "satisfy constraint: Member must have value less than or equal to 100"
+ )
+
+ cursor = None
+ token = data.get("NextToken")
+ if token:
+ cursor = _decode_token(token)
+ if cursor is None:
+ return _bad_request("The NextToken that you provided is invalid.")
+
+ jobs = list(_jobs.values())
+ if status:
+ jobs = [j for j in jobs if j.get("TranscriptionJobStatus") == status]
+ if contains:
+ # AWS documents this search as case insensitive.
+ needle = str(contains).lower()
+ jobs = [j for j in jobs if needle in j["TranscriptionJobName"].lower()]
+
+ # Most recently created first, matching the console and the API.
+ jobs.sort(key=_sort_key, reverse=True)
+ if cursor is not None:
+ jobs = [j for j in jobs if _sort_key(j) < cursor]
+
+ page = jobs[:max_results]
+ result = {"TranscriptionJobSummaries": [_public_summary(j) for j in page]}
+ if status:
+ result["Status"] = status
+ if len(jobs) > max_results:
+ result["NextToken"] = _encode_token(page[-1])
+ return json_response(result)
+
+
+def _delete_transcription_job(data):
+ name = data.get("TranscriptionJobName")
+ if not isinstance(name, str) or not name.strip():
+ return _bad_request(
+ "1 validation error detected: Value null at 'transcriptionJobName' "
+ "failed to satisfy constraint: Member must not be null"
+ )
+
+ # botocore models no NotFoundException on this operation: deleting a job
+ # that isn't there succeeds.
+ _jobs.pop(name, None)
+ return json_response({})
+
+
+_DISPATCH = {
+ "StartTranscriptionJob": _start_transcription_job,
+ "GetTranscriptionJob": _get_transcription_job,
+ "ListTranscriptionJobs": _list_transcription_jobs,
+ "DeleteTranscriptionJob": _delete_transcription_job,
+}
+
+
+async def handle_request(method, path, headers, body, query_params):
+ _request_host.set(headers.get("host", "") or headers.get("Host", ""))
+
+ target = headers.get("X-Amz-Target") or headers.get("x-amz-target") or ""
+ action = target.split(".", 1)[1] if "." in target else target
+ if not action:
+ return error_response_json("InvalidAction", "missing X-Amz-Target", 400)
+
+ body_text = body.decode("utf-8") if isinstance(body, bytes) else (body or "")
+ try:
+ payload = json.loads(body_text) if body_text else {}
+ except json.JSONDecodeError:
+ return error_response_json("SerializationException", "invalid JSON body", 400)
+
+ fn = _DISPATCH.get(action)
+ if fn is None:
+ return error_response_json(
+ "InvalidAction",
+ f"Operation '{action}' not implemented",
+ 400,
+ )
+
+ logger.debug("Transcribe: %s %s", action, body_text)
+ return fn(payload)
diff --git a/tests/conftest.py b/tests/conftest.py
index 5675fe02..acaecf82 100644
--- a/tests/conftest.py
+++ b/tests/conftest.py
@@ -728,6 +728,11 @@ def s3tables():
return make_client("s3tables")
[email protected](scope="session")
+def transcribe():
+ return make_client("transcribe")
+
+
class FakeDockerContainer:
"""Container double for tests that observe lifecycle without a daemon."""
diff --git a/tests/test_persistence.py b/tests/test_persistence.py
index 5dde53ba..728047aa 100644
--- a/tests/test_persistence.py
+++ b/tests/test_persistence.py
@@ -645,6 +645,97 @@ def test_backup_round_trip():
_round_trip("backup", "backup", populate, observe)
+def test_transcribe_round_trip():
+ import time as _time
+
+ def populate(mod):
+ mod._jobs["job-test"] = {
+ "TranscriptionJobName": "job-test",
+ "TranscriptionJobStatus": "COMPLETED",
+ "LanguageCode": "en-US",
+ "Media": {"MediaFileUri": "s3://media/call.mp3"},
+ "Transcript": {
+ "TranscriptFileUri": "http://localhost:4566/out/job-test.json"
+ },
+ "CreationTime": _time.time(),
+ "StartTime": _time.time(),
+ "CompletionTime": _time.time(),
+ "OutputLocationType": "SERVICE_BUCKET",
+ "_run_id": "run-1",
+ "_output_bucket": "out",
+ "_output_key": "job-test.json",
+ }
+
+ def observe(mod):
+ job = mod._jobs.get("job-test")
+ assert job is not None
+ assert job["TranscriptionJobStatus"] == "COMPLETED"
+ # A completed job's transcript pointer and media reference are what
+ # callers read after a warm boot; losing either leaves GetTranscriptionJob
+ # reporting COMPLETED with nothing to fetch.
+ assert job["Transcript"]["TranscriptFileUri"].endswith("job-test.json")
+ assert job["Media"]["MediaFileUri"] == "s3://media/call.mp3"
+ assert job["_output_bucket"] == "out"
+
+ _round_trip("transcribe", "transcribe", populate, observe)
+
+
+def test_transcribe_jobs_round_trip_outside_boot_region():
+ """Transcribe jobs are region-scoped. `AccountRegionScopedDict.__bool__`
+ is scope-relative, so a snapshot holding jobs only in regions other than
+ the one restore runs in must not be treated as empty."""
+ import time as _time
+
+ from ministack.core.responses import request_scope
+
+ def populate(mod):
+ with request_scope("000000000000", "eu-west-1"):
+ mod._jobs["eu-job"] = {
+ "TranscriptionJobName": "eu-job",
+ "TranscriptionJobStatus": "COMPLETED",
+ "CreationTime": _time.time(),
+ "CompletionTime": _time.time(),
+ "Media": {"MediaFileUri": "s3://b/a.mp3"},
+ "OutputLocationType": "SERVICE_BUCKET",
+ "_output_bucket": "b",
+ "_output_key": "k.json",
+ }
+
+ def observe(mod):
+ with request_scope("000000000000", "eu-west-1"):
+ job = mod._jobs.get("eu-job")
+ assert job is not None, "a job outside the restoring region was dropped"
+ assert job["TranscriptionJobStatus"] == "COMPLETED"
+
+ _round_trip("transcribe", "transcribe", populate, observe)
+
+
+def test_transcribe_round_trip_fails_a_job_left_mid_flight():
+ """A restart leaves no worker behind, so a job restored as QUEUED or
+ IN_PROGRESS would strand every caller polling GetTranscriptionJob."""
+ import time as _time
+
+ def populate(mod):
+ mod._jobs["stuck-job"] = {
+ "TranscriptionJobName": "stuck-job",
+ "TranscriptionJobStatus": "IN_PROGRESS",
+ "CreationTime": _time.time(),
+ "Media": {"MediaFileUri": "s3://b/a.mp3"},
+ "OutputLocationType": "SERVICE_BUCKET",
+ "_output_bucket": "b",
+ "_output_key": "k.json",
+ }
+
+ def observe(mod):
+ job = mod._jobs.get("stuck-job")
+ assert job is not None
+ assert job["TranscriptionJobStatus"] == "FAILED"
+ assert job["CompletionTime"] is not None
+ assert "restart" in job["FailureReason"]
+
+ _round_trip("transcribe", "transcribe", populate, observe)
+
+
def test_cloudformation_round_trip():
"""CloudFormation stack metadata (stacks, events, exports, change sets)
survives a PERSIST_STATE stop/restore cycle — otherwise ListStacks /
diff --git a/tests/test_transcribe.py b/tests/test_transcribe.py
new file mode 100644
index 00000000..b93a2892
--- /dev/null
+++ b/tests/test_transcribe.py
@@ -0,0 +1,979 @@
+import json
+import os
+import time
+import uuid
+
+import boto3
+import pytest
+from botocore.exceptions import ClientError
+
+_ENDPOINT = os.environ.get("MINISTACK_ENDPOINT", "http://localhost:4566").rstrip("/")
+
+
+def _unique(prefix):
+ return f"{prefix}-{uuid.uuid4().hex[:8]}"
+
+
+def _wait_for_status(transcribe, job_name, statuses=("COMPLETED", "FAILED"), timeout=30):
+ deadline = time.time() + timeout
+ job = None
+ while time.time() < deadline:
+ job = transcribe.get_transcription_job(TranscriptionJobName=job_name)["TranscriptionJob"]
+ if job["TranscriptionJobStatus"] in statuses:
+ return job
+ time.sleep(0.2)
+ raise AssertionError(
+ f"job {job_name} never reached {statuses}; last status "
+ f"{job['TranscriptionJobStatus'] if job else 'unknown'}"
+ )
+
+
+def _split_transcript_uri(uri):
+ """`http://host:port/bucket/key` -> (bucket, key)."""
+ _, _, rest = uri.partition("://")
+ _, _, path = rest.partition("/")
+ bucket, _, key = path.partition("/")
+ return bucket, key
+
+
+def media(s3):
+ bucket = _unique("transcribe-media")
+ key = "audio/call.mp3"
+ s3.create_bucket(Bucket=bucket)
+ s3.put_object(Bucket=bucket, Key=key, Body=b"not really audio")
+ return bucket, key, f"s3://{bucket}/{key}"
+
+
+def test_transcribe_job_completes_and_writes_a_transcript(transcribe, s3, media):
+ """The end-to-end contract the issue asks for: a job reaches COMPLETED on its
+ own, and TranscriptFileUri resolves to a real object in the real Transcribe
+ result shape, so a consumer can fetch and parse it without special-casing."""
+ _, _, media_uri = media
+ job_name = _unique("job")
+
+ started = transcribe.start_transcription_job(
+ TranscriptionJobName=job_name,
+ LanguageCode="en-US",
+ MediaFormat="mp3",
+ Media={"MediaFileUri": media_uri},
+ )["TranscriptionJob"]
+
+ assert started["TranscriptionJobName"] == job_name
+ assert started["TranscriptionJobStatus"] == "QUEUED"
+ assert started["LanguageCode"] == "en-US"
+ assert started["Media"] == {"MediaFileUri": media_uri}
+ # AWS omits Transcript until the job completes.
+ assert "Transcript" not in started
+
+ job = _wait_for_status(transcribe, job_name)
+ assert job["TranscriptionJobStatus"] == "COMPLETED"
+ assert job["CompletionTime"] >= job["StartTime"] >= job["CreationTime"]
+
+ bucket, key = _split_transcript_uri(job["Transcript"]["TranscriptFileUri"])
+ document = json.loads(s3.get_object(Bucket=bucket, Key=key)["Body"].read())
+
+ assert document["jobName"] == job_name
+ assert document["status"] == "COMPLETED"
+ text = document["results"]["transcripts"][0]["transcript"]
+ assert text
+ assert len(document["results"]["items"]) == len(text.split())
+ assert document["results"]["items"][0]["alternatives"][0]["content"] == text.split()[0]
+
+
+def test_transcribe_transcript_is_deterministic_per_media(transcribe, s3, media):
+ """There is no ASR here, so the value of the mock rests entirely on being
+ reproducible. The same media must transcribe identically on every run or
+ integration tests asserting on transcript content become flaky."""
+ _, _, media_uri = media
+ texts = []
+
+ for _ in range(2):
+ job_name = _unique("job")
+ transcribe.start_transcription_job(
+ TranscriptionJobName=job_name,
+ LanguageCode="en-US",
+ Media={"MediaFileUri": media_uri},
+ )
+ job = _wait_for_status(transcribe, job_name)
+ bucket, key = _split_transcript_uri(job["Transcript"]["TranscriptFileUri"])
+ document = json.loads(s3.get_object(Bucket=bucket, Key=key)["Body"].read())
+ texts.append(document["results"]["transcripts"][0]["transcript"])
+
+ assert texts[0] == texts[1]
+
+
+def test_transcribe_different_media_gives_different_transcripts(transcribe, s3, media):
+ """The flip side of determinism: a constant string would satisfy the previous
+ test while making it impossible to tell two jobs apart."""
+ bucket, _, first_uri = media
+ s3.put_object(Bucket=bucket, Key="audio/other.mp3", Body=b"also not audio")
+ second_uri = f"s3://{bucket}/audio/other.mp3"
+
+ texts = []
+ for media_uri in (first_uri, second_uri):
+ job_name = _unique("job")
+ transcribe.start_transcription_job(
+ TranscriptionJobName=job_name,
+ LanguageCode="en-US",
+ Media={"MediaFileUri": media_uri},
+ )
+ job = _wait_for_status(transcribe, job_name)
+ out_bucket, key = _split_transcript_uri(job["Transcript"]["TranscriptFileUri"])
+ document = json.loads(s3.get_object(Bucket=out_bucket, Key=key)["Body"].read())
+ texts.append(document["results"]["transcripts"][0]["transcript"])
+
+ assert texts[0] != texts[1]
+
+
+def test_transcribe_writes_to_the_caller_supplied_output_bucket(transcribe, s3, media):
+ """OutputBucketName/OutputKey must be honoured verbatim. A CDK stack that
+ points Transcribe at its own bucket and then reads a fixed key downstream
+ breaks if the transcript lands anywhere else."""
+ _, _, media_uri = media
+ out_bucket = _unique("transcribe-out")
+ s3.create_bucket(Bucket=out_bucket)
+ job_name = _unique("job")
+
+ transcribe.start_transcription_job(
+ TranscriptionJobName=job_name,
+ LanguageCode="en-US",
+ Media={"MediaFileUri": media_uri},
+ OutputBucketName=out_bucket,
+ OutputKey="results/transcript.json",
+ )
+
+ job = _wait_for_status(transcribe, job_name)
+ assert job["TranscriptionJobStatus"] == "COMPLETED"
+ assert job["Transcript"]["TranscriptFileUri"].endswith(
+ f"/{out_bucket}/results/transcript.json"
+ )
+ # The object is really there.
+ s3.head_object(Bucket=out_bucket, Key="results/transcript.json")
+
+ summary = next(
+ s
+ for s in transcribe.list_transcription_jobs(JobNameContains=job_name)[
+ "TranscriptionJobSummaries"
+ ]
+ if s["TranscriptionJobName"] == job_name
+ )
+ assert summary["OutputLocationType"] == "CUSTOMER_BUCKET"
+
+
+def test_transcribe_output_key_ending_in_slash_is_a_prefix(transcribe, s3, media):
+ """AWS treats a trailing slash as a prefix and appends <jobName>.json rather
+ than writing an object whose key ends in a slash."""
+ _, _, media_uri = media
+ out_bucket = _unique("transcribe-out")
+ s3.create_bucket(Bucket=out_bucket)
+ job_name = _unique("job")
+
+ transcribe.start_transcription_job(
+ TranscriptionJobName=job_name,
+ LanguageCode="en-US",
+ Media={"MediaFileUri": media_uri},
+ OutputBucketName=out_bucket,
+ OutputKey="nested/prefix/",
+ )
+
+ job = _wait_for_status(transcribe, job_name)
+ assert job["Transcript"]["TranscriptFileUri"].endswith(
+ f"/{out_bucket}/nested/prefix/{job_name}.json"
+ )
+
+
+def test_transcribe_defaults_to_a_service_managed_bucket(transcribe, media):
+ """With no OutputBucketName, real Transcribe stores the transcript in a bucket
+ it owns and hands back a presigned URL. MiniStack provisions its own bucket
+ instead, so the URI is fetchable locally rather than being a dead link."""
+ _, _, media_uri = media
+ job_name = _unique("job")
+
+ transcribe.start_transcription_job(
+ TranscriptionJobName=job_name,
+ LanguageCode="en-US",
+ Media={"MediaFileUri": media_uri},
+ )
+
+ job = _wait_for_status(transcribe, job_name)
+ bucket, _ = _split_transcript_uri(job["Transcript"]["TranscriptFileUri"])
+ assert bucket == "ministack-transcribe-output"
+
+ summary = next(
+ s
+ for s in transcribe.list_transcription_jobs(JobNameContains=job_name)[
+ "TranscriptionJobSummaries"
+ ]
+ if s["TranscriptionJobName"] == job_name
+ )
+ assert summary["OutputLocationType"] == "SERVICE_BUCKET"
+
+
+def test_transcribe_unreadable_media_fails_the_job(transcribe, s3):
+ """AWS accepts the request and fails the job asynchronously with a
+ FailureReason rather than rejecting Start, because the media is not read
+ until the job runs. A poller must be able to observe FAILED."""
+ bucket = _unique("transcribe-media")
+ s3.create_bucket(Bucket=bucket)
+ job_name = _unique("job")
+
+ # Start succeeds; AWS only discovers the object is unreadable once the job runs.
+ transcribe.start_transcription_job(
+ TranscriptionJobName=job_name,
+ LanguageCode="en-US",
+ Media={"MediaFileUri": f"s3://{bucket}/missing.mp3"},
+ )
+
+ job = _wait_for_status(transcribe, job_name)
+ assert job["TranscriptionJobStatus"] == "FAILED"
+ assert "can't be accessed" in job["FailureReason"]
+ assert "Transcript" not in job
+
+
+def test_transcribe_duplicate_job_name_conflicts(transcribe, media):
+ _, _, media_uri = media
+ job_name = _unique("job")
+ kwargs = dict(
+ TranscriptionJobName=job_name,
+ LanguageCode="en-US",
+ Media={"MediaFileUri": media_uri},
+ )
+ transcribe.start_transcription_job(**kwargs)
+
+ with pytest.raises(ClientError) as exc:
+ transcribe.start_transcription_job(**kwargs)
+ assert exc.value.response["Error"]["Code"] == "ConflictException"
+
+
+def test_transcribe_get_unknown_job_is_not_found(transcribe):
+ """NotFoundException is modelled on GetTranscriptionJob but not on Delete, so
+ the two operations deliberately differ on a missing job."""
+ with pytest.raises(ClientError) as exc:
+ transcribe.get_transcription_job(TranscriptionJobName=_unique("nope"))
+ assert exc.value.response["Error"]["Code"] == "NotFoundException"
+
+
+def test_transcribe_start_requires_a_language_directive(transcribe, media):
+ """LanguageCode is not a required member, but AWS still rejects a request that
+ supplies neither it nor an IdentifyLanguage flag."""
+ _, _, media_uri = media
+ with pytest.raises(ClientError) as exc:
+ transcribe.start_transcription_job(
+ TranscriptionJobName=_unique("job"),
+ Media={"MediaFileUri": media_uri},
+ )
+ assert exc.value.response["Error"]["Code"] == "BadRequestException"
+
+
+def test_transcribe_identify_language_without_options_falls_back_to_a_default(transcribe, media):
+ """`LanguageOptions` is optional, so a caller can ask for identification and
+ name no candidates at all. The job still has to resolve to something, or a
+ consumer reading LanguageCode off the completed job gets nothing."""
+ _, _, media_uri = media
+ job_name = _unique("job")
+ started = transcribe.start_transcription_job(
+ TranscriptionJobName=job_name,
+ IdentifyLanguage=True,
+ Media={"MediaFileUri": media_uri},
+ )["TranscriptionJob"]
+ assert started["TranscriptionJobStatus"] == "QUEUED"
+ assert "LanguageCode" not in started
+
+ job = _wait_for_status(transcribe, job_name)
+ assert job["LanguageCode"] == "en-US"
+
+
+def test_transcribe_start_rejects_an_unusable_media_uri(transcribe):
+ """AWS models MediaFileUri as `(s3://|http(s*)://).+`, so the scheme check
+ is not what makes a URI bad. An unsupported scheme and an HTTP URL with no
+ key to address are both unusable and both rejected."""
+ for uri in ("ftp://example.com/bucket/audio.mp3", "https://example.com/audio.mp3"):
+ with pytest.raises(ClientError) as exc:
+ transcribe.start_transcription_job(
+ TranscriptionJobName=_unique("job"),
+ LanguageCode="en-US",
+ Media={"MediaFileUri": uri},
+ )
+ assert exc.value.response["Error"]["Code"] == "BadRequestException", uri
+
+
[email protected]("style", ["path", "path-dash", "virtual"])
+def test_transcribe_accepts_https_s3_media_uris(transcribe, s3, media, style):
+ """The AWS SDKs hand Transcribe an HTTPS S3 URL as readily as an `s3://`
+ one, and AWS accepts both. Rejecting the HTTPS form fails the job at
+ submission for any caller that builds its URI that way."""
+ bucket, key, _ = media
+ uri = {
+ "path": f"https://s3.us-east-1.amazonaws.com/{bucket}/{key}",
+ "path-dash": f"https://s3-us-east-1.amazonaws.com/{bucket}/{key}",
+ "virtual": f"https://{bucket}.s3.us-east-1.amazonaws.com/{key}",
+ }[style]
+
+ job_name = _unique("job")
+ transcribe.start_transcription_job(
+ TranscriptionJobName=job_name,
+ LanguageCode="en-US",
+ Media={"MediaFileUri": uri},
+ )
+ job = _wait_for_status(transcribe, job_name)
+ # The media must actually have been located and read, not merely accepted.
+ assert job["TranscriptionJobStatus"] == "COMPLETED", job.get("FailureReason")
+
+
+def test_transcribe_identify_language_resolves_a_language_code(transcribe, media):
+ """`IdentifyLanguage` leaves the caller with no language at submission
+ time; AWS fills LanguageCode in once the job runs, alongside the
+ confidence score. A consumer reading the completed job must find it."""
+ _, _, media_uri = media
+ job_name = _unique("job")
+
+ started = transcribe.start_transcription_job(
+ TranscriptionJobName=job_name,
+ IdentifyLanguage=True,
+ LanguageOptions=["fr-FR", "en-US"],
+ Media={"MediaFileUri": media_uri},
+ )["TranscriptionJob"]
+ assert "LanguageCode" not in started
+
+ job = _wait_for_status(transcribe, job_name)
+ assert job["IdentifyLanguage"] is True
+ assert job["LanguageCode"] == "fr-FR"
+ assert job["IdentifiedLanguageScore"] == pytest.approx(1.0)
+
+
+def test_transcribe_identify_multiple_languages_reports_language_codes(transcribe, media):
+ """A multi-language job reports `LanguageCodes` and deliberately leaves
+ `LanguageCode` unset, exactly as AWS does, so a consumer resolving the
+ source language has to fall back to the list."""
+ _, _, media_uri = media
+ job_name = _unique("job")
+
+ transcribe.start_transcription_job(
+ TranscriptionJobName=job_name,
+ IdentifyMultipleLanguages=True,
+ LanguageOptions=["en-AU", "fr-FR"],
+ Media={"MediaFileUri": media_uri},
+ )
+ job = _wait_for_status(transcribe, job_name)
+
+ assert "LanguageCode" not in job
+ codes = job["LanguageCodes"]
+ assert [entry["LanguageCode"] for entry in codes] == ["en-AU", "fr-FR"]
+ # Every reported language must own some of the audio, otherwise the span
+ # breakdown is decorative.
+ assert all(entry["DurationInSeconds"] > 0 for entry in codes)
+
+
+def test_transcribe_transcript_carries_segments_and_speaker_labels(transcribe, s3, media):
+ """`ShowSpeakerLabels` and `ShowAlternatives` change what the transcript
+ document contains, not just what the job echoes. Consumers read
+ `results.segments`, `results.audio_segments` and `results.speaker_labels`
+ to render a transcript, so those have to be present and index-aligned."""
+ _, _, media_uri = media
+ job_name = _unique("job")
+
+ transcribe.start_transcription_job(
+ TranscriptionJobName=job_name,
+ LanguageCode="en-US",
+ Media={"MediaFileUri": media_uri},
+ Settings={
+ "ShowSpeakerLabels": True,
+ "MaxSpeakerLabels": 10,
+ "ShowAlternatives": True,
+ "MaxAlternatives": 4,
+ },
+ )
+ job = _wait_for_status(transcribe, job_name)
+ bucket, key = _split_transcript_uri(job["Transcript"]["TranscriptFileUri"])
+ results = json.loads(s3.get_object(Bucket=bucket, Key=key)["Body"].read())["results"]
+
+ segments = results["segments"]
+ audio_segments = results["audio_segments"]
+ speaker_segments = results["speaker_labels"]["segments"]
+ assert len(segments) == len(audio_segments) == len(speaker_segments) > 1
+
+ for segment, audio, speaker in zip(segments, audio_segments, speaker_segments):
+ assert segment["start_time"] == audio["start_time"] == speaker["start_time"]
+ assert len(segment["alternatives"]) == 4
+ assert audio["speaker_label"] == speaker["speaker_label"]
+ assert audio["speaker_label"].startswith("spk_")
+
+ # The joined text must still be the transcript, not a reshuffled version.
+ assert " ".join(a["transcript"] for a in audio_segments) == (
+ results["transcripts"][0]["transcript"]
+ )
+
+
+def test_transcribe_omits_diarisation_when_not_requested(transcribe, s3, media):
+ """A caller that never asked for speaker labels or alternatives should not
+ find them in the output, or it cannot tell the settings had any effect."""
+ _, _, media_uri = media
+ job_name = _unique("job")
+
+ transcribe.start_transcription_job(
+ TranscriptionJobName=job_name,
+ LanguageCode="en-US",
+ Media={"MediaFileUri": media_uri},
+ )
+ job = _wait_for_status(transcribe, job_name)
+ bucket, key = _split_transcript_uri(job["Transcript"]["TranscriptFileUri"])
+ results = json.loads(s3.get_object(Bucket=bucket, Key=key)["Body"].read())["results"]
+
+ assert "speaker_labels" not in results
+ assert "segments" not in results
+ assert results["audio_segments"]
+ assert results["items"]
+
+
+def test_transcribe_redaction_writes_only_the_redacted_transcript(transcribe, s3, media):
+ """`RedactionOutput=redacted` replaces the transcript rather than adding to
+ it: AWS writes `redacted-<name>` and no unredacted copy, and reports the
+ location as RedactedTranscriptFileUri."""
+ bucket, _, media_uri = media
+ job_name = _unique("job")
+ output_key = "out/job.json"
+
+ transcribe.start_transcription_job(
+ TranscriptionJobName=job_name,
+ LanguageCode="en-US",
+ Media={"MediaFileUri": media_uri},
+ OutputBucketName=bucket,
+ OutputKey=output_key,
+ ContentRedaction={
+ "RedactionType": "PII",
+ "RedactionOutput": "redacted",
+ "PiiEntityTypes": ["ALL"],
+ },
+ )
+ job = _wait_for_status(transcribe, job_name)
+
+ assert job["ContentRedaction"]["RedactionOutput"] == "redacted"
+ assert "TranscriptFileUri" not in job["Transcript"]
+ _, redacted_key = _split_transcript_uri(job["Transcript"]["RedactedTranscriptFileUri"])
+ # The prefix moves onto the file name, not the whole key.
+ assert redacted_key == "out/redacted-job.json"
+
+ document = json.loads(s3.get_object(Bucket=bucket, Key=redacted_key)["Body"].read())
+ assert "[PII]" in document["results"]["transcripts"][0]["transcript"]
+
+ with pytest.raises(ClientError):
+ s3.get_object(Bucket=bucket, Key=output_key)
+
+
+def test_transcribe_redaction_can_keep_both_transcripts(transcribe, s3, media):
+ """`redacted_and_unredacted` is the variant that keeps the original, and
+ both URIs have to be reported for a caller to reach either."""
+ bucket, _, media_uri = media
+ job_name = _unique("job")
+ output_key = "both/job.json"
+
+ transcribe.start_transcription_job(
+ TranscriptionJobName=job_name,
+ LanguageCode="en-US",
+ Media={"MediaFileUri": media_uri},
+ OutputBucketName=bucket,
+ OutputKey=output_key,
+ ContentRedaction={
+ "RedactionType": "PII",
+ "RedactionOutput": "redacted_and_unredacted",
+ "PiiEntityTypes": ["ALL"],
+ },
+ )
+ job = _wait_for_status(transcribe, job_name)
+
+ _, plain_key = _split_transcript_uri(job["Transcript"]["TranscriptFileUri"])
+ _, redacted_key = _split_transcript_uri(job["Transcript"]["RedactedTranscriptFileUri"])
+ assert plain_key == output_key
+ assert redacted_key == "both/redacted-job.json"
+
+ plain = json.loads(s3.get_object(Bucket=bucket, Key=plain_key)["Body"].read())
+ redacted = json.loads(s3.get_object(Bucket=bucket, Key=redacted_key)["Body"].read())
+ plain_text = plain["results"]["transcripts"][0]["transcript"]
+ redacted_text = redacted["results"]["transcripts"][0]["transcript"]
+ assert "[PII]" not in plain_text
+ assert "[PII]" in redacted_text
+
+
+def test_transcribe_writes_requested_subtitle_files(transcribe, s3, media):
+ """Subtitles are reported as SubtitleFileUris, so the files have to exist
+ and be well-formed for each requested format."""
+ bucket, _, media_uri = media
+ job_name = _unique("job")
+
+ transcribe.start_transcription_job(
+ TranscriptionJobName=job_name,
+ LanguageCode="en-US",
+ Media={"MediaFileUri": media_uri},
+ OutputBucketName=bucket,
+ OutputKey="subs/job.json",
+ Subtitles={"Formats": ["vtt", "srt"], "OutputStartIndex": 1},
+ )
+ job = _wait_for_status(transcribe, job_name)
+
+ subtitles = job["Subtitles"]
+ assert subtitles["Formats"] == ["vtt", "srt"]
+ assert subtitles["OutputStartIndex"] == 1
+ assert len(subtitles["SubtitleFileUris"]) == 2
+
+ bodies = {}
+ for uri in subtitles["SubtitleFileUris"]:
+ _, sub_key = _split_transcript_uri(uri)
+ bodies[sub_key] = s3.get_object(Bucket=bucket, Key=sub_key)["Body"].read().decode()
+
+ assert bodies["subs/job.vtt"].startswith("WEBVTT")
+ # WebVTT separates the cue timestamp with a dot, SubRip with a comma.
+ assert " --> " in bodies["subs/job.vtt"]
+ assert "00:00:00.000 --> " in bodies["subs/job.vtt"]
+ assert bodies["subs/job.srt"].startswith("1\n00:00:00,000 --> ")
+
+
+def test_transcribe_redaction_redacts_the_subtitle_files_too(transcribe, s3, media):
+ """Subtitles carry the transcript text. If they are rendered from the
+ unredacted document, `RedactionOutput=redacted` writes the raw text to the
+ bucket in a `.vtt` while claiming it never left, which defeats the whole
+ point of that layout."""
+ bucket, _, media_uri = media
+ job_name = _unique("job")
+
+ transcribe.start_transcription_job(
+ TranscriptionJobName=job_name,
+ LanguageCode="en-US",
+ Media={"MediaFileUri": media_uri},
+ OutputBucketName=bucket,
+ OutputKey="redsubs/job.json",
+ ContentRedaction={
+ "RedactionType": "PII",
+ "RedactionOutput": "redacted",
+ "PiiEntityTypes": ["ALL"],
+ },
+ Subtitles={"Formats": ["vtt"]},
+ )
+ job = _wait_for_status(transcribe, job_name)
+
+ _, subtitle_key = _split_transcript_uri(job["Subtitles"]["SubtitleFileUris"][0])
+ # The name has to match the content, otherwise `redsubs/job.vtt` sits next
+ # to `redsubs/redacted-job.json` looking like the unredacted counterpart.
+ assert subtitle_key == "redsubs/redacted-job.vtt"
+
+ body = s3.get_object(Bucket=bucket, Key=subtitle_key)["Body"].read().decode()
+ assert "[PII]" in body
+
+ # The digest token is what redaction masks; finding it means the raw
+ # transcript reached the bucket.
+ _, redacted_key = _split_transcript_uri(job["Transcript"]["RedactedTranscriptFileUri"])
+ document = json.loads(s3.get_object(Bucket=bucket, Key=redacted_key)["Body"].read())
+ assert "media#" not in body, document["results"]["transcripts"][0]["transcript"]
+
+ with pytest.raises(ClientError):
+ s3.get_object(Bucket=bucket, Key="redsubs/job.vtt")
+
+
+def test_transcribe_multi_language_items_agree_with_their_segment(transcribe, s3, media):
+ """Every word item belongs to exactly one audio segment, so its
+ `language_code` has to be that segment's. Tagging all items with the first
+ resolved language gives a document that contradicts itself, and a consumer
+ grouping items by language gets the wrong grouping."""
+ _, _, media_uri = media
+ job_name = _unique("job")
+
+ transcribe.start_transcription_job(
+ TranscriptionJobName=job_name,
+ IdentifyMultipleLanguages=True,
+ LanguageOptions=["en-AU", "fr-FR"],
+ Media={"MediaFileUri": media_uri},
+ )
+ job = _wait_for_status(transcribe, job_name)
+ bucket, key = _split_transcript_uri(job["Transcript"]["TranscriptFileUri"])
+ results = json.loads(s3.get_object(Bucket=bucket, Key=key)["Body"].read())["results"]
+
+ items = results["items"]
+ audio_segments = results["audio_segments"]
+ # Both languages must actually appear, or the check is vacuous.
+ assert len({s["language_code"] for s in audio_segments}) == 2
+
+ for segment in audio_segments:
+ for index in segment["items"]:
+ assert items[index]["language_code"] == segment["language_code"], (
+ f"item {index} disagrees with audio segment {segment['id']}"
+ )
+
+
+def test_transcribe_single_language_items_carry_no_language_tag(transcribe, s3, media):
+ """A single-language job carries the code on the job record, so repeating
+ it on every item is noise the real service does not emit."""
+ _, _, media_uri = media
+ job_name = _unique("job")
+
+ transcribe.start_transcription_job(
+ TranscriptionJobName=job_name,
+ LanguageCode="en-US",
+ Media={"MediaFileUri": media_uri},
+ )
+ job = _wait_for_status(transcribe, job_name)
+ bucket, key = _split_transcript_uri(job["Transcript"]["TranscriptFileUri"])
+ results = json.loads(s3.get_object(Bucket=bucket, Key=key)["Body"].read())["results"]
+
+ assert all("language_code" not in item for item in results["items"])
+
+
+def test_transcribe_identify_multiple_languages_reports_each_language_once(transcribe, media):
+ """Repeated LanguageOptions must not become repeated LanguageCodes entries,
+ each claiming the full audio. The reported spans would then sum to more
+ audio than the job contains."""
+ _, _, media_uri = media
+ job_name = _unique("job")
+
+ transcribe.start_transcription_job(
+ TranscriptionJobName=job_name,
+ IdentifyMultipleLanguages=True,
+ LanguageOptions=["en-AU", "en-AU"],
+ Media={"MediaFileUri": media_uri},
+ )
+ job = _wait_for_status(transcribe, job_name)
+
+ codes = job["LanguageCodes"]
+ assert [entry["LanguageCode"] for entry in codes] == ["en-AU"]
+
+
+ "language_params",
+ [
+ {"IdentifyMultipleLanguages": True, "LanguageOptions": ["en-AU", "fr-FR"]},
+ {"IdentifyMultipleLanguages": True, "LanguageOptions": ["en-AU"]},
+ {"IdentifyMultipleLanguages": True},
+ ],
+ ids=["two-options", "one-option", "no-options"],
+)
+def test_transcribe_identify_multiple_languages_always_reports_the_array(
+ transcribe, media, language_params
+):
+ """A multi-language job reports `LanguageCodes` however many options it was
+ given, including one or none. Collapsing to a scalar `LanguageCode` when
+ only one language resolves is a fidelity gap: AWS populates the array
+ whenever the flag is set."""
+ _, _, media_uri = media
+ job_name = _unique("job")
+
+ transcribe.start_transcription_job(
+ TranscriptionJobName=job_name,
+ Media={"MediaFileUri": media_uri},
+ **language_params,
+ )
+ job = _wait_for_status(transcribe, job_name)
+
+ assert job["IdentifyMultipleLanguages"] is True
+ assert "LanguageCode" not in job
+ assert job["LanguageCodes"]
+
+
+def test_transcribe_start_rejects_an_invalid_job_name(transcribe, media):
+ _, _, media_uri = media
+ with pytest.raises(ClientError) as exc:
+ transcribe.start_transcription_job(
+ TranscriptionJobName="not a valid name",
+ LanguageCode="en-US",
+ Media={"MediaFileUri": media_uri},
+ )
+ assert exc.value.response["Error"]["Code"] == "BadRequestException"
+
+
+def test_transcribe_list_filters_by_status_and_name(transcribe, media):
+ _, _, media_uri = media
+ # A hex marker can come out all digits, for which `.upper()` is the
+ # identity and the case-insensitivity assertion below proves nothing.
+ marker = f"zz{uuid.uuid4().hex[:8]}"
+ job_name = f"filter-{marker}"
+
+ transcribe.start_transcription_job(
+ TranscriptionJobName=job_name,
+ LanguageCode="en-US",
+ Media={"MediaFileUri": media_uri},
+ )
+ _wait_for_status(transcribe, job_name)
+
+ listed = transcribe.list_transcription_jobs(JobNameContains=marker)
+ assert [s["TranscriptionJobName"] for s in listed["TranscriptionJobSummaries"]] == [job_name]
+
+ # The search is not case sensitive on AWS.
+ assert transcribe.list_transcription_jobs(JobNameContains=marker.upper())[
+ "TranscriptionJobSummaries"
+ ]
+
+ completed = transcribe.list_transcription_jobs(
+ Status="COMPLETED", JobNameContains=marker
+ )
+ assert completed["Status"] == "COMPLETED"
+ assert len(completed["TranscriptionJobSummaries"]) == 1
+
+ assert (
+ transcribe.list_transcription_jobs(Status="FAILED", JobNameContains=marker)[
+ "TranscriptionJobSummaries"
+ ]
+ == []
+ )
+
+
+def test_transcribe_list_paginates(transcribe, media):
+ """MaxResults must cap the page and hand back a NextToken that resumes exactly
+ where the previous page stopped, with no gaps or repeats."""
+ _, _, media_uri = media
+ marker = uuid.uuid4().hex[:8]
+ names = {f"page-{marker}-{i}" for i in range(3)}
+ for name in names:
+ transcribe.start_transcription_job(
+ TranscriptionJobName=name,
+ LanguageCode="en-US",
+ Media={"MediaFileUri": media_uri},
+ )
+
+ first = transcribe.list_transcription_jobs(JobNameContains=marker, MaxResults=2)
+ assert len(first["TranscriptionJobSummaries"]) == 2
+ assert first["NextToken"]
+
+ second = transcribe.list_transcription_jobs(
+ JobNameContains=marker, MaxResults=2, NextToken=first["NextToken"]
+ )
+ assert len(second["TranscriptionJobSummaries"]) == 1
+ assert "NextToken" not in second
+
+ seen = {
+ s["TranscriptionJobName"]
+ for s in first["TranscriptionJobSummaries"] + second["TranscriptionJobSummaries"]
+ }
+ assert seen == names
+
+
+def test_transcribe_delete_removes_the_job_and_is_idempotent(transcribe, media):
+ """botocore models no NotFoundException on DeleteTranscriptionJob, so deleting
+ a job that is not there succeeds rather than erroring."""
+ _, _, media_uri = media
+ job_name = _unique("job")
+ transcribe.start_transcription_job(
+ TranscriptionJobName=job_name,
+ LanguageCode="en-US",
+ Media={"MediaFileUri": media_uri},
+ )
+ _wait_for_status(transcribe, job_name)
+
+ transcribe.delete_transcription_job(TranscriptionJobName=job_name)
+
+ with pytest.raises(ClientError) as exc:
+ transcribe.get_transcription_job(TranscriptionJobName=job_name)
+ assert exc.value.response["Error"]["Code"] == "NotFoundException"
+
+ # AWS models no NotFoundException on delete, so a second call succeeds.
+ transcribe.delete_transcription_job(TranscriptionJobName=job_name)
+
+
+def test_transcribe_list_pagination_survives_a_deletion_between_pages(transcribe, media):
+ _, _, media_uri = media
+ marker = uuid.uuid4().hex[:8]
+ names = [f"del-{marker}-{i}" for i in range(4)]
+ for name in names:
+ transcribe.start_transcription_job(
+ TranscriptionJobName=name,
+ LanguageCode="en-US",
+ Media={"MediaFileUri": media_uri},
+ )
+
+ first = transcribe.list_transcription_jobs(JobNameContains=marker, MaxResults=2)
+ returned = [s["TranscriptionJobName"] for s in first["TranscriptionJobSummaries"]]
+
+ # Drop a job that page one already returned. An offset-based token would
+ # shift the remaining jobs down a slot and skip one.
+ transcribe.delete_transcription_job(TranscriptionJobName=returned[0])
+
+ second = transcribe.list_transcription_jobs(
+ JobNameContains=marker, MaxResults=2, NextToken=first["NextToken"]
+ )
+ returned += [s["TranscriptionJobName"] for s in second["TranscriptionJobSummaries"]]
+
+ assert set(returned) == set(names)
+ assert len(returned) == len(set(returned))
+
+
+def test_transcribe_list_rejects_a_malformed_next_token(transcribe):
+ """A token the service did not issue must be refused rather than silently
+ treated as the first page, which would loop a paginating client forever."""
+ with pytest.raises(ClientError) as exc:
+ transcribe.list_transcription_jobs(NextToken="not-a-real-token")
+ assert exc.value.response["Error"]["Code"] == "BadRequestException"
+
+
+def test_transcribe_recreating_a_deleted_job_name_emits_one_event(
+ transcribe, eb, sqs, media
+):
+ """A deleted job's worker is not cancelled. Without a per-run guard it
+ would also drive the replacement job, completing it early and emitting a
+ second state change event, so anything chained off the event runs twice."""
+ _, _, media_uri = media
+ suffix = uuid.uuid4().hex[:8]
+ queue_url = sqs.create_queue(QueueName=f"transcribe-rerun-{suffix}")["QueueUrl"]
+ queue_arn = sqs.get_queue_attributes(
+ QueueUrl=queue_url, AttributeNames=["QueueArn"]
+ )["Attributes"]["QueueArn"]
+
+ job_name = f"rerun-{suffix}"
+ rule_name = f"transcribe-rerun-rule-{suffix}"
+ eb.put_rule(
+ Name=rule_name,
+ EventPattern=json.dumps(
+ {
+ "source": ["aws.transcribe"],
+ "detail": {"TranscriptionJobName": [job_name]},
+ }
+ ),
+ State="ENABLED",
+ )
+ eb.put_targets(Rule=rule_name, Targets=[{"Id": "1", "Arn": queue_arn}])
+
+ try:
+ start = dict(
+ TranscriptionJobName=job_name,
+ LanguageCode="en-US",
+ Media={"MediaFileUri": media_uri},
+ )
+ transcribe.start_transcription_job(**start)
+ # Delete while still QUEUED, then immediately reuse the name.
+ transcribe.delete_transcription_job(TranscriptionJobName=job_name)
+ transcribe.start_transcription_job(**start)
+
+ job = _wait_for_status(transcribe, job_name)
+ assert job["TranscriptionJobStatus"] == "COMPLETED"
+
+ # Drain well past the point the orphaned worker would have fired.
+ deadline = time.time() + 12
+ events = []
+ while time.time() < deadline:
+ batch = sqs.receive_message(
+ QueueUrl=queue_url, MaxNumberOfMessages=10, WaitTimeSeconds=1
+ ).get("Messages", [])
+ events += batch
+ for message in batch:
+ sqs.delete_message(
+ QueueUrl=queue_url, ReceiptHandle=message["ReceiptHandle"]
+ )
+
+ assert len(events) == 1, (
+ f"expected exactly one state change event, got {len(events)}: "
+ f"{[json.loads(e['Body'])['detail'] for e in events]}"
+ )
+ finally:
+ eb.remove_targets(Rule=rule_name, Ids=["1"])
+ eb.delete_rule(Name=rule_name)
+ sqs.delete_queue(QueueUrl=queue_url)
+
+
+def test_transcribe_jobs_are_region_scoped(transcribe, s3, media):
+ bucket, _, east_uri = media
+ s3.put_object(Bucket=bucket, Key="audio/west.mp3", Body=b"a different sound")
+ west_uri = f"s3://{bucket}/audio/west.mp3"
+
+ west = boto3.client(
+ "transcribe",
+ endpoint_url=_ENDPOINT,
+ aws_access_key_id="test",
+ aws_secret_access_key="test",
+ region_name="us-west-2",
+ )
+ job_name = _unique("regional")
+
+ transcribe.start_transcription_job(
+ TranscriptionJobName=job_name,
+ LanguageCode="en-US",
+ Media={"MediaFileUri": east_uri},
+ )
+
+ assert west.list_transcription_jobs(JobNameContains=job_name)[
+ "TranscriptionJobSummaries"
+ ] == []
+ with pytest.raises(ClientError) as exc:
+ west.get_transcription_job(TranscriptionJobName=job_name)
+ assert exc.value.response["Error"]["Code"] == "NotFoundException"
+
+ # The background worker pins the region it was started for, so the job
+ # still finishes in us-east-1 rather than leaking into whichever request
+ # happens to be in flight.
+ east_job = _wait_for_status(transcribe, job_name)
+ assert east_job["TranscriptionJobStatus"] == "COMPLETED"
+
+ # A same-named job in another region is a different job, transcribing
+ # different media. S3 buckets are account-scoped rather than region-scoped,
+ # so the two transcripts must not collide on one object.
+ west.start_transcription_job(
+ TranscriptionJobName=job_name,
+ LanguageCode="en-US",
+ Media={"MediaFileUri": west_uri},
+ )
+ west_job = _wait_for_status(west, job_name)
+ assert west_job["TranscriptionJobStatus"] == "COMPLETED"
+
+ east_uri_out = east_job["Transcript"]["TranscriptFileUri"]
+ west_uri_out = west_job["Transcript"]["TranscriptFileUri"]
+ assert east_uri_out != west_uri_out
+
+ def _text(transcript_uri):
+ out_bucket, key = _split_transcript_uri(transcript_uri)
+ document = json.loads(s3.get_object(Bucket=out_bucket, Key=key)["Body"].read())
+ return document["results"]["transcripts"][0]["transcript"]
+
+ # Each region's transcript is the one for its own media.
+ assert _text(east_uri_out) != _text(west_uri_out)
+
+
+def test_transcribe_job_state_change_reaches_an_eventbridge_target(
+ transcribe, eb, sqs, media
+):
+ """The hop the reporter's pipeline depends on: S3 -> Lambda -> Transcribe ->
+ EventBridge -> Translate/Bedrock. Without the terminal-state event the four
+ operations work but nothing downstream of the job ever runs."""
+ _, _, media_uri = media
+ suffix = uuid.uuid4().hex[:8]
+ queue_url = sqs.create_queue(QueueName=f"transcribe-events-{suffix}")["QueueUrl"]
+ queue_arn = sqs.get_queue_attributes(
+ QueueUrl=queue_url, AttributeNames=["QueueArn"]
+ )["Attributes"]["QueueArn"]
+
+ job_name = f"evented-{suffix}"
+ rule_name = f"transcribe-rule-{suffix}"
+ eb.put_rule(
+ Name=rule_name,
+ EventPattern=json.dumps(
+ {
+ "source": ["aws.transcribe"],
+ "detail-type": ["Transcribe Job State Change"],
+ "detail": {"TranscriptionJobName": [job_name]},
+ }
+ ),
+ State="ENABLED",
+ )
+ eb.put_targets(Rule=rule_name, Targets=[{"Id": "1", "Arn": queue_arn}])
+
+ try:
+ transcribe.start_transcription_job(
+ TranscriptionJobName=job_name,
+ LanguageCode="en-US",
+ Media={"MediaFileUri": media_uri},
+ )
+ _wait_for_status(transcribe, job_name)
+
+ deadline = time.time() + 15
+ messages = []
+ while time.time() < deadline and not messages:
+ messages = sqs.receive_message(
+ QueueUrl=queue_url, MaxNumberOfMessages=10, WaitTimeSeconds=1
+ ).get("Messages", [])
+
+ assert messages, "no Transcribe Job State Change event was delivered"
+ event = json.loads(messages[0]["Body"])
+ assert event["source"] == "aws.transcribe"
+ assert event["detail-type"] == "Transcribe Job State Change"
+ assert event["detail"] == {
+ "TranscriptionJobName": job_name,
+ "TranscriptionJobStatus": "COMPLETED",
+ }
+ finally:
+ eb.remove_targets(Rule=rule_name, Ids=["1"])
+ eb.delete_rule(Name=rule_name)
+ sqs.delete_queue(QueueUrl=queue_url)