Skip to content

Open-source speech recognition toolkit for training, inference, streaming ASR, VAD, punctuation, speaker diarization pipelines, and OpenAI-compatible/MCP serving.

docs: add Training page with complete fine-tuning guide (data prep, multi-GPU, DeepSpeed, monitoring)

游雁abc132e7bb8b
diff --git a/gh-pages-output/api.html b/gh-pages-output/api.html
new file mode 100644
index 000000000..dc7bcbffb
--- /dev/null
+++ b/gh-pages-output/api.html
@@ -0,0 +1,27935 @@
+<!DOCTYPE html>
+<html lang="en"><head><meta charset="UTF-8"><meta name="viewport" content="width=device-width,initial-scale=1.0">
+<title>FunASR API</title>
+<link href="https://fonts.googleapis.com/css2?family=Inter:wght@400;500;600;700&family=JetBrains+Mono:wght@400;500&display=swap" rel="stylesheet">
+<link rel="stylesheet" href="style.css">
+<style>
+body{display:flex;flex-direction:column;min-height:100vh;background:var(--surface)}
+.api-layout{display:flex;flex:1}
+.api-sidebar{width:260px;min-width:260px;height:calc(100vh - 57px);position:sticky;top:57px;overflow-y:auto;border-right:1px solid var(--line);background:var(--page);padding:8px 0}
+.sb-search{margin:4px 8px 10px;padding:7px 10px;width:calc(100% - 16px);border:1px solid var(--line);border-radius:5px;font-size:0.78rem}
+.sb-search:focus{outline:none;border-color:var(--primary)}
+.l1{border-bottom:1px solid var(--line)}.l1-title{padding:9px 12px;font-size:0.78rem;font-weight:700;color:var(--ink-soft);cursor:pointer;display:flex;align-items:center;gap:6px}
+.l1-title:hover{background:var(--surface-2)}.l1-title .cnt{font-size:0.65rem;background:var(--surface-2);color:var(--muted);padding:1px 5px;border-radius:8px;margin-left:auto}
+.l1-title .arr{font-size:0.6rem;color:var(--muted);transition:transform 0.2s}.l1-children{max-height:0;overflow:hidden;transition:max-height 0.3s}
+.l1.open>.l1-children{max-height:20000px}.l1.open>.l1-title .arr{transform:rotate(90deg)}
+.l2-title{padding:4px 12px 4px 22px;font-size:0.74rem;font-weight:600;color:var(--muted);cursor:pointer;display:flex;align-items:center}
+.l2-title:hover{background:var(--surface-2);color:var(--ink-soft)}.l2-title .arr{margin-left:auto;font-size:0.55rem;transition:transform 0.2s}
+.l2-children{max-height:0;overflow:hidden;transition:max-height 0.25s}.l2.open>.l2-children{max-height:20000px}.l2.open>.l2-title .arr{transform:rotate(90deg)}
+.l3-item{display:flex;align-items:center;gap:5px;padding:2px 12px 2px 34px;font-size:0.73rem;color:var(--ink-soft);text-decoration:none;cursor:pointer;border-left:2px solid transparent}
+.l3-item:hover{background:var(--surface-2)}.l3-item.active{background:var(--primary-soft);color:var(--primary-dark);border-left-color:var(--primary);font-weight:600}
+.l3m{padding-left:44px;font-size:0.7rem}
+.mb{font-size:0.5rem;font-weight:700;width:13px;height:13px;display:inline-flex;align-items:center;justify-content:center;border-radius:2px;flex-shrink:0}
+.badge-c{background:#dbeafe;color:#1d4ed8}.badge-m{background:#d1fae5;color:#059669}.badge-f{background:#fef3c7;color:#b45309}
+.api-content{flex:1;padding:32px 40px;overflow-y:auto;max-width:800px}
+.api-detail{display:none}.api-detail.active{display:block}
+.api-welcome{color:var(--muted);padding:60px 20px;text-align:center}.api-welcome h2{color:var(--ink-soft);margin-bottom:8px}
+.dtag{font-size:0.63rem;font-weight:700;padding:2px 7px;border-radius:3px;text-transform:uppercase}
+h2{font-size:1.1rem;margin:8px 0}h2 code{font-family:'JetBrains Mono',monospace;font-size:0.92rem;background:none;color:var(--ink);padding:0}
+.dmod{font-size:0.7rem;color:var(--muted);font-family:monospace;margin-bottom:18px;padding-bottom:12px;border-bottom:1px solid var(--line)}
+.dmod a{color:var(--primary);text-decoration:none;font-size:0.7rem}.dmod a:hover{text-decoration:underline}
+.ddoc{line-height:1.7}.ddoc p{margin-bottom:8px;font-size:0.87rem}.ddoc h4{font-size:0.82rem;font-weight:700;margin:16px 0 4px;color:var(--ink)}
+.ddoc ul{list-style:none;padding:0;margin:0 0 12px}.ddoc li{padding:3px 0 3px 12px;border-left:2px solid var(--line);font-size:0.83rem;color:var(--ink-soft)}
+.ddoc li code{background:var(--surface-2);padding:1px 4px;border-radius:3px;font-size:0.77rem;color:#b91c1c}
+.ddoc li.sub{padding-left:24px;font-size:0.79rem;color:var(--muted);border-left-color:transparent}
+.mt{font-size:0.85rem;font-weight:700;margin:24px 0 10px;padding-top:14px;border-top:1px solid var(--line)}
+.mblk{margin:8px 0;padding:10px 14px;background:var(--page);border-radius:6px;border:1px solid var(--line)}
+.mblk code{font-family:'JetBrains Mono',monospace;font-size:0.84rem}
+.mdoc p{font-size:0.83rem}.mdoc h4{font-size:0.78rem}
+.gh-link{font-size:0.68rem;color:var(--muted);text-decoration:none;margin-left:8px}.gh-link:hover{color:var(--primary)}
+.src-block{margin:12px 0;border:1px solid var(--line);border-radius:6px;overflow:hidden}
+.src-block summary{padding:8px 14px;font-size:0.78rem;cursor:pointer;color:var(--muted);background:var(--surface-2)}
+.src-block summary:hover{color:var(--ink-soft)}
+.src-block pre{margin:0;border-radius:0;font-size:0.75rem;max-height:400px;overflow-y:auto}
+.src-more{display:block;padding:8px 14px;font-size:0.75rem;color:var(--primary);text-decoration:none;border-top:1px solid var(--line)}
+.src-more:hover{background:var(--surface-2)}
+.muted{color:var(--muted-2);font-style:italic}
+.hidden{display:none!important}
+</style></head><body>
+<nav class="nav"><div class="container">
+<a href="index.html" class="nav-logo">FunASR</a>
+<div class="nav-links"><a href="index.html">Home</a><a href="tutorial.html">Tutorial</a><a href="model-registration.html">Develop</a><a href="api.html" class="active">API</a></div>
+<a href="https://github.com/modelscope/FunASR" class="nav-github">GitHub</a>
+</div></nav>
+<div class="api-layout">
+<div class="api-sidebar">
+<input type="text" class="sb-search" placeholder="Search..." oninput="filterSidebar(this.value)">
+<div class="l1"><div class="l1-title" onclick="toggleL1(this)"><span>funasr.auto</span><span class="cnt">1</span><span class="arr">▸</span></div><div class="l1-children">
+<div class="l2"><div class="l2-title" onclick="toggleL2(this)"><span>auto_model</span><span class="arr">▸</span></div><div class="l2-children">
+<a class="l3-item" data-target="e1" onclick="showEntry('e1')"><span class="mb badge-f">F</span>is_npu_available</a>
+<a class="l3-item" data-target="e2" onclick="showEntry('e2')"><span class="mb badge-f">F</span>prepare_data_iterator</a>
+<a class="l3-item" data-target="e3" onclick="showEntry('e3')"><span class="mb badge-c">C</span>AutoModel</a>
+<a class="l3-item l3m" data-target="e4" onclick="showEntry('e4')"><span class="mb badge-m">M</span>.build_model()</a>
+<a class="l3-item l3m" data-target="e5" onclick="showEntry('e5')"><span class="mb badge-m">M</span>.generate()</a>
+<a class="l3-item l3m" data-target="e6" onclick="showEntry('e6')"><span class="mb badge-m">M</span>.inference()</a>
+<a class="l3-item l3m" data-target="e7" onclick="showEntry('e7')"><span class="mb badge-m">M</span>.inference_with_vad()</a>
+<a class="l3-item l3m" data-target="e8" onclick="showEntry('e8')"><span class="mb badge-m">M</span>.export()</a>
+</div></div>
+</div></div>
+<div class="l1"><div class="l1-title" onclick="toggleL1(this)"><span>funasr.register</span><span class="cnt">1</span><span class="arr">▸</span></div><div class="l1-children">
+<div class="l2"><div class="l2-title" onclick="toggleL2(this)"><span>register</span><span class="arr">▸</span></div><div class="l2-children">
+<a class="l3-item" data-target="e9" onclick="showEntry('e9')"><span class="mb badge-c">C</span>RegisterTables</a>
+<a class="l3-item l3m" data-target="e10" onclick="showEntry('e10')"><span class="mb badge-m">M</span>.print()</a>
+<a class="l3-item l3m" data-target="e11" onclick="showEntry('e11')"><span class="mb badge-m">M</span>.register()</a>
+</div></div>
+</div></div>
+<div class="l1"><div class="l1-title" onclick="toggleL1(this)"><span>funasr.models</span><span class="cnt">39</span><span class="arr">▸</span></div><div class="l1-children">
+<div class="l2"><div class="l2-title" onclick="toggleL2(this)"><span>bat</span><span class="arr">▸</span></div><div class="l2-children">
+<a class="l3-item" data-target="e12" onclick="showEntry('e12')"><span class="mb badge-c">C</span>BAT</a>
+</div></div>
+<div class="l2"><div class="l2-title" onclick="toggleL2(this)"><span>bicif_paraformer</span><span class="arr">▸</span></div><div class="l2-children">
+<a class="l3-item" data-target="e13" onclick="showEntry('e13')"><span class="mb badge-c">C</span>BiCifParaformer</a>
+<a class="l3-item l3m" data-target="e14" onclick="showEntry('e14')"><span class="mb badge-m">M</span>.calc_predictor()</a>
+<a class="l3-item l3m" data-target="e15" onclick="showEntry('e15')"><span class="mb badge-m">M</span>.calc_predictor_timestamp()</a>
+<a class="l3-item l3m" data-target="e16" onclick="showEntry('e16')"><span class="mb badge-m">M</span>.forward()</a>
+<a class="l3-item l3m" data-target="e17" onclick="showEntry('e17')"><span class="mb badge-m">M</span>.inference()</a>
+<a class="l3-item l3m" data-target="e18" onclick="showEntry('e18')"><span class="mb badge-m">M</span>.export()</a>
+</div></div>
+<div class="l2"><div class="l2-title" onclick="toggleL2(this)"><span>branchformer</span><span class="arr">▸</span></div><div class="l2-children">
+<a class="l3-item" data-target="e19" onclick="showEntry('e19')"><span class="mb badge-c">C</span>Branchformer</a>
+</div></div>
+<div class="l2"><div class="l2-title" onclick="toggleL2(this)"><span>campplus</span><span class="arr">▸</span></div><div class="l2-children">
+<a class="l3-item" data-target="e20" onclick="showEntry('e20')"><span class="mb badge-c">C</span>CAMPPlus</a>
+<a class="l3-item l3m" data-target="e21" onclick="showEntry('e21')"><span class="mb badge-m">M</span>.forward()</a>
+<a class="l3-item l3m" data-target="e22" onclick="showEntry('e22')"><span class="mb badge-m">M</span>.inference()</a>
+</div></div>
+<div class="l2"><div class="l2-title" onclick="toggleL2(this)"><span>conformer</span><span class="arr">▸</span></div><div class="l2-children">
+<a class="l3-item" data-target="e23" onclick="showEntry('e23')"><span class="mb badge-c">C</span>Conformer</a>
+</div></div>
+<div class="l2"><div class="l2-title" onclick="toggleL2(this)"><span>conformer_rwkv</span><span class="arr">▸</span></div><div class="l2-children">
+<a class="l3-item" data-target="e24" onclick="showEntry('e24')"><span class="mb badge-c">C</span>Conformer</a>
+</div></div>
+<div class="l2"><div class="l2-title" onclick="toggleL2(this)"><span>contextual_paraformer</span><span class="arr">▸</span></div><div class="l2-children">
+<a class="l3-item" data-target="e25" onclick="showEntry('e25')"><span class="mb badge-c">C</span>ContextualParaformer</a>
+<a class="l3-item l3m" data-target="e26" onclick="showEntry('e26')"><span class="mb badge-m">M</span>.forward()</a>
+<a class="l3-item l3m" data-target="e27" onclick="showEntry('e27')"><span class="mb badge-m">M</span>.sampler()</a>
+<a class="l3-item l3m" data-target="e28" onclick="showEntry('e28')"><span class="mb badge-m">M</span>.cal_decoder_with_predictor()</a>
+<a class="l3-item l3m" data-target="e29" onclick="showEntry('e29')"><span class="mb badge-m">M</span>.inference()</a>
+<a class="l3-item l3m" data-target="e30" onclick="showEntry('e30')"><span class="mb badge-m">M</span>.generate_hotwords_list()</a>
+<a class="l3-item l3m" data-target="e31" onclick="showEntry('e31')"><span class="mb badge-m">M</span>.export()</a>
+</div></div>
+<div class="l2"><div class="l2-title" onclick="toggleL2(this)"><span>ct_transformer</span><span class="arr">▸</span></div><div class="l2-children">
+<a class="l3-item" data-target="e32" onclick="showEntry('e32')"><span class="mb badge-c">C</span>CTTransformer</a>
+<a class="l3-item l3m" data-target="e33" onclick="showEntry('e33')"><span class="mb badge-m">M</span>.punc_forward()</a>
+<a class="l3-item l3m" data-target="e34" onclick="showEntry('e34')"><span class="mb badge-m">M</span>.with_vad()</a>
+<a class="l3-item l3m" data-target="e35" onclick="showEntry('e35')"><span class="mb badge-m">M</span>.score()</a>
+<a class="l3-item l3m" data-target="e36" onclick="showEntry('e36')"><span class="mb badge-m">M</span>.batch_score()</a>
+<a class="l3-item l3m" data-target="e37" onclick="showEntry('e37')"><span class="mb badge-m">M</span>.nll()</a>
+<a class="l3-item l3m" data-target="e38" onclick="showEntry('e38')"><span class="mb badge-m">M</span>.forward()</a>
+<a class="l3-item l3m" data-target="e39" onclick="showEntry('e39')"><span class="mb badge-m">M</span>.inference()</a>
+<a class="l3-item l3m" data-target="e40" onclick="showEntry('e40')"><span class="mb badge-m">M</span>.export()</a>
+</div></div>
+<div class="l2"><div class="l2-title" onclick="toggleL2(this)"><span>ct_transformer_streaming</span><span class="arr">▸</span></div><div class="l2-children">
+<a class="l3-item" data-target="e41" onclick="showEntry('e41')"><span class="mb badge-c">C</span>CTTransformerStreaming</a>
+<a class="l3-item l3m" data-target="e42" onclick="showEntry('e42')"><span class="mb badge-m">M</span>.punc_forward()</a>
+<a class="l3-item l3m" data-target="e43" onclick="showEntry('e43')"><span class="mb badge-m">M</span>.with_vad()</a>
+<a class="l3-item l3m" data-target="e44" onclick="showEntry('e44')"><span class="mb badge-m">M</span>.inference()</a>
+<a class="l3-item l3m" data-target="e45" onclick="showEntry('e45')"><span class="mb badge-m">M</span>.export()</a>
+</div></div>
+<div class="l2"><div class="l2-title" onclick="toggleL2(this)"><span>ctc</span><span class="arr">▸</span></div><div class="l2-children">
+<a class="l3-item" data-target="e46" onclick="showEntry('e46')"><span class="mb badge-c">C</span>Transformer</a>
+<a class="l3-item l3m" data-target="e47" onclick="showEntry('e47')"><span class="mb badge-m">M</span>.forward()</a>
+<a class="l3-item l3m" data-target="e48" onclick="showEntry('e48')"><span class="mb badge-m">M</span>.encode()</a>
+<a class="l3-item l3m" data-target="e49" onclick="showEntry('e49')"><span class="mb badge-m">M</span>.inference()</a>
+</div></div>
+<div class="l2"><div class="l2-title" onclick="toggleL2(this)"><span>e_branchformer</span><span class="arr">▸</span></div><div class="l2-children">
+<a class="l3-item" data-target="e50" onclick="showEntry('e50')"><span class="mb badge-c">C</span>EBranchformer</a>
+</div></div>
+<div class="l2"><div class="l2-title" onclick="toggleL2(this)"><span>e_paraformer</span><span class="arr">▸</span></div><div class="l2-children">
+<a class="l3-item" data-target="e51" onclick="showEntry('e51')"><span class="mb badge-c">C</span>EParaformer</a>
+<a class="l3-item l3m" data-target="e52" onclick="showEntry('e52')"><span class="mb badge-m">M</span>.forward()</a>
+<a class="l3-item l3m" data-target="e53" onclick="showEntry('e53')"><span class="mb badge-m">M</span>.encode()</a>
+<a class="l3-item l3m" data-target="e54" onclick="showEntry('e54')"><span class="mb badge-m">M</span>.calc_predictor()</a>
+<a class="l3-item l3m" data-target="e55" onclick="showEntry('e55')"><span class="mb badge-m">M</span>.cal_decoder_with_predictor()</a>
+<a class="l3-item l3m" data-target="e56" onclick="showEntry('e56')"><span class="mb badge-m">M</span>.sampler()</a>
+<a class="l3-item l3m" data-target="e57" onclick="showEntry('e57')"><span class="mb badge-m">M</span>.sampler_with_grad()</a>
+<a class="l3-item l3m" data-target="e58" onclick="showEntry('e58')"><span class="mb badge-m">M</span>.init_beam_search()</a>
+<a class="l3-item l3m" data-target="e59" onclick="showEntry('e59')"><span class="mb badge-m">M</span>.inference()</a>
+<a class="l3-item l3m" data-target="e60" onclick="showEntry('e60')"><span class="mb badge-m">M</span>.export()</a>
+</div></div>
+<div class="l2"><div class="l2-title" onclick="toggleL2(this)"><span>emotion2vec</span><span class="arr">▸</span></div><div class="l2-children">
+<a class="l3-item" data-target="e61" onclick="showEntry('e61')"><span class="mb badge-c">C</span>Emotion2vec</a>
+<a class="l3-item l3m" data-target="e62" onclick="showEntry('e62')"><span class="mb badge-m">M</span>.forward()</a>
+<a class="l3-item l3m" data-target="e63" onclick="showEntry('e63')"><span class="mb badge-m">M</span>.extract_features()</a>
+<a class="l3-item l3m" data-target="e64" onclick="showEntry('e64')"><span class="mb badge-m">M</span>.inference()</a>
+<a class="l3-item l3m" data-target="e65" onclick="showEntry('e65')"><span class="mb badge-m">M</span>.export()</a>
+</div></div>
+<div class="l2"><div class="l2-title" onclick="toggleL2(this)"><span>eres2net</span><span class="arr">▸</span></div><div class="l2-children">
+<a class="l3-item" data-target="e66" onclick="showEntry('e66')"><span class="mb badge-c">C</span>ERes2NetV2SV</a>
+<a class="l3-item l3m" data-target="e67" onclick="showEntry('e67')"><span class="mb badge-m">M</span>.forward()</a>
+<a class="l3-item l3m" data-target="e68" onclick="showEntry('e68')"><span class="mb badge-m">M</span>.inference()</a>
+</div></div>
+<div class="l2"><div class="l2-title" onclick="toggleL2(this)"><span>fsmn_kws</span><span class="arr">▸</span></div><div class="l2-children">
+<a class="l3-item" data-target="e69" onclick="showEntry('e69')"><span class="mb badge-c">C</span>FsmnKWS</a>
+<a class="l3-item l3m" data-target="e70" onclick="showEntry('e70')"><span class="mb badge-m">M</span>.forward()</a>
+<a class="l3-item l3m" data-target="e71" onclick="showEntry('e71')"><span class="mb badge-m">M</span>.encode()</a>
+<a class="l3-item l3m" data-target="e72" onclick="showEntry('e72')"><span class="mb badge-m">M</span>.inference()</a>
+<a class="l3-item" data-target="e73" onclick="showEntry('e73')"><span class="mb badge-c">C</span>FsmnKWSConvert</a>
+<a class="l3-item l3m" data-target="e74" onclick="showEntry('e74')"><span class="mb badge-m">M</span>.to_kaldi_net()</a>
+<a class="l3-item l3m" data-target="e75" onclick="showEntry('e75')"><span class="mb badge-m">M</span>.to_pytorch_net()</a>
+</div></div>
+<div class="l2"><div class="l2-title" onclick="toggleL2(this)"><span>fsmn_kws_mt</span><span class="arr">▸</span></div><div class="l2-children">
+<a class="l3-item" data-target="e76" onclick="showEntry('e76')"><span class="mb badge-c">C</span>FsmnKWSMT</a>
+<a class="l3-item l3m" data-target="e77" onclick="showEntry('e77')"><span class="mb badge-m">M</span>.forward()</a>
+<a class="l3-item l3m" data-target="e78" onclick="showEntry('e78')"><span class="mb badge-m">M</span>.encode()</a>
+<a class="l3-item l3m" data-target="e79" onclick="showEntry('e79')"><span class="mb badge-m">M</span>.inference()</a>
+<a class="l3-item" data-target="e80" onclick="showEntry('e80')"><span class="mb badge-c">C</span>FsmnKWSMTConvert</a>
+<a class="l3-item l3m" data-target="e81" onclick="showEntry('e81')"><span class="mb badge-m">M</span>.to_kaldi_net()</a>
+<a class="l3-item l3m" data-target="e82" onclick="showEntry('e82')"><span class="mb badge-m">M</span>.to_kaldi_net2()</a>
+<a class="l3-item l3m" data-target="e83" onclick="showEntry('e83')"><span class="mb badge-m">M</span>.to_pytorch_net()</a>
+</div></div>
+<div class="l2"><div class="l2-title" onclick="toggleL2(this)"><span>fsmn_vad_streaming</span><span class="arr">▸</span></div><div class="l2-children">
+<a class="l3-item" data-target="e84" onclick="showEntry('e84')"><span class="mb badge-c">C</span>FsmnVADStreaming</a>
+<a class="l3-item l3m" data-target="e85" onclick="showEntry('e85')"><span class="mb badge-m">M</span>.ResetDetection()</a>
+<a class="l3-item l3m" data-target="e86" onclick="showEntry('e86')"><span class="mb badge-m">M</span>.ComputeDecibel()</a>
+<a class="l3-item l3m" data-target="e87" onclick="showEntry('e87')"><span class="mb badge-m">M</span>.ComputeScores()</a>
+<a class="l3-item l3m" data-target="e88" onclick="showEntry('e88')"><span class="mb badge-m">M</span>.PopDataBufTillFrame()</a>
+<a class="l3-item l3m" data-target="e89" onclick="showEntry('e89')"><span class="mb badge-m">M</span>.PopDataToOutputBuf()</a>
+<a class="l3-item l3m" data-target="e90" onclick="showEntry('e90')"><span class="mb badge-m">M</span>.OnSilenceDetected()</a>
+<a class="l3-item l3m" data-target="e91" onclick="showEntry('e91')"><span class="mb badge-m">M</span>.OnVoiceDetected()</a>
+<a class="l3-item l3m" data-target="e92" onclick="showEntry('e92')"><span class="mb badge-m">M</span>.OnVoiceStart()</a>
+<a class="l3-item l3m" data-target="e93" onclick="showEntry('e93')"><span class="mb badge-m">M</span>.OnVoiceEnd()</a>
+<a class="l3-item l3m" data-target="e94" onclick="showEntry('e94')"><span class="mb badge-m">M</span>.MaybeOnVoiceEndIfLastFrame()</a>
+<a class="l3-item l3m" data-target="e95" onclick="showEntry('e95')"><span class="mb badge-m">M</span>.GetLatency()</a>
+<a class="l3-item l3m" data-target="e96" onclick="showEntry('e96')"><span class="mb badge-m">M</span>.LatencyFrmNumAtStartPoint()</a>
+<a class="l3-item l3m" data-target="e97" onclick="showEntry('e97')"><span class="mb badge-m">M</span>.GetFrameState()</a>
+<a class="l3-item l3m" data-target="e98" onclick="showEntry('e98')"><span class="mb badge-m">M</span>.forward()</a>
+<a class="l3-item l3m" data-target="e99" onclick="showEntry('e99')"><span class="mb badge-m">M</span>.init_cache()</a>
+<a class="l3-item l3m" data-target="e100" onclick="showEntry('e100')"><span class="mb badge-m">M</span>.inference()</a>
+<a class="l3-item l3m" data-target="e101" onclick="showEntry('e101')"><span class="mb badge-m">M</span>.export()</a>
+<a class="l3-item l3m" data-target="e102" onclick="showEntry('e102')"><span class="mb badge-m">M</span>.DetectCommonFrames()</a>
+<a class="l3-item l3m" data-target="e103" onclick="showEntry('e103')"><span class="mb badge-m">M</span>.DetectLastFrames()</a>
+<a class="l3-item l3m" data-target="e104" onclick="showEntry('e104')"><span class="mb badge-m">M</span>.DetectOneFrame()</a>
+</div></div>
+<div class="l2"><div class="l2-title" onclick="toggleL2(this)"><span>fun_asr_nano</span><span class="arr">▸</span></div><div class="l2-children">
+<a class="l3-item" data-target="e105" onclick="showEntry('e105')"><span class="mb badge-c">C</span>FunASRNano</a>
+<a class="l3-item l3m" data-target="e106" onclick="showEntry('e106')"><span class="mb badge-m">M</span>.forward()</a>
+<a class="l3-item l3m" data-target="e107" onclick="showEntry('e107')"><span class="mb badge-m">M</span>.forward_export()</a>
+<a class="l3-item l3m" data-target="e108" onclick="showEntry('e108')"><span class="mb badge-m">M</span>.encode()</a>
+<a class="l3-item l3m" data-target="e109" onclick="showEntry('e109')"><span class="mb badge-m">M</span>.data_template()</a>
+<a class="l3-item l3m" data-target="e110" onclick="showEntry('e110')"><span class="mb badge-m">M</span>.data_load_speech()</a>
+<a class="l3-item l3m" data-target="e111" onclick="showEntry('e111')"><span class="mb badge-m">M</span>.inference_prepare()</a>
+<a class="l3-item l3m" data-target="e112" onclick="showEntry('e112')"><span class="mb badge-m">M</span>.get_prompt()</a>
+<a class="l3-item l3m" data-target="e113" onclick="showEntry('e113')"><span class="mb badge-m">M</span>.generate_chatml()</a>
+<a class="l3-item l3m" data-target="e114" onclick="showEntry('e114')"><span class="mb badge-m">M</span>.inference()</a>
+<a class="l3-item l3m" data-target="e115" onclick="showEntry('e115')"><span class="mb badge-m">M</span>.inference_llm()</a>
+<a class="l3-item l3m" data-target="e116" onclick="showEntry('e116')"><span class="mb badge-m">M</span>.from_pretrained()</a>
+</div></div>
+<div class="l2"><div class="l2-title" onclick="toggleL2(this)"><span>glm_asr</span><span class="arr">▸</span></div><div class="l2-children">
+<a class="l3-item" data-target="e117" onclick="showEntry('e117')"><span class="mb badge-c">C</span>GLMASR</a>
+<a class="l3-item l3m" data-target="e118" onclick="showEntry('e118')"><span class="mb badge-m">M</span>.forward()</a>
+<a class="l3-item l3m" data-target="e119" onclick="showEntry('e119')"><span class="mb badge-m">M</span>.inference()</a>
+</div></div>
+<div class="l2"><div class="l2-title" onclick="toggleL2(this)"><span>lcbnet</span><span class="arr">▸</span></div><div class="l2-children">
+<a class="l3-item" data-target="e120" onclick="showEntry('e120')"><span class="mb badge-c">C</span>LCBNet</a>
+<a class="l3-item l3m" data-target="e121" onclick="showEntry('e121')"><span class="mb badge-m">M</span>.forward()</a>
+<a class="l3-item l3m" data-target="e122" onclick="showEntry('e122')"><span class="mb badge-m">M</span>.encode()</a>
+<a class="l3-item l3m" data-target="e123" onclick="showEntry('e123')"><span class="mb badge-m">M</span>.init_beam_search()</a>
+<a class="l3-item l3m" data-target="e124" onclick="showEntry('e124')"><span class="mb badge-m">M</span>.inference()</a>
+</div></div>
+<div class="l2"><div class="l2-title" onclick="toggleL2(this)"><span>llm_asr</span><span class="arr">▸</span></div><div class="l2-children">
+<a class="l3-item" data-target="e125" onclick="showEntry('e125')"><span class="mb badge-c">C</span>LLMASR</a>
+<a class="l3-item l3m" data-target="e126" onclick="showEntry('e126')"><span class="mb badge-m">M</span>.forward()</a>
+<a class="l3-item l3m" data-target="e127" onclick="showEntry('e127')"><span class="mb badge-m">M</span>.encode()</a>
+<a class="l3-item l3m" data-target="e128" onclick="showEntry('e128')"><span class="mb badge-m">M</span>.inference()</a>
+<a class="l3-item" data-target="e129" onclick="showEntry('e129')"><span class="mb badge-c">C</span>LLMASR2</a>
+<a class="l3-item l3m" data-target="e130" onclick="showEntry('e130')"><span class="mb badge-m">M</span>.forward()</a>
+<a class="l3-item l3m" data-target="e131" onclick="showEntry('e131')"><span class="mb badge-m">M</span>.encode()</a>
+<a class="l3-item l3m" data-target="e132" onclick="showEntry('e132')"><span class="mb badge-m">M</span>.data_template()</a>
+<a class="l3-item l3m" data-target="e133" onclick="showEntry('e133')"><span class="mb badge-m">M</span>.data_load_speech()</a>
+<a class="l3-item l3m" data-target="e134" onclick="showEntry('e134')"><span class="mb badge-m">M</span>.inference()</a>
+<a class="l3-item" data-target="e135" onclick="showEntry('e135')"><span class="mb badge-c">C</span>LLMASR3</a>
+<a class="l3-item l3m" data-target="e136" onclick="showEntry('e136')"><span class="mb badge-m">M</span>.encode()</a>
+<a class="l3-item" data-target="e137" onclick="showEntry('e137')"><span class="mb badge-c">C</span>LLMASR4</a>
+<a class="l3-item l3m" data-target="e138" onclick="showEntry('e138')"><span class="mb badge-m">M</span>.forward()</a>
+<a class="l3-item l3m" data-target="e139" onclick="showEntry('e139')"><span class="mb badge-m">M</span>.encode()</a>
+<a class="l3-item l3m" data-target="e140" onclick="showEntry('e140')"><span class="mb badge-m">M</span>.data_template()</a>
+<a class="l3-item l3m" data-target="e141" onclick="showEntry('e141')"><span class="mb badge-m">M</span>.data_load_speech()</a>
+<a class="l3-item l3m" data-target="e142" onclick="showEntry('e142')"><span class="mb badge-m">M</span>.inference_prepare()</a>
+<a class="l3-item l3m" data-target="e143" onclick="showEntry('e143')"><span class="mb badge-m">M</span>.inference()</a>
+</div></div>
+<div class="l2"><div class="l2-title" onclick="toggleL2(this)"><span>llm_asr_nar</span><span class="arr">▸</span></div><div class="l2-children">
+<a class="l3-item" data-target="e144" onclick="showEntry('e144')"><span class="mb badge-c">C</span>LLMASRNAR</a>
+<a class="l3-item l3m" data-target="e145" onclick="showEntry('e145')"><span class="mb badge-m">M</span>.forward()</a>
+<a class="l3-item l3m" data-target="e146" onclick="showEntry('e146')"><span class="mb badge-m">M</span>.encode()</a>
+<a class="l3-item l3m" data-target="e147" onclick="showEntry('e147')"><span class="mb badge-m">M</span>.inference()</a>
+<a class="l3-item" data-target="e148" onclick="showEntry('e148')"><span class="mb badge-c">C</span>LLMASRNARPrompt</a>
+<a class="l3-item l3m" data-target="e149" onclick="showEntry('e149')"><span class="mb badge-m">M</span>.forward()</a>
+<a class="l3-item l3m" data-target="e150" onclick="showEntry('e150')"><span class="mb badge-m">M</span>.encode()</a>
+<a class="l3-item l3m" data-target="e151" onclick="showEntry('e151')"><span class="mb badge-m">M</span>.inference()</a>
+</div></div>
+<div class="l2"><div class="l2-title" onclick="toggleL2(this)"><span>monotonic_aligner</span><span class="arr">▸</span></div><div class="l2-children">
+<a class="l3-item" data-target="e152" onclick="showEntry('e152')"><span class="mb badge-c">C</span>MonotonicAligner</a>
+<a class="l3-item l3m" data-target="e153" onclick="showEntry('e153')"><span class="mb badge-m">M</span>.forward()</a>
+<a class="l3-item l3m" data-target="e154" onclick="showEntry('e154')"><span class="mb badge-m">M</span>.calc_predictor_timestamp()</a>
+<a class="l3-item l3m" data-target="e155" onclick="showEntry('e155')"><span class="mb badge-m">M</span>.encode()</a>
+<a class="l3-item l3m" data-target="e156" onclick="showEntry('e156')"><span class="mb badge-m">M</span>.inference()</a>
+</div></div>
+<div class="l2"><div class="l2-title" onclick="toggleL2(this)"><span>paraformer</span><span class="arr">▸</span></div><div class="l2-children">
+<a class="l3-item" data-target="e157" onclick="showEntry('e157')"><span class="mb badge-c">C</span>Paraformer</a>
+<a class="l3-item l3m" data-target="e158" onclick="showEntry('e158')"><span class="mb badge-m">M</span>.forward()</a>
+<a class="l3-item l3m" data-target="e159" onclick="showEntry('e159')"><span class="mb badge-m">M</span>.encode()</a>
+<a class="l3-item l3m" data-target="e160" onclick="showEntry('e160')"><span class="mb badge-m">M</span>.calc_predictor()</a>
+<a class="l3-item l3m" data-target="e161" onclick="showEntry('e161')"><span class="mb badge-m">M</span>.cal_decoder_with_predictor()</a>
+<a class="l3-item l3m" data-target="e162" onclick="showEntry('e162')"><span class="mb badge-m">M</span>.sampler()</a>
+<a class="l3-item l3m" data-target="e163" onclick="showEntry('e163')"><span class="mb badge-m">M</span>.init_beam_search()</a>
+<a class="l3-item l3m" data-target="e164" onclick="showEntry('e164')"><span class="mb badge-m">M</span>.inference()</a>
+<a class="l3-item l3m" data-target="e165" onclick="showEntry('e165')"><span class="mb badge-m">M</span>.export()</a>
+</div></div>
+<div class="l2"><div class="l2-title" onclick="toggleL2(this)"><span>paraformer_streaming</span><span class="arr">▸</span></div><div class="l2-children">
+<a class="l3-item" data-target="e166" onclick="showEntry('e166')"><span class="mb badge-c">C</span>ParaformerStreaming</a>
+<a class="l3-item l3m" data-target="e167" onclick="showEntry('e167')"><span class="mb badge-m">M</span>.forward()</a>
+<a class="l3-item l3m" data-target="e168" onclick="showEntry('e168')"><span class="mb badge-m">M</span>.encode_chunk()</a>
+<a class="l3-item l3m" data-target="e169" onclick="showEntry('e169')"><span class="mb badge-m">M</span>.sampler()</a>
+<a class="l3-item l3m" data-target="e170" onclick="showEntry('e170')"><span class="mb badge-m">M</span>.calc_predictor()</a>
+<a class="l3-item l3m" data-target="e171" onclick="showEntry('e171')"><span class="mb badge-m">M</span>.calc_predictor_chunk()</a>
+<a class="l3-item l3m" data-target="e172" onclick="showEntry('e172')"><span class="mb badge-m">M</span>.cal_decoder_with_predictor()</a>
+<a class="l3-item l3m" data-target="e173" onclick="showEntry('e173')"><span class="mb badge-m">M</span>.cal_decoder_with_predictor_chunk()</a>
+<a class="l3-item l3m" data-target="e174" onclick="showEntry('e174')"><span class="mb badge-m">M</span>.init_cache()</a>
+<a class="l3-item l3m" data-target="e175" onclick="showEntry('e175')"><span class="mb badge-m">M</span>.generate_chunk()</a>
+<a class="l3-item l3m" data-target="e176" onclick="showEntry('e176')"><span class="mb badge-m">M</span>.inference()</a>
+<a class="l3-item l3m" data-target="e177" onclick="showEntry('e177')"><span class="mb badge-m">M</span>.export()</a>
+</div></div>
+<div class="l2"><div class="l2-title" onclick="toggleL2(this)"><span>paraformer_v2_community</span><span class="arr">▸</span></div><div class="l2-children">
+<a class="l3-item" data-target="e178" onclick="showEntry('e178')"><span class="mb badge-c">C</span>Paraformer</a>
+<a class="l3-item l3m" data-target="e179" onclick="showEntry('e179')"><span class="mb badge-m">M</span>.forward()</a>
+<a class="l3-item l3m" data-target="e180" onclick="showEntry('e180')"><span class="mb badge-m">M</span>.encode()</a>
+<a class="l3-item l3m" data-target="e181" onclick="showEntry('e181')"><span class="mb badge-m">M</span>.map_alignment_to_target_index()</a>
+<a class="l3-item l3m" data-target="e182" onclick="showEntry('e182')"><span class="mb badge-m">M</span>.force_align()</a>
+<a class="l3-item l3m" data-target="e183" onclick="showEntry('e183')"><span class="mb badge-m">M</span>.average_repeats_training()</a>
+<a class="l3-item l3m" data-target="e184" onclick="showEntry('e184')"><span class="mb badge-m">M</span>.average_repeats_inference()</a>
+<a class="l3-item l3m" data-target="e185" onclick="showEntry('e185')"><span class="mb badge-m">M</span>.inference()</a>
+<a class="l3-item l3m" data-target="e186" onclick="showEntry('e186')"><span class="mb badge-m">M</span>.export()</a>
+</div></div>
+<div class="l2"><div class="l2-title" onclick="toggleL2(this)"><span>qwen3_asr</span><span class="arr">▸</span></div><div class="l2-children">
+<a class="l3-item" data-target="e187" onclick="showEntry('e187')"><span class="mb badge-c">C</span>Qwen3ASR</a>
+<a class="l3-item l3m" data-target="e188" onclick="showEntry('e188')"><span class="mb badge-m">M</span>.forward()</a>
+<a class="l3-item l3m" data-target="e189" onclick="showEntry('e189')"><span class="mb badge-m">M</span>.inference()</a>
+</div></div>
+<div class="l2"><div class="l2-title" onclick="toggleL2(this)"><span>qwen_audio</span><span class="arr">▸</span></div><div class="l2-children">
+<a class="l3-item" data-target="e190" onclick="showEntry('e190')"><span class="mb badge-c">C</span>QwenAudioWarp</a>
+<a class="l3-item l3m" data-target="e191" onclick="showEntry('e191')"><span class="mb badge-m">M</span>.forward()</a>
+<a class="l3-item l3m" data-target="e192" onclick="showEntry('e192')"><span class="mb badge-m">M</span>.inference()</a>
+<a class="l3-item" data-target="e193" onclick="showEntry('e193')"><span class="mb badge-c">C</span>QwenAudioChatWarp</a>
+<a class="l3-item l3m" data-target="e194" onclick="showEntry('e194')"><span class="mb badge-m">M</span>.forward()</a>
+<a class="l3-item l3m" data-target="e195" onclick="showEntry('e195')"><span class="mb badge-m">M</span>.inference()</a>
+</div></div>
+<div class="l2"><div class="l2-title" onclick="toggleL2(this)"><span>sanm</span><span class="arr">▸</span></div><div class="l2-children">
+<a class="l3-item" data-target="e196" onclick="showEntry('e196')"><span class="mb badge-c">C</span>SANM</a>
+</div></div>
+<div class="l2"><div class="l2-title" onclick="toggleL2(this)"><span>sanm_kws</span><span class="arr">▸</span></div><div class="l2-children">
+<a class="l3-item" data-target="e197" onclick="showEntry('e197')"><span class="mb badge-c">C</span>SanmKWS</a>
+<a class="l3-item l3m" data-target="e198" onclick="showEntry('e198')"><span class="mb badge-m">M</span>.forward()</a>
+<a class="l3-item l3m" data-target="e199" onclick="showEntry('e199')"><span class="mb badge-m">M</span>.encode()</a>
+<a class="l3-item l3m" data-target="e200" onclick="showEntry('e200')"><span class="mb badge-m">M</span>.inference()</a>
+<a class="l3-item l3m" data-target="e201" onclick="showEntry('e201')"><span class="mb badge-m">M</span>.export()</a>
+</div></div>
+<div class="l2"><div class="l2-title" onclick="toggleL2(this)"><span>sanm_kws_streaming</span><span class="arr">▸</span></div><div class="l2-children">
+<a class="l3-item" data-target="e202" onclick="showEntry('e202')"><span class="mb badge-c">C</span>SanmKWSStreaming</a>
+<a class="l3-item l3m" data-target="e203" onclick="showEntry('e203')"><span class="mb badge-m">M</span>.forward()</a>
+<a class="l3-item l3m" data-target="e204" onclick="showEntry('e204')"><span class="mb badge-m">M</span>.encode_chunk()</a>
+<a class="l3-item l3m" data-target="e205" onclick="showEntry('e205')"><span class="mb badge-m">M</span>.init_cache()</a>
+<a class="l3-item l3m" data-target="e206" onclick="showEntry('e206')"><span class="mb badge-m">M</span>.generate_chunk()</a>
+<a class="l3-item l3m" data-target="e207" onclick="showEntry('e207')"><span class="mb badge-m">M</span>.inference()</a>
+<a class="l3-item l3m" data-target="e208" onclick="showEntry('e208')"><span class="mb badge-m">M</span>.export()</a>
+</div></div>
+<div class="l2"><div class="l2-title" onclick="toggleL2(this)"><span>scama</span><span class="arr">▸</span></div><div class="l2-children">
+<a class="l3-item" data-target="e209" onclick="showEntry('e209')"><span class="mb badge-c">C</span>SCAMA</a>
+<a class="l3-item l3m" data-target="e210" onclick="showEntry('e210')"><span class="mb badge-m">M</span>.forward()</a>
+<a class="l3-item l3m" data-target="e211" onclick="showEntry('e211')"><span class="mb badge-m">M</span>.encode()</a>
+<a class="l3-item l3m" data-target="e212" onclick="showEntry('e212')"><span class="mb badge-m">M</span>.encode_chunk()</a>
+<a class="l3-item l3m" data-target="e213" onclick="showEntry('e213')"><span class="mb badge-m">M</span>.calc_predictor_chunk()</a>
+<a class="l3-item l3m" data-target="e214" onclick="showEntry('e214')"><span class="mb badge-m">M</span>.calc_predictor_mask()</a>
+<a class="l3-item l3m" data-target="e215" onclick="showEntry('e215')"><span class="mb badge-m">M</span>.init_beam_search()</a>
+<a class="l3-item l3m" data-target="e216" onclick="showEntry('e216')"><span class="mb badge-m">M</span>.generate_chunk()</a>
+<a class="l3-item l3m" data-target="e217" onclick="showEntry('e217')"><span class="mb badge-m">M</span>.init_cache()</a>
+<a class="l3-item l3m" data-target="e218" onclick="showEntry('e218')"><span class="mb badge-m">M</span>.inference()</a>
+</div></div>
+<div class="l2"><div class="l2-title" onclick="toggleL2(this)"><span>seaco_paraformer</span><span class="arr">▸</span></div><div class="l2-children">
+<a class="l3-item" data-target="e219" onclick="showEntry('e219')"><span class="mb badge-c">C</span>SeacoParaformer</a>
+<a class="l3-item l3m" data-target="e220" onclick="showEntry('e220')"><span class="mb badge-m">M</span>.forward()</a>
+<a class="l3-item l3m" data-target="e221" onclick="showEntry('e221')"><span class="mb badge-m">M</span>.calc_predictor()</a>
+<a class="l3-item l3m" data-target="e222" onclick="showEntry('e222')"><span class="mb badge-m">M</span>.inference()</a>
+<a class="l3-item l3m" data-target="e223" onclick="showEntry('e223')"><span class="mb badge-m">M</span>.generate_hotwords_list()</a>
+<a class="l3-item l3m" data-target="e224" onclick="showEntry('e224')"><span class="mb badge-m">M</span>.export()</a>
+<a class="l3-item" data-target="e225" onclick="showEntry('e225')"><span class="mb badge-f">F</span>load_seg_dict</a>
+</div></div>
+<div class="l2"><div class="l2-title" onclick="toggleL2(this)"><span>sense_voice</span><span class="arr">▸</span></div><div class="l2-children">
+<a class="l3-item" data-target="e226" onclick="showEntry('e226')"><span class="mb badge-f">F</span>sequence_mask</a>
+<a class="l3-item" data-target="e227" onclick="showEntry('e227')"><span class="mb badge-c">C</span>SenseVoiceEncoderSmall</a>
+<a class="l3-item l3m" data-target="e228" onclick="showEntry('e228')"><span class="mb badge-m">M</span>.output_size()</a>
+<a class="l3-item l3m" data-target="e229" onclick="showEntry('e229')"><span class="mb badge-m">M</span>.forward()</a>
+<a class="l3-item" data-target="e230" onclick="showEntry('e230')"><span class="mb badge-c">C</span>SenseVoiceSmall</a>
+<a class="l3-item l3m" data-target="e231" onclick="showEntry('e231')"><span class="mb badge-m">M</span>.from_pretrained()</a>
+<a class="l3-item l3m" data-target="e232" onclick="showEntry('e232')"><span class="mb badge-m">M</span>.forward()</a>
+<a class="l3-item l3m" data-target="e233" onclick="showEntry('e233')"><span class="mb badge-m">M</span>.encode()</a>
+<a class="l3-item l3m" data-target="e234" onclick="showEntry('e234')"><span class="mb badge-m">M</span>.inference()</a>
+<a class="l3-item l3m" data-target="e235" onclick="showEntry('e235')"><span class="mb badge-m">M</span>.post()</a>
+<a class="l3-item l3m" data-target="e236" onclick="showEntry('e236')"><span class="mb badge-m">M</span>.export()</a>
+</div></div>
+<div class="l2"><div class="l2-title" onclick="toggleL2(this)"><span>transducer</span><span class="arr">▸</span></div><div class="l2-children">
+<a class="l3-item" data-target="e237" onclick="showEntry('e237')"><span class="mb badge-c">C</span>Transducer</a>
+<a class="l3-item l3m" data-target="e238" onclick="showEntry('e238')"><span class="mb badge-m">M</span>.forward()</a>
+<a class="l3-item l3m" data-target="e239" onclick="showEntry('e239')"><span class="mb badge-m">M</span>.encode()</a>
+<a class="l3-item l3m" data-target="e240" onclick="showEntry('e240')"><span class="mb badge-m">M</span>.init_beam_search()</a>
+<a class="l3-item l3m" data-target="e241" onclick="showEntry('e241')"><span class="mb badge-m">M</span>.inference()</a>
+</div></div>
+<div class="l2"><div class="l2-title" onclick="toggleL2(this)"><span>transformer</span><span class="arr">▸</span></div><div class="l2-children">
+<a class="l3-item" data-target="e242" onclick="showEntry('e242')"><span class="mb badge-c">C</span>Transformer</a>
+<a class="l3-item l3m" data-target="e243" onclick="showEntry('e243')"><span class="mb badge-m">M</span>.forward()</a>
+<a class="l3-item l3m" data-target="e244" onclick="showEntry('e244')"><span class="mb badge-m">M</span>.encode()</a>
+<a class="l3-item l3m" data-target="e245" onclick="showEntry('e245')"><span class="mb badge-m">M</span>.init_beam_search()</a>
+<a class="l3-item l3m" data-target="e246" onclick="showEntry('e246')"><span class="mb badge-m">M</span>.inference()</a>
+</div></div>
+<div class="l2"><div class="l2-title" onclick="toggleL2(this)"><span>uniasr</span><span class="arr">▸</span></div><div class="l2-children">
+<a class="l3-item" data-target="e247" onclick="showEntry('e247')"><span class="mb badge-c">C</span>UniASR</a>
+<a class="l3-item l3m" data-target="e248" onclick="showEntry('e248')"><span class="mb badge-m">M</span>.forward()</a>
+<a class="l3-item l3m" data-target="e249" onclick="showEntry('e249')"><span class="mb badge-m">M</span>.collect_feats()</a>
+<a class="l3-item l3m" data-target="e250" onclick="showEntry('e250')"><span class="mb badge-m">M</span>.encode()</a>
+<a class="l3-item l3m" data-target="e251" onclick="showEntry('e251')"><span class="mb badge-m">M</span>.encode2()</a>
+<a class="l3-item l3m" data-target="e252" onclick="showEntry('e252')"><span class="mb badge-m">M</span>.nll()</a>
+<a class="l3-item l3m" data-target="e253" onclick="showEntry('e253')"><span class="mb badge-m">M</span>.batchify_nll()</a>
+<a class="l3-item l3m" data-target="e254" onclick="showEntry('e254')"><span class="mb badge-m">M</span>.calc_predictor_mask()</a>
+<a class="l3-item l3m" data-target="e255" onclick="showEntry('e255')"><span class="mb badge-m">M</span>.calc_predictor_mask2()</a>
+<a class="l3-item l3m" data-target="e256" onclick="showEntry('e256')"><span class="mb badge-m">M</span>.init_beam_search()</a>
+<a class="l3-item l3m" data-target="e257" onclick="showEntry('e257')"><span class="mb badge-m">M</span>.inference()</a>
+</div></div>
+<div class="l2"><div class="l2-title" onclick="toggleL2(this)"><span>whisper</span><span class="arr">▸</span></div><div class="l2-children">
+<a class="l3-item" data-target="e258" onclick="showEntry('e258')"><span class="mb badge-c">C</span>WhisperWarp</a>
+<a class="l3-item l3m" data-target="e259" onclick="showEntry('e259')"><span class="mb badge-m">M</span>.forward()</a>
+<a class="l3-item l3m" data-target="e260" onclick="showEntry('e260')"><span class="mb badge-m">M</span>.inference()</a>
+</div></div>
+<div class="l2"><div class="l2-title" onclick="toggleL2(this)"><span>whisper_lid</span><span class="arr">▸</span></div><div class="l2-children">
+<a class="l3-item" data-target="e261" onclick="showEntry('e261')"><span class="mb badge-c">C</span>OpenAIWhisperModel</a>
+<a class="l3-item l3m" data-target="e262" onclick="showEntry('e262')"><span class="mb badge-m">M</span>.forward()</a>
+<a class="l3-item l3m" data-target="e263" onclick="showEntry('e263')"><span class="mb badge-m">M</span>.encode()</a>
+<a class="l3-item l3m" data-target="e264" onclick="showEntry('e264')"><span class="mb badge-m">M</span>.init_beam_search()</a>
+<a class="l3-item l3m" data-target="e265" onclick="showEntry('e265')"><span class="mb badge-m">M</span>.inference()</a>
+<a class="l3-item" data-target="e266" onclick="showEntry('e266')"><span class="mb badge-c">C</span>OpenAIWhisperLIDModel</a>
+<a class="l3-item l3m" data-target="e267" onclick="showEntry('e267')"><span class="mb badge-m">M</span>.forward()</a>
+<a class="l3-item l3m" data-target="e268" onclick="showEntry('e268')"><span class="mb badge-m">M</span>.encode()</a>
+<a class="l3-item l3m" data-target="e269" onclick="showEntry('e269')"><span class="mb badge-m">M</span>.inference()</a>
+</div></div>
+</div></div>
+<div class="l1"><div class="l1-title" onclick="toggleL1(this)"><span>funasr.frontends</span><span class="cnt">7</span><span class="arr">▸</span></div><div class="l1-children">
+<div class="l2"><div class="l2-title" onclick="toggleL2(this)"><span>default</span><span class="arr">▸</span></div><div class="l2-children">
+<a class="l3-item" data-target="e270" onclick="showEntry('e270')"><span class="mb badge-c">C</span>DefaultFrontend</a>
+<a class="l3-item l3m" data-target="e271" onclick="showEntry('e271')"><span class="mb badge-m">M</span>.output_size()</a>
+<a class="l3-item l3m" data-target="e272" onclick="showEntry('e272')"><span class="mb badge-m">M</span>.forward()</a>
+<a class="l3-item" data-target="e273" onclick="showEntry('e273')"><span class="mb badge-c">C</span>MultiChannelFrontend</a>
+<a class="l3-item l3m" data-target="e274" onclick="showEntry('e274')"><span class="mb badge-m">M</span>.output_size()</a>
+<a class="l3-item l3m" data-target="e275" onclick="showEntry('e275')"><span class="mb badge-m">M</span>.forward()</a>
+</div></div>
+<div class="l2"><div class="l2-title" onclick="toggleL2(this)"><span>eend_ola_feature</span><span class="arr">▸</span></div><div class="l2-children">
+<a class="l3-item" data-target="e276" onclick="showEntry('e276')"><span class="mb badge-f">F</span>transform</a>
+<a class="l3-item" data-target="e277" onclick="showEntry('e277')"><span class="mb badge-f">F</span>subsample</a>
+<a class="l3-item" data-target="e278" onclick="showEntry('e278')"><span class="mb badge-f">F</span>splice</a>
+<a class="l3-item" data-target="e279" onclick="showEntry('e279')"><span class="mb badge-f">F</span>stft</a>
+</div></div>
+<div class="l2"><div class="l2-title" onclick="toggleL2(this)"><span>fused</span><span class="arr">▸</span></div><div class="l2-children">
+<a class="l3-item" data-target="e280" onclick="showEntry('e280')"><span class="mb badge-c">C</span>FusedFrontends</a>
+<a class="l3-item l3m" data-target="e281" onclick="showEntry('e281')"><span class="mb badge-m">M</span>.output_size()</a>
+<a class="l3-item l3m" data-target="e282" onclick="showEntry('e282')"><span class="mb badge-m">M</span>.forward()</a>
+</div></div>
+<div class="l2"><div class="l2-title" onclick="toggleL2(this)"><span>s3prl</span><span class="arr">▸</span></div><div class="l2-children">
+<a class="l3-item" data-target="e283" onclick="showEntry('e283')"><span class="mb badge-f">F</span>base_s3prl_setup</a>
+<a class="l3-item" data-target="e284" onclick="showEntry('e284')"><span class="mb badge-c">C</span>S3prlFrontend</a>
+<a class="l3-item l3m" data-target="e285" onclick="showEntry('e285')"><span class="mb badge-m">M</span>.output_size()</a>
+<a class="l3-item l3m" data-target="e286" onclick="showEntry('e286')"><span class="mb badge-m">M</span>.forward()</a>
+<a class="l3-item l3m" data-target="e287" onclick="showEntry('e287')"><span class="mb badge-m">M</span>.reload_pretrained_parameters()</a>
+</div></div>
+<div class="l2"><div class="l2-title" onclick="toggleL2(this)"><span>wav_frontend</span><span class="arr">▸</span></div><div class="l2-children">
+<a class="l3-item" data-target="e288" onclick="showEntry('e288')"><span class="mb badge-f">F</span>load_cmvn</a>
+<a class="l3-item" data-target="e289" onclick="showEntry('e289')"><span class="mb badge-f">F</span>apply_cmvn</a>
+<a class="l3-item" data-target="e290" onclick="showEntry('e290')"><span class="mb badge-f">F</span>apply_lfr</a>
+<a class="l3-item" data-target="e291" onclick="showEntry('e291')"><span class="mb badge-c">C</span>WavFrontend</a>
+<a class="l3-item l3m" data-target="e292" onclick="showEntry('e292')"><span class="mb badge-m">M</span>.output_size()</a>
+<a class="l3-item l3m" data-target="e293" onclick="showEntry('e293')"><span class="mb badge-m">M</span>.forward()</a>
+<a class="l3-item l3m" data-target="e294" onclick="showEntry('e294')"><span class="mb badge-m">M</span>.forward_fbank()</a>
+<a class="l3-item l3m" data-target="e295" onclick="showEntry('e295')"><span class="mb badge-m">M</span>.forward_lfr_cmvn()</a>
+<a class="l3-item" data-target="e296" onclick="showEntry('e296')"><span class="mb badge-c">C</span>WavFrontendOnline</a>
+<a class="l3-item l3m" data-target="e297" onclick="showEntry('e297')"><span class="mb badge-m">M</span>.output_size()</a>
+<a class="l3-item l3m" data-target="e298" onclick="showEntry('e298')"><span class="mb badge-m">M</span>.apply_cmvn()</a>
+<a class="l3-item l3m" data-target="e299" onclick="showEntry('e299')"><span class="mb badge-m">M</span>.apply_lfr()</a>
+<a class="l3-item l3m" data-target="e300" onclick="showEntry('e300')"><span class="mb badge-m">M</span>.compute_frame_num()</a>
+<a class="l3-item l3m" data-target="e301" onclick="showEntry('e301')"><span class="mb badge-m">M</span>.forward_fbank()</a>
+<a class="l3-item l3m" data-target="e302" onclick="showEntry('e302')"><span class="mb badge-m">M</span>.forward_lfr_cmvn()</a>
+<a class="l3-item l3m" data-target="e303" onclick="showEntry('e303')"><span class="mb badge-m">M</span>.forward()</a>
+<a class="l3-item l3m" data-target="e304" onclick="showEntry('e304')"><span class="mb badge-m">M</span>.init_cache()</a>
+<a class="l3-item" data-target="e305" onclick="showEntry('e305')"><span class="mb badge-c">C</span>WavFrontendMel23</a>
+<a class="l3-item l3m" data-target="e306" onclick="showEntry('e306')"><span class="mb badge-m">M</span>.output_size()</a>
+<a class="l3-item l3m" data-target="e307" onclick="showEntry('e307')"><span class="mb badge-m">M</span>.forward()</a>
+</div></div>
+<div class="l2"><div class="l2-title" onclick="toggleL2(this)"><span>whisper_frontend</span><span class="arr">▸</span></div><div class="l2-children">
+<a class="l3-item" data-target="e308" onclick="showEntry('e308')"><span class="mb badge-c">C</span>WhisperFrontend</a>
+<a class="l3-item l3m" data-target="e309" onclick="showEntry('e309')"><span class="mb badge-m">M</span>.output_size()</a>
+<a class="l3-item l3m" data-target="e310" onclick="showEntry('e310')"><span class="mb badge-m">M</span>.log_mel_spectrogram()</a>
+<a class="l3-item l3m" data-target="e311" onclick="showEntry('e311')"><span class="mb badge-m">M</span>.forward()</a>
+</div></div>
+<div class="l2"><div class="l2-title" onclick="toggleL2(this)"><span>windowing</span><span class="arr">▸</span></div><div class="l2-children">
+<a class="l3-item" data-target="e312" onclick="showEntry('e312')"><span class="mb badge-c">C</span>SlidingWindow</a>
+<a class="l3-item l3m" data-target="e313" onclick="showEntry('e313')"><span class="mb badge-m">M</span>.forward()</a>
+<a class="l3-item l3m" data-target="e314" onclick="showEntry('e314')"><span class="mb badge-m">M</span>.output_size()</a>
+</div></div>
+</div></div>
+<div class="l1"><div class="l1-title" onclick="toggleL1(this)"><span>funasr.tokenizer</span><span class="cnt">10</span><span class="arr">▸</span></div><div class="l1-children">
+<div class="l2"><div class="l2-title" onclick="toggleL2(this)"><span>build_tokenizer</span><span class="arr">▸</span></div><div class="l2-children">
+<a class="l3-item" data-target="e315" onclick="showEntry('e315')"><span class="mb badge-f">F</span>build_tokenizer</a>
+</div></div>
+<div class="l2"><div class="l2-title" onclick="toggleL2(this)"><span>char_tokenizer</span><span class="arr">▸</span></div><div class="l2-children">
+<a class="l3-item" data-target="e316" onclick="showEntry('e316')"><span class="mb badge-c">C</span>CharTokenizer</a>
+<a class="l3-item l3m" data-target="e317" onclick="showEntry('e317')"><span class="mb badge-m">M</span>.text2tokens()</a>
+<a class="l3-item l3m" data-target="e318" onclick="showEntry('e318')"><span class="mb badge-m">M</span>.tokens2text()</a>
+<a class="l3-item" data-target="e319" onclick="showEntry('e319')"><span class="mb badge-f">F</span>load_seg_dict</a>
+<a class="l3-item" data-target="e320" onclick="showEntry('e320')"><span class="mb badge-f">F</span>seg_tokenize</a>
+</div></div>
+<div class="l2"><div class="l2-title" onclick="toggleL2(this)"><span>cleaner</span><span class="arr">▸</span></div><div class="l2-children">
+<a class="l3-item" data-target="e321" onclick="showEntry('e321')"><span class="mb badge-c">C</span>TextCleaner</a>
+</div></div>
+<div class="l2"><div class="l2-title" onclick="toggleL2(this)"><span>hf_tokenizer</span><span class="arr">▸</span></div><div class="l2-children">
+<a class="l3-item" data-target="e322" onclick="showEntry('e322')"><span class="mb badge-f">F</span>HuggingfaceTokenizer</a>
+</div></div>
+<div class="l2"><div class="l2-title" onclick="toggleL2(this)"><span>korean_cleaner</span><span class="arr">▸</span></div><div class="l2-children">
+<a class="l3-item" data-target="e323" onclick="showEntry('e323')"><span class="mb badge-c">C</span>KoreanCleaner</a>
+<a class="l3-item l3m" data-target="e324" onclick="showEntry('e324')"><span class="mb badge-m">M</span>.normalize_text()</a>
+</div></div>
+<div class="l2"><div class="l2-title" onclick="toggleL2(this)"><span>phoneme_tokenizer</span><span class="arr">▸</span></div><div class="l2-children">
+<a class="l3-item" data-target="e325" onclick="showEntry('e325')"><span class="mb badge-f">F</span>split_by_space</a>
+<a class="l3-item" data-target="e326" onclick="showEntry('e326')"><span class="mb badge-f">F</span>pyopenjtalk_g2p</a>
+<a class="l3-item" data-target="e327" onclick="showEntry('e327')"><span class="mb badge-f">F</span>pyopenjtalk_g2p_accent</a>
+<a class="l3-item" data-target="e328" onclick="showEntry('e328')"><span class="mb badge-f">F</span>pyopenjtalk_g2p_accent_with_pause</a>
+<a class="l3-item" data-target="e329" onclick="showEntry('e329')"><span class="mb badge-f">F</span>pyopenjtalk_g2p_kana</a>
+<a class="l3-item" data-target="e330" onclick="showEntry('e330')"><span class="mb badge-f">F</span>pyopenjtalk_g2p_prosody</a>
+<a class="l3-item" data-target="e331" onclick="showEntry('e331')"><span class="mb badge-f">F</span>pypinyin_g2p</a>
+<a class="l3-item" data-target="e332" onclick="showEntry('e332')"><span class="mb badge-f">F</span>pypinyin_g2p_phone</a>
+<a class="l3-item" data-target="e333" onclick="showEntry('e333')"><span class="mb badge-c">C</span>G2p_en</a>
+<a class="l3-item" data-target="e334" onclick="showEntry('e334')"><span class="mb badge-c">C</span>G2pk</a>
+<a class="l3-item" data-target="e335" onclick="showEntry('e335')"><span class="mb badge-c">C</span>Jaso</a>
+<a class="l3-item" data-target="e336" onclick="showEntry('e336')"><span class="mb badge-c">C</span>Phonemizer</a>
+<a class="l3-item" data-target="e337" onclick="showEntry('e337')"><span class="mb badge-c">C</span>PhonemeTokenizer</a>
+<a class="l3-item l3m" data-target="e338" onclick="showEntry('e338')"><span class="mb badge-m">M</span>.text2tokens()</a>
+<a class="l3-item l3m" data-target="e339" onclick="showEntry('e339')"><span class="mb badge-m">M</span>.tokens2text()</a>
+</div></div>
+<div class="l2"><div class="l2-title" onclick="toggleL2(this)"><span>sentencepiece_tokenizer</span><span class="arr">▸</span></div><div class="l2-children">
+<a class="l3-item" data-target="e340" onclick="showEntry('e340')"><span class="mb badge-c">C</span>SentencepiecesTokenizer</a>
+<a class="l3-item l3m" data-target="e341" onclick="showEntry('e341')"><span class="mb badge-m">M</span>.text2tokens()</a>
+<a class="l3-item l3m" data-target="e342" onclick="showEntry('e342')"><span class="mb badge-m">M</span>.tokens2text()</a>
+<a class="l3-item l3m" data-target="e343" onclick="showEntry('e343')"><span class="mb badge-m">M</span>.encode()</a>
+<a class="l3-item l3m" data-target="e344" onclick="showEntry('e344')"><span class="mb badge-m">M</span>.decode()</a>
+<a class="l3-item l3m" data-target="e345" onclick="showEntry('e345')"><span class="mb badge-m">M</span>.get_vocab_size()</a>
+<a class="l3-item l3m" data-target="e346" onclick="showEntry('e346')"><span class="mb badge-m">M</span>.ids2tokens()</a>
+<a class="l3-item l3m" data-target="e347" onclick="showEntry('e347')"><span class="mb badge-m">M</span>.tokens2ids()</a>
+</div></div>
+<div class="l2"><div class="l2-title" onclick="toggleL2(this)"><span>token_id_converter</span><span class="arr">▸</span></div><div class="l2-children">
+<a class="l3-item" data-target="e348" onclick="showEntry('e348')"><span class="mb badge-c">C</span>TokenIDConverter</a>
+<a class="l3-item l3m" data-target="e349" onclick="showEntry('e349')"><span class="mb badge-m">M</span>.get_num_vocabulary_size()</a>
+<a class="l3-item l3m" data-target="e350" onclick="showEntry('e350')"><span class="mb badge-m">M</span>.ids2tokens()</a>
+<a class="l3-item l3m" data-target="e351" onclick="showEntry('e351')"><span class="mb badge-m">M</span>.tokens2ids()</a>
+</div></div>
+<div class="l2"><div class="l2-title" onclick="toggleL2(this)"><span>whisper_tokenizer</span><span class="arr">▸</span></div><div class="l2-children">
+<a class="l3-item" data-target="e352" onclick="showEntry('e352')"><span class="mb badge-f">F</span>WhisperTokenizer</a>
+<a class="l3-item" data-target="e353" onclick="showEntry('e353')"><span class="mb badge-f">F</span>SenseVoiceTokenizer</a>
+</div></div>
+<div class="l2"><div class="l2-title" onclick="toggleL2(this)"><span>word_tokenizer</span><span class="arr">▸</span></div><div class="l2-children">
+<a class="l3-item" data-target="e354" onclick="showEntry('e354')"><span class="mb badge-c">C</span>WordTokenizer</a>
+<a class="l3-item l3m" data-target="e355" onclick="showEntry('e355')"><span class="mb badge-m">M</span>.text2tokens()</a>
+<a class="l3-item l3m" data-target="e356" onclick="showEntry('e356')"><span class="mb badge-m">M</span>.tokens2text()</a>
+</div></div>
+</div></div>
+<div class="l1"><div class="l1-title" onclick="toggleL1(this)"><span>funasr.utils</span><span class="cnt">15</span><span class="arr">▸</span></div><div class="l1-children">
+<div class="l2"><div class="l2-title" onclick="toggleL2(this)"><span>compute_det_ctc</span><span class="arr">▸</span></div><div class="l2-children">
+<a class="l3-item" data-target="e357" onclick="showEntry('e357')"><span class="mb badge-c">C</span>thread_wrapper</a>
+<a class="l3-item l3m" data-target="e358" onclick="showEntry('e358')"><span class="mb badge-m">M</span>.run()</a>
+<a class="l3-item l3m" data-target="e359" onclick="showEntry('e359')"><span class="mb badge-m">M</span>.get_result()</a>
+<a class="l3-item" data-target="e360" onclick="showEntry('e360')"><span class="mb badge-f">F</span>space_mixed_label</a>
+<a class="l3-item" data-target="e361" onclick="showEntry('e361')"><span class="mb badge-f">F</span>read_lists</a>
+<a class="l3-item" data-target="e362" onclick="showEntry('e362')"><span class="mb badge-f">F</span>make_pair</a>
+<a class="l3-item" data-target="e363" onclick="showEntry('e363')"><span class="mb badge-f">F</span>count_duration</a>
+<a class="l3-item" data-target="e364" onclick="showEntry('e364')"><span class="mb badge-f">F</span>load_data_and_score</a>
+</div></div>
+<div class="l2"><div class="l2-title" onclick="toggleL2(this)"><span>datadir_writer</span><span class="arr">▸</span></div><div class="l2-children">
+<a class="l3-item" data-target="e365" onclick="showEntry('e365')"><span class="mb badge-c">C</span>DatadirWriter</a>
+<a class="l3-item l3m" data-target="e366" onclick="showEntry('e366')"><span class="mb badge-m">M</span>.close()</a>
+</div></div>
+<div class="l2"><div class="l2-title" onclick="toggleL2(this)"><span>dynamic_import</span><span class="arr">▸</span></div><div class="l2-children">
+<a class="l3-item" data-target="e367" onclick="showEntry('e367')"><span class="mb badge-f">F</span>load_module_from_path</a>
+<a class="l3-item" data-target="e368" onclick="showEntry('e368')"><span class="mb badge-f">F</span>import_module_from_path</a>
+</div></div>
+<div class="l2"><div class="l2-title" onclick="toggleL2(this)"><span>export_utils</span><span class="arr">▸</span></div><div class="l2-children">
+<a class="l3-item" data-target="e369" onclick="showEntry('e369')"><span class="mb badge-f">F</span>export</a>
+</div></div>
+<div class="l2"><div class="l2-title" onclick="toggleL2(this)"><span>install_model_requirements</span><span class="arr">▸</span></div><div class="l2-children">
+<a class="l3-item" data-target="e370" onclick="showEntry('e370')"><span class="mb badge-f">F</span>install_requirements</a>
+<a class="l3-item" data-target="e371" onclick="showEntry('e371')"><span class="mb badge-f">F</span>pip_install_r</a>
+</div></div>
+<div class="l2"><div class="l2-title" onclick="toggleL2(this)"><span>kws_utils</span><span class="arr">▸</span></div><div class="l2-children">
+<a class="l3-item" data-target="e372" onclick="showEntry('e372')"><span class="mb badge-f">F</span>split_mixed_label</a>
+<a class="l3-item" data-target="e373" onclick="showEntry('e373')"><span class="mb badge-f">F</span>query_token_set</a>
+<a class="l3-item" data-target="e374" onclick="showEntry('e374')"><span class="mb badge-c">C</span>KwsCtcPrefixDecoder</a>
+<a class="l3-item l3m" data-target="e375" onclick="showEntry('e375')"><span class="mb badge-m">M</span>.beam_search()</a>
+<a class="l3-item l3m" data-target="e376" onclick="showEntry('e376')"><span class="mb badge-m">M</span>.is_sublist()</a>
+<a class="l3-item l3m" data-target="e377" onclick="showEntry('e377')"><span class="mb badge-m">M</span>.decode()</a>
+</div></div>
+<div class="l2"><div class="l2-title" onclick="toggleL2(this)"><span>load_utils</span><span class="arr">▸</span></div><div class="l2-children">
+<a class="l3-item" data-target="e378" onclick="showEntry('e378')"><span class="mb badge-f">F</span>is_ffmpeg_installed</a>
+<a class="l3-item" data-target="e379" onclick="showEntry('e379')"><span class="mb badge-f">F</span>load_audio_text_image_video</a>
+<a class="l3-item" data-target="e380" onclick="showEntry('e380')"><span class="mb badge-f">F</span>load_bytes</a>
+<a class="l3-item" data-target="e381" onclick="showEntry('e381')"><span class="mb badge-f">F</span>validate_frame_rate</a>
+<a class="l3-item" data-target="e382" onclick="showEntry('e382')"><span class="mb badge-f">F</span>extract_fbank</a>
+</div></div>
+<div class="l2"><div class="l2-title" onclick="toggleL2(this)"><span>misc</span><span class="arr">▸</span></div><div class="l2-children">
+<a class="l3-item" data-target="e383" onclick="showEntry('e383')"><span class="mb badge-f">F</span>statistic_model_parameters</a>
+<a class="l3-item" data-target="e384" onclick="showEntry('e384')"><span class="mb badge-f">F</span>int2vec</a>
+<a class="l3-item" data-target="e385" onclick="showEntry('e385')"><span class="mb badge-f">F</span>seq2arr</a>
+<a class="l3-item" data-target="e386" onclick="showEntry('e386')"><span class="mb badge-f">F</span>load_scp_as_dict</a>
+<a class="l3-item" data-target="e387" onclick="showEntry('e387')"><span class="mb badge-f">F</span>load_scp_as_list</a>
+<a class="l3-item" data-target="e388" onclick="showEntry('e388')"><span class="mb badge-f">F</span>deep_update</a>
+<a class="l3-item" data-target="e389" onclick="showEntry('e389')"><span class="mb badge-f">F</span>prepare_model_dir</a>
+<a class="l3-item" data-target="e390" onclick="showEntry('e390')"><span class="mb badge-f">F</span>extract_filename_without_extension</a>
+<a class="l3-item" data-target="e391" onclick="showEntry('e391')"><span class="mb badge-f">F</span>smart_remove</a>
+</div></div>
+<div class="l2"><div class="l2-title" onclick="toggleL2(this)"><span>postprocess_utils</span><span class="arr">▸</span></div><div class="l2-children">
+<a class="l3-item" data-target="e392" onclick="showEntry('e392')"><span class="mb badge-f">F</span>isChinese</a>
+<a class="l3-item" data-target="e393" onclick="showEntry('e393')"><span class="mb badge-f">F</span>isAllChinese</a>
+<a class="l3-item" data-target="e394" onclick="showEntry('e394')"><span class="mb badge-f">F</span>isAllAlpha</a>
+<a class="l3-item" data-target="e395" onclick="showEntry('e395')"><span class="mb badge-f">F</span>abbr_dispose</a>
+<a class="l3-item" data-target="e396" onclick="showEntry('e396')"><span class="mb badge-f">F</span>sentence_postprocess</a>
+<a class="l3-item" data-target="e397" onclick="showEntry('e397')"><span class="mb badge-f">F</span>sentence_postprocess_sentencepiece</a>
+<a class="l3-item" data-target="e398" onclick="showEntry('e398')"><span class="mb badge-f">F</span>format_str_v2</a>
+<a class="l3-item" data-target="e399" onclick="showEntry('e399')"><span class="mb badge-f">F</span>rich_transcription_postprocess</a>
+</div></div>
+<div class="l2"><div class="l2-title" onclick="toggleL2(this)"><span>speaker_utils</span><span class="arr">▸</span></div><div class="l2-children">
+<a class="l3-item" data-target="e400" onclick="showEntry('e400')"><span class="mb badge-f">F</span>check_audio_list</a>
+<a class="l3-item" data-target="e401" onclick="showEntry('e401')"><span class="mb badge-f">F</span>sv_preprocess</a>
+<a class="l3-item" data-target="e402" onclick="showEntry('e402')"><span class="mb badge-f">F</span>sv_chunk</a>
+<a class="l3-item" data-target="e403" onclick="showEntry('e403')"><span class="mb badge-f">F</span>extract_feature</a>
+<a class="l3-item" data-target="e404" onclick="showEntry('e404')"><span class="mb badge-f">F</span>postprocess</a>
+<a class="l3-item" data-target="e405" onclick="showEntry('e405')"><span class="mb badge-f">F</span>correct_labels</a>
+<a class="l3-item" data-target="e406" onclick="showEntry('e406')"><span class="mb badge-f">F</span>merge_seque</a>
+<a class="l3-item" data-target="e407" onclick="showEntry('e407')"><span class="mb badge-f">F</span>smooth</a>
+<a class="l3-item" data-target="e408" onclick="showEntry('e408')"><span class="mb badge-f">F</span>distribute_spk</a>
+</div></div>
+<div class="l2"><div class="l2-title" onclick="toggleL2(this)"><span>timestamp_tools</span><span class="arr">▸</span></div><div class="l2-children">
+<a class="l3-item" data-target="e409" onclick="showEntry('e409')"><span class="mb badge-f">F</span>cif_wo_hidden</a>
+<a class="l3-item" data-target="e410" onclick="showEntry('e410')"><span class="mb badge-f">F</span>ts_prediction_lfr6_standard</a>
+<a class="l3-item" data-target="e411" onclick="showEntry('e411')"><span class="mb badge-f">F</span>timestamp_sentence</a>
+<a class="l3-item" data-target="e412" onclick="showEntry('e412')"><span class="mb badge-f">F</span>timestamp_sentence_en</a>
+</div></div>
+<div class="l2"><div class="l2-title" onclick="toggleL2(this)"><span>torch_function</span><span class="arr">▸</span></div><div class="l2-children">
+<a class="l3-item" data-target="e413" onclick="showEntry('e413')"><span class="mb badge-c">C</span>MakePadMask</a>
+<a class="l3-item l3m" data-target="e414" onclick="showEntry('e414')"><span class="mb badge-m">M</span>.forward()</a>
+<a class="l3-item" data-target="e415" onclick="showEntry('e415')"><span class="mb badge-c">C</span>sequence_mask</a>
+<a class="l3-item l3m" data-target="e416" onclick="showEntry('e416')"><span class="mb badge-m">M</span>.forward()</a>
+<a class="l3-item" data-target="e417" onclick="showEntry('e417')"><span class="mb badge-f">F</span>normalize</a>
+<a class="l3-item" data-target="e418" onclick="showEntry('e418')"><span class="mb badge-f">F</span>subsequent_mask</a>
+<a class="l3-item" data-target="e419" onclick="showEntry('e419')"><span class="mb badge-f">F</span>MakePadMask_test</a>
+</div></div>
+<div class="l2"><div class="l2-title" onclick="toggleL2(this)"><span>type_utils</span><span class="arr">▸</span></div><div class="l2-children">
+<a class="l3-item" data-target="e420" onclick="showEntry('e420')"><span class="mb badge-f">F</span>str2bool</a>
+<a class="l3-item" data-target="e421" onclick="showEntry('e421')"><span class="mb badge-f">F</span>remove_parenthesis</a>
+<a class="l3-item" data-target="e422" onclick="showEntry('e422')"><span class="mb badge-f">F</span>remove_quotes</a>
+<a class="l3-item" data-target="e423" onclick="showEntry('e423')"><span class="mb badge-f">F</span>int_or_none</a>
+<a class="l3-item" data-target="e424" onclick="showEntry('e424')"><span class="mb badge-f">F</span>float_or_none</a>
+<a class="l3-item" data-target="e425" onclick="showEntry('e425')"><span class="mb badge-f">F</span>humanfriendly_parse_size_or_none</a>
+<a class="l3-item" data-target="e426" onclick="showEntry('e426')"><span class="mb badge-f">F</span>str_or_int</a>
+<a class="l3-item" data-target="e427" onclick="showEntry('e427')"><span class="mb badge-f">F</span>str_or_none</a>
+<a class="l3-item" data-target="e428" onclick="showEntry('e428')"><span class="mb badge-f">F</span>str2pair_str</a>
+<a class="l3-item" data-target="e429" onclick="showEntry('e429')"><span class="mb badge-f">F</span>str2triple_str</a>
+</div></div>
+<div class="l2"><div class="l2-title" onclick="toggleL2(this)"><span>vad_utils</span><span class="arr">▸</span></div><div class="l2-children">
+<a class="l3-item" data-target="e430" onclick="showEntry('e430')"><span class="mb badge-f">F</span>slice_padding_fbank</a>
+<a class="l3-item" data-target="e431" onclick="showEntry('e431')"><span class="mb badge-f">F</span>slice_padding_audio_samples</a>
+<a class="l3-item" data-target="e432" onclick="showEntry('e432')"><span class="mb badge-f">F</span>merge_vad</a>
+</div></div>
+<div class="l2"><div class="l2-title" onclick="toggleL2(this)"><span>version_checker</span><span class="arr">▸</span></div><div class="l2-children">
+<a class="l3-item" data-target="e433" onclick="showEntry('e433')"><span class="mb badge-f">F</span>get_pypi_version</a>
+<a class="l3-item" data-target="e434" onclick="showEntry('e434')"><span class="mb badge-f">F</span>check_for_update</a>
+</div></div>
+</div></div>
+<div class="l1"><div class="l1-title" onclick="toggleL1(this)"><span>funasr.bin</span><span class="cnt">6</span><span class="arr">▸</span></div><div class="l1-children">
+<div class="l2"><div class="l2-title" onclick="toggleL2(this)"><span>compute_audio_cmvn</span><span class="arr">▸</span></div><div class="l2-children">
+<a class="l3-item" data-target="e435" onclick="showEntry('e435')"><span class="mb badge-f">F</span>main_hydra</a>
+<a class="l3-item" data-target="e436" onclick="showEntry('e436')"><span class="mb badge-f">F</span>main</a>
+</div></div>
+<div class="l2"><div class="l2-title" onclick="toggleL2(this)"><span>export</span><span class="arr">▸</span></div><div class="l2-children">
+<a class="l3-item" data-target="e437" onclick="showEntry('e437')"><span class="mb badge-f">F</span>main_hydra</a>
+</div></div>
+<div class="l2"><div class="l2-title" onclick="toggleL2(this)"><span>inference</span><span class="arr">▸</span></div><div class="l2-children">
+<a class="l3-item" data-target="e438" onclick="showEntry('e438')"><span class="mb badge-f">F</span>main_hydra</a>
+</div></div>
+<div class="l2"><div class="l2-title" onclick="toggleL2(this)"><span>tokenize_text</span><span class="arr">▸</span></div><div class="l2-children">
+<a class="l3-item" data-target="e439" onclick="showEntry('e439')"><span class="mb badge-f">F</span>field2slice</a>
+<a class="l3-item" data-target="e440" onclick="showEntry('e440')"><span class="mb badge-f">F</span>tokenize</a>
+<a class="l3-item" data-target="e441" onclick="showEntry('e441')"><span class="mb badge-f">F</span>get_parser</a>
+<a class="l3-item" data-target="e442" onclick="showEntry('e442')"><span class="mb badge-f">F</span>main</a>
+</div></div>
+<div class="l2"><div class="l2-title" onclick="toggleL2(this)"><span>train</span><span class="arr">▸</span></div><div class="l2-children">
+<a class="l3-item" data-target="e443" onclick="showEntry('e443')"><span class="mb badge-f">F</span>main_hydra</a>
+<a class="l3-item" data-target="e444" onclick="showEntry('e444')"><span class="mb badge-f">F</span>main</a>
+</div></div>
+<div class="l2"><div class="l2-title" onclick="toggleL2(this)"><span>train_ds</span><span class="arr">▸</span></div><div class="l2-children">
+<a class="l3-item" data-target="e445" onclick="showEntry('e445')"><span class="mb badge-f">F</span>main_hydra</a>
+<a class="l3-item" data-target="e446" onclick="showEntry('e446')"><span class="mb badge-f">F</span>main</a>
+</div></div>
+</div></div>
+<div class="l1"><div class="l1-title" onclick="toggleL1(this)"><span>funasr.download</span><span class="cnt">4</span><span class="arr">▸</span></div><div class="l1-children">
+<div class="l2"><div class="l2-title" onclick="toggleL2(this)"><span>download_dataset_from_hub</span><span class="arr">▸</span></div><div class="l2-children">
+<a class="l3-item" data-target="e447" onclick="showEntry('e447')"><span class="mb badge-f">F</span>download_dataset</a>
+<a class="l3-item" data-target="e448" onclick="showEntry('e448')"><span class="mb badge-f">F</span>download_dataset_from_ms</a>
+</div></div>
+<div class="l2"><div class="l2-title" onclick="toggleL2(this)"><span>download_model_from_hub</span><span class="arr">▸</span></div><div class="l2-children">
+<a class="l3-item" data-target="e449" onclick="showEntry('e449')"><span class="mb badge-f">F</span>download_model</a>
+<a class="l3-item" data-target="e450" onclick="showEntry('e450')"><span class="mb badge-f">F</span>download_from_ms</a>
+<a class="l3-item" data-target="e451" onclick="showEntry('e451')"><span class="mb badge-f">F</span>download_from_hf</a>
+<a class="l3-item" data-target="e452" onclick="showEntry('e452')"><span class="mb badge-f">F</span>add_file_root_path</a>
+<a class="l3-item" data-target="e453" onclick="showEntry('e453')"><span class="mb badge-f">F</span>get_or_download_model_dir</a>
+<a class="l3-item" data-target="e454" onclick="showEntry('e454')"><span class="mb badge-f">F</span>get_or_download_model_dir_hf</a>
+</div></div>
+<div class="l2"><div class="l2-title" onclick="toggleL2(this)"><span>file</span><span class="arr">▸</span></div><div class="l2-children">
+<a class="l3-item" data-target="e455" onclick="showEntry('e455')"><span class="mb badge-f">F</span>download_from_url</a>
+<a class="l3-item" data-target="e456" onclick="showEntry('e456')"><span class="mb badge-c">C</span>Storage</a>
+<a class="l3-item l3m" data-target="e457" onclick="showEntry('e457')"><span class="mb badge-m">M</span>.read()</a>
+<a class="l3-item l3m" data-target="e458" onclick="showEntry('e458')"><span class="mb badge-m">M</span>.read_text()</a>
+<a class="l3-item l3m" data-target="e459" onclick="showEntry('e459')"><span class="mb badge-m">M</span>.write()</a>
+<a class="l3-item l3m" data-target="e460" onclick="showEntry('e460')"><span class="mb badge-m">M</span>.write_text()</a>
+<a class="l3-item" data-target="e461" onclick="showEntry('e461')"><span class="mb badge-c">C</span>LocalStorage</a>
+<a class="l3-item l3m" data-target="e462" onclick="showEntry('e462')"><span class="mb badge-m">M</span>.read()</a>
+<a class="l3-item l3m" data-target="e463" onclick="showEntry('e463')"><span class="mb badge-m">M</span>.read_text()</a>
+<a class="l3-item l3m" data-target="e464" onclick="showEntry('e464')"><span class="mb badge-m">M</span>.write()</a>
+<a class="l3-item l3m" data-target="e465" onclick="showEntry('e465')"><span class="mb badge-m">M</span>.write_text()</a>
+<a class="l3-item l3m" data-target="e466" onclick="showEntry('e466')"><span class="mb badge-m">M</span>.as_local_path()</a>
+<a class="l3-item" data-target="e467" onclick="showEntry('e467')"><span class="mb badge-c">C</span>HTTPStorage</a>
+<a class="l3-item l3m" data-target="e468" onclick="showEntry('e468')"><span class="mb badge-m">M</span>.read()</a>
+<a class="l3-item l3m" data-target="e469" onclick="showEntry('e469')"><span class="mb badge-m">M</span>.read_text()</a>
+<a class="l3-item l3m" data-target="e470" onclick="showEntry('e470')"><span class="mb badge-m">M</span>.as_local_path()</a>
+<a class="l3-item l3m" data-target="e471" onclick="showEntry('e471')"><span class="mb badge-m">M</span>.write()</a>
+<a class="l3-item l3m" data-target="e472" onclick="showEntry('e472')"><span class="mb badge-m">M</span>.write_text()</a>
+<a class="l3-item" data-target="e473" onclick="showEntry('e473')"><span class="mb badge-c">C</span>OSSStorage</a>
+<a class="l3-item l3m" data-target="e474" onclick="showEntry('e474')"><span class="mb badge-m">M</span>.read()</a>
+<a class="l3-item l3m" data-target="e475" onclick="showEntry('e475')"><span class="mb badge-m">M</span>.read_text()</a>
+<a class="l3-item l3m" data-target="e476" onclick="showEntry('e476')"><span class="mb badge-m">M</span>.as_local_path()</a>
+<a class="l3-item l3m" data-target="e477" onclick="showEntry('e477')"><span class="mb badge-m">M</span>.write()</a>
+<a class="l3-item l3m" data-target="e478" onclick="showEntry('e478')"><span class="mb badge-m">M</span>.write_text()</a>
+<a class="l3-item" data-target="e479" onclick="showEntry('e479')"><span class="mb badge-c">C</span>File</a>
+<a class="l3-item l3m" data-target="e480" onclick="showEntry('e480')"><span class="mb badge-m">M</span>.read()</a>
+<a class="l3-item l3m" data-target="e481" onclick="showEntry('e481')"><span class="mb badge-m">M</span>.read_text()</a>
+<a class="l3-item l3m" data-target="e482" onclick="showEntry('e482')"><span class="mb badge-m">M</span>.write()</a>
+<a class="l3-item l3m" data-target="e483" onclick="showEntry('e483')"><span class="mb badge-m">M</span>.write_text()</a>
+<a class="l3-item l3m" data-target="e484" onclick="showEntry('e484')"><span class="mb badge-m">M</span>.as_local_path()</a>
+</div></div>
+<div class="l2"><div class="l2-title" onclick="toggleL2(this)"><span>runtime_sdk_download_tool</span><span class="arr">▸</span></div><div class="l2-children">
+<a class="l3-item" data-target="e485" onclick="showEntry('e485')"><span class="mb badge-f">F</span>main</a>
+</div></div>
+</div></div>
+<div class="l1"><div class="l1-title" onclick="toggleL1(this)"><span>funasr.train_utils</span><span class="cnt">11</span><span class="arr">▸</span></div><div class="l1-children">
+<div class="l2"><div class="l2-title" onclick="toggleL2(this)"><span>add_gradient_noise</span><span class="arr">▸</span></div><div class="l2-children">
+<a class="l3-item" data-target="e486" onclick="showEntry('e486')"><span class="mb badge-f">F</span>add_gradient_noise</a>
+</div></div>
+<div class="l2"><div class="l2-title" onclick="toggleL2(this)"><span>average_nbest_models</span><span class="arr">▸</span></div><div class="l2-children">
+<a class="l3-item" data-target="e487" onclick="showEntry('e487')"><span class="mb badge-f">F</span>average_checkpoints</a>
+</div></div>
+<div class="l2"><div class="l2-title" onclick="toggleL2(this)"><span>device_funcs</span><span class="arr">▸</span></div><div class="l2-children">
+<a class="l3-item" data-target="e488" onclick="showEntry('e488')"><span class="mb badge-f">F</span>to_device</a>
+<a class="l3-item" data-target="e489" onclick="showEntry('e489')"><span class="mb badge-f">F</span>force_gatherable</a>
+</div></div>
+<div class="l2"><div class="l2-title" onclick="toggleL2(this)"><span>forward_adaptor</span><span class="arr">▸</span></div><div class="l2-children">
+<a class="l3-item" data-target="e490" onclick="showEntry('e490')"><span class="mb badge-c">C</span>ForwardAdaptor</a>
+<a class="l3-item l3m" data-target="e491" onclick="showEntry('e491')"><span class="mb badge-m">M</span>.forward()</a>
+</div></div>
+<div class="l2"><div class="l2-title" onclick="toggleL2(this)"><span>initialize</span><span class="arr">▸</span></div><div class="l2-children">
+<a class="l3-item" data-target="e492" onclick="showEntry('e492')"><span class="mb badge-f">F</span>initialize</a>
+</div></div>
+<div class="l2"><div class="l2-title" onclick="toggleL2(this)"><span>load_pretrained_model</span><span class="arr">▸</span></div><div class="l2-children">
+<a class="l3-item" data-target="e493" onclick="showEntry('e493')"><span class="mb badge-f">F</span>load_pretrained_model</a>
+</div></div>
+<div class="l2"><div class="l2-title" onclick="toggleL2(this)"><span>model_summary</span><span class="arr">▸</span></div><div class="l2-children">
+<a class="l3-item" data-target="e494" onclick="showEntry('e494')"><span class="mb badge-f">F</span>get_human_readable_count</a>
+<a class="l3-item" data-target="e495" onclick="showEntry('e495')"><span class="mb badge-f">F</span>to_bytes</a>
+<a class="l3-item" data-target="e496" onclick="showEntry('e496')"><span class="mb badge-f">F</span>model_summary</a>
+</div></div>
+<div class="l2"><div class="l2-title" onclick="toggleL2(this)"><span>recursive_op</span><span class="arr">▸</span></div><div class="l2-children">
+<a class="l3-item" data-target="e497" onclick="showEntry('e497')"><span class="mb badge-f">F</span>recursive_sum</a>
+<a class="l3-item" data-target="e498" onclick="showEntry('e498')"><span class="mb badge-f">F</span>recursive_divide</a>
+<a class="l3-item" data-target="e499" onclick="showEntry('e499')"><span class="mb badge-f">F</span>recursive_average</a>
+</div></div>
+<div class="l2"><div class="l2-title" onclick="toggleL2(this)"><span>set_all_random_seed</span><span class="arr">▸</span></div><div class="l2-children">
+<a class="l3-item" data-target="e500" onclick="showEntry('e500')"><span class="mb badge-f">F</span>set_all_random_seed</a>
+</div></div>
+<div class="l2"><div class="l2-title" onclick="toggleL2(this)"><span>trainer</span><span class="arr">▸</span></div><div class="l2-children">
+<a class="l3-item" data-target="e501" onclick="showEntry('e501')"><span class="mb badge-f">F</span>maybe_autocast</a>
+<a class="l3-item" data-target="e502" onclick="showEntry('e502')"><span class="mb badge-c">C</span>Trainer</a>
+<a class="l3-item l3m" data-target="e503" onclick="showEntry('e503')"><span class="mb badge-m">M</span>.save_checkpoint()</a>
+<a class="l3-item l3m" data-target="e504" onclick="showEntry('e504')"><span class="mb badge-m">M</span>.resume_checkpoint()</a>
+<a class="l3-item l3m" data-target="e505" onclick="showEntry('e505')"><span class="mb badge-m">M</span>.train_epoch()</a>
+<a class="l3-item l3m" data-target="e506" onclick="showEntry('e506')"><span class="mb badge-m">M</span>.validate_epoch()</a>
+<a class="l3-item l3m" data-target="e507" onclick="showEntry('e507')"><span class="mb badge-m">M</span>.log()</a>
+<a class="l3-item l3m" data-target="e508" onclick="showEntry('e508')"><span class="mb badge-m">M</span>.close()</a>
+</div></div>
+<div class="l2"><div class="l2-title" onclick="toggleL2(this)"><span>trainer_ds</span><span class="arr">▸</span></div><div class="l2-children">
+<a class="l3-item" data-target="e509" onclick="showEntry('e509')"><span class="mb badge-f">F</span>maybe_autocast</a>
+<a class="l3-item" data-target="e510" onclick="showEntry('e510')"><span class="mb badge-c">C</span>Trainer</a>
+<a class="l3-item l3m" data-target="e511" onclick="showEntry('e511')"><span class="mb badge-m">M</span>.save_checkpoint()</a>
+<a class="l3-item l3m" data-target="e512" onclick="showEntry('e512')"><span class="mb badge-m">M</span>.resume_checkpoint()</a>
+<a class="l3-item l3m" data-target="e513" onclick="showEntry('e513')"><span class="mb badge-m">M</span>.train_epoch()</a>
+<a class="l3-item l3m" data-target="e514" onclick="showEntry('e514')"><span class="mb badge-m">M</span>.forward_step()</a>
+<a class="l3-item l3m" data-target="e515" onclick="showEntry('e515')"><span class="mb badge-m">M</span>.backward_step()</a>
+<a class="l3-item l3m" data-target="e516" onclick="showEntry('e516')"><span class="mb badge-m">M</span>.update_step()</a>
+<a class="l3-item l3m" data-target="e517" onclick="showEntry('e517')"><span class="mb badge-m">M</span>.validate_epoch()</a>
+<a class="l3-item l3m" data-target="e518" onclick="showEntry('e518')"><span class="mb badge-m">M</span>.log()</a>
+<a class="l3-item l3m" data-target="e519" onclick="showEntry('e519')"><span class="mb badge-m">M</span>.close()</a>
+<a class="l3-item l3m" data-target="e520" onclick="showEntry('e520')"><span class="mb badge-m">M</span>.warp_model()</a>
+<a class="l3-item l3m" data-target="e521" onclick="showEntry('e521')"><span class="mb badge-m">M</span>.warp_optim_scheduler()</a>
+</div></div>
+</div></div>
+<div class="l1"><div class="l1-title" onclick="toggleL1(this)"><span>funasr.datasets</span><span class="cnt">11</span><span class="arr">▸</span></div><div class="l1-children">
+<div class="l2"><div class="l2-title" onclick="toggleL2(this)"><span>datasets</span><span class="arr">▸</span></div><div class="l2-children">
+<a class="l3-item" data-target="e522" onclick="showEntry('e522')"><span class="mb badge-c">C</span>AudioDataset</a>
+<a class="l3-item l3m" data-target="e523" onclick="showEntry('e523')"><span class="mb badge-m">M</span>.get_source_len()</a>
+<a class="l3-item l3m" data-target="e524" onclick="showEntry('e524')"><span class="mb badge-m">M</span>.get_target_len()</a>
+<a class="l3-item l3m" data-target="e525" onclick="showEntry('e525')"><span class="mb badge-m">M</span>.collator()</a>
+<a class="l3-item" data-target="e526" onclick="showEntry('e526')"><span class="mb badge-c">C</span>AudioDatasetHotword</a>
+<a class="l3-item l3m" data-target="e527" onclick="showEntry('e527')"><span class="mb badge-m">M</span>.collator()</a>
+</div></div>
+<div class="l2"><div class="l2-title" onclick="toggleL2(this)"><span>espnet_samplers</span><span class="arr">▸</span></div><div class="l2-children">
+<a class="l3-item" data-target="e528" onclick="showEntry('e528')"><span class="mb badge-f">F</span>EspnetStyleBatchSampler_fn</a>
+<a class="l3-item" data-target="e529" onclick="showEntry('e529')"><span class="mb badge-c">C</span>EspnetStyleBatchSampler</a>
+<a class="l3-item l3m" data-target="e530" onclick="showEntry('e530')"><span class="mb badge-m">M</span>.set_epoch()</a>
+</div></div>
+<div class="l2"><div class="l2-title" onclick="toggleL2(this)"><span>index_ds</span><span class="arr">▸</span></div><div class="l2-children">
+<a class="l3-item" data-target="e531" onclick="showEntry('e531')"><span class="mb badge-c">C</span>IndexDSJsonlRankFull</a>
+<a class="l3-item l3m" data-target="e532" onclick="showEntry('e532')"><span class="mb badge-m">M</span>.get_source_len()</a>
+<a class="l3-item l3m" data-target="e533" onclick="showEntry('e533')"><span class="mb badge-m">M</span>.get_target_len()</a>
+</div></div>
+<div class="l2"><div class="l2-title" onclick="toggleL2(this)"><span>jsonl2scp</span><span class="arr">▸</span></div><div class="l2-children">
+<a class="l3-item" data-target="e534" onclick="showEntry('e534')"><span class="mb badge-f">F</span>gen_scp_from_jsonl</a>
+<a class="l3-item" data-target="e535" onclick="showEntry('e535')"><span class="mb badge-f">F</span>main_hydra</a>
+</div></div>
+<div class="l2"><div class="l2-title" onclick="toggleL2(this)"><span>preprocessor</span><span class="arr">▸</span></div><div class="l2-children">
+<a class="l3-item" data-target="e536" onclick="showEntry('e536')"><span class="mb badge-c">C</span>SpeechPreprocessSpeedPerturb</a>
+<a class="l3-item l3m" data-target="e537" onclick="showEntry('e537')"><span class="mb badge-m">M</span>.forward()</a>
+<a class="l3-item" data-target="e538" onclick="showEntry('e538')"><span class="mb badge-c">C</span>TextPreprocessSegDict</a>
+<a class="l3-item l3m" data-target="e539" onclick="showEntry('e539')"><span class="mb badge-m">M</span>.forward()</a>
+</div></div>
+<div class="l2"><div class="l2-title" onclick="toggleL2(this)"><span>samplers</span><span class="arr">▸</span></div><div class="l2-children">
+<a class="l3-item" data-target="e540" onclick="showEntry('e540')"><span class="mb badge-f">F</span>CustomDistributedBatchSampler_fn</a>
+<a class="l3-item" data-target="e541" onclick="showEntry('e541')"><span class="mb badge-c">C</span>CustomDistributedBatchSampler</a>
+<a class="l3-item l3m" data-target="e542" onclick="showEntry('e542')"><span class="mb badge-m">M</span>.set_epoch()</a>
+<a class="l3-item" data-target="e543" onclick="showEntry('e543')"><span class="mb badge-c">C</span>CustomDistributedBufferBatchSampler</a>
+<a class="l3-item l3m" data-target="e544" onclick="showEntry('e544')"><span class="mb badge-m">M</span>.set_epoch()</a>
+<a class="l3-item" data-target="e545" onclick="showEntry('e545')"><span class="mb badge-c">C</span>CustomDistributedDynamicBatchSampler</a>
+<a class="l3-item l3m" data-target="e546" onclick="showEntry('e546')"><span class="mb badge-m">M</span>.set_epoch()</a>
+<a class="l3-item" data-target="e547" onclick="showEntry('e547')"><span class="mb badge-c">C</span>CustomDistributedBufferDynamicBatchSampler</a>
+<a class="l3-item l3m" data-target="e548" onclick="showEntry('e548')"><span class="mb badge-m">M</span>.set_epoch()</a>
+<a class="l3-item" data-target="e549" onclick="showEntry('e549')"><span class="mb badge-c">C</span>DistributedSamplerWarp</a>
+<a class="l3-item l3m" data-target="e550" onclick="showEntry('e550')"><span class="mb badge-m">M</span>.set_epoch()</a>
+</div></div>
+<div class="l2"><div class="l2-title" onclick="toggleL2(this)"><span>scp2jsonl</span><span class="arr">▸</span></div><div class="l2-children">
+<a class="l3-item" data-target="e551" onclick="showEntry('e551')"><span class="mb badge-f">F</span>gen_jsonl_from_wav_text_list</a>
+<a class="l3-item" data-target="e552" onclick="showEntry('e552')"><span class="mb badge-f">F</span>parse_context_length</a>
+<a class="l3-item" data-target="e553" onclick="showEntry('e553')"><span class="mb badge-f">F</span>main_hydra</a>
+</div></div>
+<div class="l2"><div class="l2-title" onclick="toggleL2(this)"><span>scp2len</span><span class="arr">▸</span></div><div class="l2-children">
+<a class="l3-item" data-target="e554" onclick="showEntry('e554')"><span class="mb badge-f">F</span>gen_jsonl_from_wav_text_list</a>
+<a class="l3-item" data-target="e555" onclick="showEntry('e555')"><span class="mb badge-f">F</span>parse_context_length</a>
+<a class="l3-item" data-target="e556" onclick="showEntry('e556')"><span class="mb badge-f">F</span>main_hydra</a>
+</div></div>
+<div class="l2"><div class="l2-title" onclick="toggleL2(this)"><span>sensevoice2jsonl</span><span class="arr">▸</span></div><div class="l2-children">
+<a class="l3-item" data-target="e557" onclick="showEntry('e557')"><span class="mb badge-f">F</span>gen_jsonl_from_wav_text_list</a>
+<a class="l3-item" data-target="e558" onclick="showEntry('e558')"><span class="mb badge-f">F</span>contains_punctuation</a>
+<a class="l3-item" data-target="e559" onclick="showEntry('e559')"><span class="mb badge-f">F</span>parse_context_length</a>
+<a class="l3-item" data-target="e560" onclick="showEntry('e560')"><span class="mb badge-f">F</span>main_hydra</a>
+</div></div>
+<div class="l2"><div class="l2-title" onclick="toggleL2(this)"><span>update_jsonl</span><span class="arr">▸</span></div><div class="l2-children">
+<a class="l3-item" data-target="e561" onclick="showEntry('e561')"><span class="mb badge-f">F</span>gen_scp_from_jsonl</a>
+<a class="l3-item" data-target="e562" onclick="showEntry('e562')"><span class="mb badge-f">F</span>update_data</a>
+<a class="l3-item" data-target="e563" onclick="showEntry('e563')"><span class="mb badge-f">F</span>update_wav_len</a>
+<a class="l3-item" data-target="e564" onclick="showEntry('e564')"><span class="mb badge-f">F</span>main_hydra</a>
+</div></div>
+<div class="l2"><div class="l2-title" onclick="toggleL2(this)"><span>dataloader_entry</span><span class="arr">▸</span></div><div class="l2-children">
+<a class="l3-item" data-target="e565" onclick="showEntry('e565')"><span class="mb badge-f">F</span>DataloaderMapStyle</a>
+<a class="l3-item" data-target="e566" onclick="showEntry('e566')"><span class="mb badge-c">C</span>DataloaderMapStyle</a>
+<a class="l3-item l3m" data-target="e567" onclick="showEntry('e567')"><span class="mb badge-m">M</span>.build_iter()</a>
+<a class="l3-item" data-target="e568" onclick="showEntry('e568')"><span class="mb badge-f">F</span>DataloaderIterable</a>
+</div></div>
+</div></div>
+<div class="l1"><div class="l1-title" onclick="toggleL1(this)"><span>funasr.losses</span><span class="cnt">1</span><span class="arr">▸</span></div><div class="l1-children">
+<div class="l2"><div class="l2-title" onclick="toggleL2(this)"><span>label_smoothing_loss</span><span class="arr">▸</span></div><div class="l2-children">
+<a class="l3-item" data-target="e569" onclick="showEntry('e569')"><span class="mb badge-c">C</span>LabelSmoothingLoss</a>
+<a class="l3-item l3m" data-target="e570" onclick="showEntry('e570')"><span class="mb badge-m">M</span>.forward()</a>
+<a class="l3-item" data-target="e571" onclick="showEntry('e571')"><span class="mb badge-c">C</span>SequenceBinaryCrossEntropy</a>
+<a class="l3-item l3m" data-target="e572" onclick="showEntry('e572')"><span class="mb badge-m">M</span>.forward()</a>
+<a class="l3-item" data-target="e573" onclick="showEntry('e573')"><span class="mb badge-c">C</span>NllLoss</a>
+<a class="l3-item l3m" data-target="e574" onclick="showEntry('e574')"><span class="mb badge-m">M</span>.forward()</a>
+</div></div>
+</div></div>
+<div class="l1"><div class="l1-title" onclick="toggleL1(this)"><span>funasr.schedulers</span><span class="cnt">4</span><span class="arr">▸</span></div><div class="l1-children">
+<div class="l2"><div class="l2-title" onclick="toggleL2(this)"><span>lambdalr_cus</span><span class="arr">▸</span></div><div class="l2-children">
+<a class="l3-item" data-target="e575" onclick="showEntry('e575')"><span class="mb badge-c">C</span>CustomLambdaLR</a>
+<a class="l3-item l3m" data-target="e576" onclick="showEntry('e576')"><span class="mb badge-m">M</span>.get_lr()</a>
+</div></div>
+<div class="l2"><div class="l2-title" onclick="toggleL2(this)"><span>noam_lr</span><span class="arr">▸</span></div><div class="l2-children">
+<a class="l3-item" data-target="e577" onclick="showEntry('e577')"><span class="mb badge-c">C</span>NoamLR</a>
+<a class="l3-item l3m" data-target="e578" onclick="showEntry('e578')"><span class="mb badge-m">M</span>.lr_for_WarmupLR()</a>
+<a class="l3-item l3m" data-target="e579" onclick="showEntry('e579')"><span class="mb badge-m">M</span>.get_lr()</a>
+</div></div>
+<div class="l2"><div class="l2-title" onclick="toggleL2(this)"><span>tri_stage_scheduler</span><span class="arr">▸</span></div><div class="l2-children">
+<a class="l3-item" data-target="e580" onclick="showEntry('e580')"><span class="mb badge-c">C</span>TriStageLR</a>
+<a class="l3-item l3m" data-target="e581" onclick="showEntry('e581')"><span class="mb badge-m">M</span>.init_tri_stage_scheudler()</a>
+<a class="l3-item l3m" data-target="e582" onclick="showEntry('e582')"><span class="mb badge-m">M</span>.step_update()</a>
+<a class="l3-item l3m" data-target="e583" onclick="showEntry('e583')"><span class="mb badge-m">M</span>.set_optimizer_lr()</a>
+<a class="l3-item l3m" data-target="e584" onclick="showEntry('e584')"><span class="mb badge-m">M</span>.get_lr()</a>
+</div></div>
+<div class="l2"><div class="l2-title" onclick="toggleL2(this)"><span>warmup_lr</span><span class="arr">▸</span></div><div class="l2-children">
+<a class="l3-item" data-target="e585" onclick="showEntry('e585')"><span class="mb badge-c">C</span>WarmupLR</a>
+<a class="l3-item l3m" data-target="e586" onclick="showEntry('e586')"><span class="mb badge-m">M</span>.get_lr()</a>
+</div></div>
+</div></div>
+
+</div>
+<div class="api-content">
+<div class="api-welcome" id="api-welcome"><h2>API Reference</h2><p>250 entries · auto-generated from source<br>Click source code to expand. Links point to latest GitHub code.</p></div>
+<div class="api-detail" id="e1"><span class="dtag badge-f">function</span><h2><code>is_npu_available()</code></h2><div class="dmod">funasr.auto.auto_model · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/auto/auto_model.py#L34" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>检查NPU是否可用。</p>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code>def is_npu_available():
+ &quot;&quot;&quot;检查NPU是否可用。&quot;&quot;&quot;
+ try:
+ import torch_npu
+
+ return torch_npu.npu.is_available()
+ except ImportError:
+ return False
+</code></pre></details></div>
+<div class="api-detail" id="e2"><span class="dtag badge-f">function</span><h2><code>prepare_data_iterator(data_in, input_len, data_type, key)</code></h2><div class="dmod">funasr.auto.auto_model · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/auto/auto_model.py#L111" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p class="muted">No documentation yet.</p></div><details class="src-block"><summary>📄 Source code</summary><pre><code>def prepare_data_iterator(data_in, input_len=None, data_type=None, key=None):
+ &quot;&quot;&quot; &quot;&quot;&quot;
+ data_list = []
+ key_list = []
+ filelist = [&quot;.scp&quot;, &quot;.txt&quot;, &quot;.json&quot;, &quot;.jsonl&quot;, &quot;.text&quot;]
+
+ chars = string.ascii_letters + string.digits
+ if isinstance(data_in, str):
+ if data_in.startswith(&quot;http://&quot;) or data_in.startswith(&quot;https://&quot;): # url
+ data_in = download_from_url(data_in)
+
+ if isinstance(data_in, str) and os.path.exists(
+ data_in
+ ): # wav_path; filelist: wav.scp, file.jsonl;text.txt;
+ _, file_extension = os.path.splitext(data_in)
+ file_extension = file_extension.lower()
+ if file_extension in filelist: # filelist: wav.scp, file.jsonl;text.txt;
+ with open(data_in, encoding=&quot;utf-8&quot;) as fin:
+ for line in fin:
+ key = &quot;rand_key_&quot; + &quot;&quot;.join(random.choice(chars) for _ in range(13))
+ if data_in.endswith(&quot;.jsonl&quot;): # file.jsonl: json.dumps({&quot;source&quot;: data})
+ lines = json.loads(line.strip())
+ data = lines[&quot;source&quot;]
+ key = lines.get(&quot;key&quot;, key)
+ else: # filelist, wav.scp, text.txt: id \t data or data
+ lines = line.strip().split(maxsplit=1)
+ data = lines[1] if len(lines) &gt; 1 else lines[0]
+ key = lines[0] if len(lines) &gt; 1 else key
+
+ data_list.append(data)
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/auto/auto_model.py#L111" target="_blank" class="src-more">View full source on GitHub →</a></details></div>
+<div class="api-detail" id="e3"><span class="dtag badge-c">class</span><h2>AutoModel</h2><div class="dmod">funasr.auto.auto_model · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/auto/auto_model.py#L182" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p class="muted">No documentation yet.</p></div><details class="src-block"><summary>📄 Source code</summary><pre><code>class AutoModel:
+
+ def __init__(self, **kwargs):
+ &quot;&quot;&quot;Initialize AutoModel with ASR model and optional sub-models.
+
+ Args:
+ model (str): Model name (hub alias or full ID) or local path.
+ device (str): Device for inference. &quot;cuda:0&quot;, &quot;cpu&quot;, &quot;mps&quot;, &quot;npu:0&quot;.
+ Falls back to CPU if specified device is unavailable.
+ vad_model (str, optional): VAD model for long audio segmentation.
+ Enables processing of any-length audio.
+ vad_kwargs (dict, optional): VAD config, e.g. {&quot;max_single_segment_time&quot;: 60000}.
+ punc_model (str, optional): Punctuation restoration model.
+ Not needed for Fun-ASR-Nano/SenseVoice/Qwen3-ASR (they output punctuation natively).
+ spk_model (str, optional): Speaker model for diarization (&quot;cam++&quot; or full model ID).
+ Requires vad_model. For Qwen3-ASR, also requires forced_aligner.
+ spk_mode (str, optional): Speaker diarization mode. &quot;punc_segment&quot; (default) or &quot;vad_segment&quot;.
+ hub (str): Model hub. &quot;ms&quot; (ModelScope, default) or &quot;hf&quot; (HuggingFace).
+ ncpu (int): CPU threads (default: 4).
+ disable_update (bool): Skip version check on startup.
+ disable_pbar (bool): Disable tqdm progress bars.
+ **kwargs: Additional model-specific parameters (passed to config.yaml overrides).
+
+ Examples:
+ &gt;&gt;&gt; model = AutoModel(model=&quot;paraformer-zh&quot;, vad_model=&quot;fsmn-vad&quot;, punc_model=&quot;ct-punc&quot;)
+ &gt;&gt;&gt; model = AutoModel(model=&quot;FunAudioLLM/Fun-ASR-Nano-2512&quot;, trust_remote_code=True,
+ ... remote_code=&quot;./model.py&quot;, vad_model=&quot;fsmn-vad&quot;, spk_model=&quot;cam++&quot;, hub=&quot;hf&quot;)
+ &quot;&quot;&quot;
+ try:
+ from funasr.utils.version_checker import check_for_update
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/auto/auto_model.py#L182" target="_blank" class="src-more">View full source on GitHub →</a></details><h3 class="mt">Methods</h3><div class="mblk"><code>.build_model(**kwargs)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/auto/auto_model.py#L275" target="_blank" class="gh-link">L275</a><div class="mdoc"><p>Download model from hub, build all components, and load pretrained weights.</p>
+<p>This method handles the full model construction pipeline:</p>
+<p>1. Download model files from ModelScope/HuggingFace (if not local)</p>
+<p>2. Parse config.yaml to determine model class, tokenizer, frontend</p>
+<p>3. Instantiate tokenizer, frontend, and model via the registry</p>
+<p>4. Load pretrained weights from model.pt</p>
+<h4>Args:</h4>
+<ul>
+<li><code>**kwargs</code> — Must include &#x27;model&#x27; (str). All other config.yaml fields can be overridden.</li>
+</ul>
+<h4>Returns:</h4>
+<ul>
+<li><code>tuple</code> — (model, kwargs) where model is the instantiated nn.Module and</li>
+<li class="sub">kwargs contains the resolved configuration.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def build_model(**kwargs):
+ &quot;&quot;&quot;Download model from hub, build all components, and load pretrained weights.
+
+ This method handles the full model construction pipeline:
+ 1. Download model files from ModelScope/HuggingFace (if not local)
+ 2. Parse config.yaml to determine model class, tokenizer, frontend
+ 3. Instantiate tokenizer, frontend, and model via the registry
+ 4. Load pretrained weights from model.pt
+
+ Args:
+ **kwargs: Must include &#x27;model&#x27; (str). All other config.yaml fields can be overridden.
+
+ Returns:
+ tuple: (model, kwargs) where model is the instantiated nn.Module and
+ kwargs contains the resolved configuration.
+ &quot;&quot;&quot;
+ assert &quot;model&quot; in kwargs
+ if &quot;model_conf&quot; not in kwargs:
+ logging.info(&quot;download models from model hub: {}&quot;.format(kwargs.get(&quot;hub&quot;, &quot;ms&quot;)))
+ kwargs = download_model(**kwargs)
+
+ set_all_random_seed(kwargs.get(&quot;seed&quot;, 0))
+
+ device = kwargs.get(&quot;device&quot;, &quot;cuda&quot;)
+ if (
+ (device.startswith(&quot;cuda&quot;) and not torch.cuda.is_available())
+ or (device.startswith(&quot;xpu&quot;) and not torch.xpu.is_available())
+ or (device.startswith(&quot;mps&quot;) and not torch.backends.mps.is_available())
+ or (device.startswith(&quot;npu&quot;) and not is_npu_available())
+ or kwargs.get(&quot;ngpu&quot;, 1) == 0
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/auto/auto_model.py#L275" target="_blank" class="src-more">View full source →</a></details></div><div class="mblk"><code>.generate(input, input_len, progress_callback, **cfg)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/auto/auto_model.py#L436" target="_blank" class="gh-link">L436</a><div class="mdoc"><p>Run speech recognition on input audio.</p>
+<p>This is the primary user-facing method. It automatically routes to:</p>
+<ul>
+<li>inference() if no vad_model is configured (single utterance)</li>
+<li>inference_with_vad() if vad_model is configured (long audio with segmentation)</li>
+</ul>
+<h4>Args:</h4>
+<ul>
+<li><code>input</code> — Audio input. Accepts:</li>
+<li>File path (str): &quot;audio.wav&quot;, &quot;audio.mp3&quot;</li>
+<li>URL (str): &quot;https://...&quot;</li>
+<li>numpy array: raw audio samples (float32, 16kHz)</li>
+<li>list: batch of file paths or arrays</li>
+<li>bytes: raw audio bytes</li>
+<li><code>input_len (tensor, optional)</code> — Length of each input sample.</li>
+<li><code>progress_callback (callable, optional)</code> — fn(current, total) called during processing.</li>
+<li><code>**cfg</code> — Runtime parameters:</li>
+<li>cache (dict): State cache for streaming mode. Pass {} for first call.</li>
+<li>hotword (str/list): Keywords to boost recognition accuracy.</li>
+<li>language (str): Language hint (&quot;auto&quot;, &quot;zh&quot;, &quot;en&quot;, &quot;Chinese&quot;, etc.)</li>
+<li>batch_size_s (int): Dynamic batch total duration in seconds.</li>
+<li>is_final (bool): Last chunk flag for streaming mode.</li>
+<li>return_spk_res (bool): Return speaker diarization results.</li>
+<li>sentence_timestamp (bool): Return sentence-level timestamps.</li>
+<li>use_itn (bool): Apply inverse text normalization (SenseVoice).</li>
+</ul>
+<h4>Returns:</h4>
+<p>list[dict]: Results for each input sample. Common fields:</p>
+<ul>
+<li>&quot;key&quot; (str): Sample identifier</li>
+<li>&quot;text&quot; (str): Recognized text</li>
+<li>&quot;timestamp&quot; (list): [[start_ms, end_ms], ...] per character/word</li>
+<li>&quot;sentence_info&quot; (list): [{text, start, end, spk, timestamp}, ...] when spk enabled</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def generate(self, input, input_len=None, progress_callback=None, **cfg):
+ &quot;&quot;&quot;Run speech recognition on input audio.
+
+ This is the primary user-facing method. It automatically routes to:
+ - inference() if no vad_model is configured (single utterance)
+ - inference_with_vad() if vad_model is configured (long audio with segmentation)
+
+ Args:
+ input: Audio input. Accepts:
+ - File path (str): &quot;audio.wav&quot;, &quot;audio.mp3&quot;
+ - URL (str): &quot;https://...&quot;
+ - numpy array: raw audio samples (float32, 16kHz)
+ - list: batch of file paths or arrays
+ - bytes: raw audio bytes
+ input_len (tensor, optional): Length of each input sample.
+ progress_callback (callable, optional): fn(current, total) called during processing.
+ **cfg: Runtime parameters:
+ - cache (dict): State cache for streaming mode. Pass {} for first call.
+ - hotword (str/list): Keywords to boost recognition accuracy.
+ - language (str): Language hint (&quot;auto&quot;, &quot;zh&quot;, &quot;en&quot;, &quot;Chinese&quot;, etc.)
+ - batch_size_s (int): Dynamic batch total duration in seconds.
+ - is_final (bool): Last chunk flag for streaming mode.
+ - return_spk_res (bool): Return speaker diarization results.
+ - sentence_timestamp (bool): Return sentence-level timestamps.
+ - use_itn (bool): Apply inverse text normalization (SenseVoice).
+
+ Returns:
+ list[dict]: Results for each input sample. Common fields:
+ - &quot;key&quot; (str): Sample identifier
+ - &quot;text&quot; (str): Recognized text
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/auto/auto_model.py#L436" target="_blank" class="src-more">View full source →</a></details></div><div class="mblk"><code>.inference(input, input_len, model, kwargs, key, progress_callback, **cfg)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/auto/auto_model.py#L490" target="_blank" class="gh-link">L490</a><div class="mdoc"><p>Run model inference on input data (internal method).</p>
+<p>Handles batching, timing, and progress reporting. Called by generate()</p>
+<p>and inference_with_vad(). Typically not called directly by users.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>input</code> — Audio data, file path, or text (for punc model).</li>
+<li><code>input_len (tensor, optional)</code> — Input lengths for batch.</li>
+<li><code>model (nn.Module, optional)</code> — Override model (used for VAD/PUNC/SPK sub-models).</li>
+<li><code>kwargs (dict, optional)</code> — Override kwargs (used for sub-model configs).</li>
+<li><code>key (list, optional)</code> — Sample identifiers.</li>
+<li><code>progress_callback (callable, optional)</code> — Progress reporting function.</li>
+<li><code>**cfg</code> — Additional config merged into kwargs.</li>
+</ul>
+<h4>Returns:</h4>
+<p>list[dict]: Model inference results.</p>
+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def inference(
+ self,
+ input,
+ input_len=None,
+ model=None,
+ kwargs=None,
+ key=None,
+ progress_callback=None,
+ **cfg,
+ ):
+ &quot;&quot;&quot;Run model inference on input data (internal method).
+
+ Handles batching, timing, and progress reporting. Called by generate()
+ and inference_with_vad(). Typically not called directly by users.
+
+ Args:
+ input: Audio data, file path, or text (for punc model).
+ input_len (tensor, optional): Input lengths for batch.
+ model (nn.Module, optional): Override model (used for VAD/PUNC/SPK sub-models).
+ kwargs (dict, optional): Override kwargs (used for sub-model configs).
+ key (list, optional): Sample identifiers.
+ progress_callback (callable, optional): Progress reporting function.
+ **cfg: Additional config merged into kwargs.
+
+ Returns:
+ list[dict]: Model inference results.
+ &quot;&quot;&quot;
+ if kwargs is None:
+ self._reset_runtime_configs()
+ kwargs = self.kwargs if kwargs is None else kwargs
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/auto/auto_model.py#L490" target="_blank" class="src-more">View full source →</a></details></div><div class="mblk"><code>.inference_with_vad(input, input_len, **cfg)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/auto/auto_model.py#L592" target="_blank" class="gh-link">L592</a><div class="mdoc"><p>Run ASR with VAD segmentation, punctuation, and optional speaker diarization.</p>
+<p>Pipeline:</p>
+<p>1. VAD: Segment audio into speech regions</p>
+<p>2. ASR: Recognize each segment (sorted by length for efficient batching)</p>
+<p>3. Timestamp merge: Combine per-segment timestamps with VAD offsets</p>
+<p>4. Punctuation: Add punctuation to combined text (if punc_model configured)</p>
+<p>5. Speaker diarization: Cluster speaker embeddings and assign labels (if spk_model configured)</p>
+<h4>Args:</h4>
+<ul>
+<li><code>input</code> — Audio file path, URL, or numpy array.</li>
+<li><code>input_len</code> — Not used (kept for interface consistency).</li>
+<li><code>**cfg</code> — Runtime parameters (same as generate()).</li>
+</ul>
+<h4>Returns:</h4>
+<p>list[dict]: Results with fields: key, text, timestamp, sentence_info, raw_text.</p>
+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def inference_with_vad(self, input, input_len=None, **cfg):
+ &quot;&quot;&quot;Run ASR with VAD segmentation, punctuation, and optional speaker diarization.
+
+ Pipeline:
+ 1. VAD: Segment audio into speech regions
+ 2. ASR: Recognize each segment (sorted by length for efficient batching)
+ 3. Timestamp merge: Combine per-segment timestamps with VAD offsets
+ 4. Punctuation: Add punctuation to combined text (if punc_model configured)
+ 5. Speaker diarization: Cluster speaker embeddings and assign labels (if spk_model configured)
+
+ Args:
+ input: Audio file path, URL, or numpy array.
+ input_len: Not used (kept for interface consistency).
+ **cfg: Runtime parameters (same as generate()).
+
+ Returns:
+ list[dict]: Results with fields: key, text, timestamp, sentence_info, raw_text.
+ &quot;&quot;&quot;
+ self._reset_runtime_configs()
+ if self.spk_model is not None and &quot;output_timestamp&quot; not in cfg:
+ cfg[&quot;output_timestamp&quot;] = True
+ cfg[&quot;return_time_stamps&quot;] = True
+ kwargs = self.kwargs
+ # step.1: compute the vad model
+ deep_update(self.vad_kwargs, cfg)
+ beg_vad = time.time()
+ res = self.inference(
+ input, input_len=input_len, model=self.vad_model, kwargs=self.vad_kwargs, **cfg
+ )
+ end_vad = time.time()
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/auto/auto_model.py#L592" target="_blank" class="src-more">View full source →</a></details></div><div class="mblk"><code>.export(input, **cfg)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/auto/auto_model.py#L911" target="_blank" class="gh-link">L911</a><div class="mdoc"><p>Export model to ONNX format.</p>
+<p>Creates a deep copy of the model to isolate ONNX operator monkey-patching,</p>
+<p>then runs torch.onnx.export. The original model remains usable after export.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>input</code> — Sample input for tracing (auto-generated if None).</li>
+<li><code>**cfg</code> — Export parameters:</li>
+<li>type (str): Export format, &quot;onnx&quot; (default).</li>
+<li>quantize (bool): Whether to quantize the model.</li>
+<li>device (str): Device for export.</li>
+</ul>
+<h4>Returns:</h4>
+<ul>
+<li><code>str</code> — Path to the exported model directory.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def export(self, input=None, **cfg):
+ &quot;&quot;&quot;Export model to ONNX format.
+
+ Creates a deep copy of the model to isolate ONNX operator monkey-patching,
+ then runs torch.onnx.export. The original model remains usable after export.
+
+ Args:
+ input: Sample input for tracing (auto-generated if None).
+ **cfg: Export parameters:
+ - type (str): Export format, &quot;onnx&quot; (default).
+ - quantize (bool): Whether to quantize the model.
+ - device (str): Device for export.
+
+ Returns:
+ str: Path to the exported model directory.
+ &quot;&quot;&quot;
+ &quot;&quot;&quot;
+
+ :param input:
+ :param type:
+ :param quantize:
+ :param fallback_num:
+ :param calib_num:
+ :param opset_version:
+ :param cfg:
+ :return:
+ &quot;&quot;&quot;
+
+ device = cfg.get(&quot;device&quot;, &quot;cpu&quot;)
+
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/auto/auto_model.py#L911" target="_blank" class="src-more">View full source →</a></details></div></div>
+<div class="api-detail" id="e4"><span class="dtag badge-m">method</span><h2><code>AutoModel.build_model(**kwargs)</code></h2><div class="dmod">funasr.auto.auto_model.AutoModel · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/auto/auto_model.py#L275" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Download model from hub, build all components, and load pretrained weights.</p>
+<p>This method handles the full model construction pipeline:</p>
+<p>1. Download model files from ModelScope/HuggingFace (if not local)</p>
+<p>2. Parse config.yaml to determine model class, tokenizer, frontend</p>
+<p>3. Instantiate tokenizer, frontend, and model via the registry</p>
+<p>4. Load pretrained weights from model.pt</p>
+<h4>Args:</h4>
+<ul>
+<li><code>**kwargs</code> — Must include &#x27;model&#x27; (str). All other config.yaml fields can be overridden.</li>
+</ul>
+<h4>Returns:</h4>
+<ul>
+<li><code>tuple</code> — (model, kwargs) where model is the instantiated nn.Module and</li>
+<li class="sub">kwargs contains the resolved configuration.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def build_model(**kwargs):
+ &quot;&quot;&quot;Download model from hub, build all components, and load pretrained weights.
+
+ This method handles the full model construction pipeline:
+ 1. Download model files from ModelScope/HuggingFace (if not local)
+ 2. Parse config.yaml to determine model class, tokenizer, frontend
+ 3. Instantiate tokenizer, frontend, and model via the registry
+ 4. Load pretrained weights from model.pt
+
+ Args:
+ **kwargs: Must include &#x27;model&#x27; (str). All other config.yaml fields can be overridden.
+
+ Returns:
+ tuple: (model, kwargs) where model is the instantiated nn.Module and
+ kwargs contains the resolved configuration.
+ &quot;&quot;&quot;
+ assert &quot;model&quot; in kwargs
+ if &quot;model_conf&quot; not in kwargs:
+ logging.info(&quot;download models from model hub: {}&quot;.format(kwargs.get(&quot;hub&quot;, &quot;ms&quot;)))
+ kwargs = download_model(**kwargs)
+
+ set_all_random_seed(kwargs.get(&quot;seed&quot;, 0))
+
+ device = kwargs.get(&quot;device&quot;, &quot;cuda&quot;)
+ if (
+ (device.startswith(&quot;cuda&quot;) and not torch.cuda.is_available())
+ or (device.startswith(&quot;xpu&quot;) and not torch.xpu.is_available())
+ or (device.startswith(&quot;mps&quot;) and not torch.backends.mps.is_available())
+ or (device.startswith(&quot;npu&quot;) and not is_npu_available())
+ or kwargs.get(&quot;ngpu&quot;, 1) == 0
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/auto/auto_model.py#L275" target="_blank" class="src-more">View full source on GitHub →</a></details></div>
+<div class="api-detail" id="e5"><span class="dtag badge-m">method</span><h2><code>AutoModel.generate(input, input_len, progress_callback, **cfg)</code></h2><div class="dmod">funasr.auto.auto_model.AutoModel · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/auto/auto_model.py#L436" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Run speech recognition on input audio.</p>
+<p>This is the primary user-facing method. It automatically routes to:</p>
+<ul>
+<li>inference() if no vad_model is configured (single utterance)</li>
+<li>inference_with_vad() if vad_model is configured (long audio with segmentation)</li>
+</ul>
+<h4>Args:</h4>
+<ul>
+<li><code>input</code> — Audio input. Accepts:</li>
+<li>File path (str): &quot;audio.wav&quot;, &quot;audio.mp3&quot;</li>
+<li>URL (str): &quot;https://...&quot;</li>
+<li>numpy array: raw audio samples (float32, 16kHz)</li>
+<li>list: batch of file paths or arrays</li>
+<li>bytes: raw audio bytes</li>
+<li><code>input_len (tensor, optional)</code> — Length of each input sample.</li>
+<li><code>progress_callback (callable, optional)</code> — fn(current, total) called during processing.</li>
+<li><code>**cfg</code> — Runtime parameters:</li>
+<li>cache (dict): State cache for streaming mode. Pass {} for first call.</li>
+<li>hotword (str/list): Keywords to boost recognition accuracy.</li>
+<li>language (str): Language hint (&quot;auto&quot;, &quot;zh&quot;, &quot;en&quot;, &quot;Chinese&quot;, etc.)</li>
+<li>batch_size_s (int): Dynamic batch total duration in seconds.</li>
+<li>is_final (bool): Last chunk flag for streaming mode.</li>
+<li>return_spk_res (bool): Return speaker diarization results.</li>
+<li>sentence_timestamp (bool): Return sentence-level timestamps.</li>
+<li>use_itn (bool): Apply inverse text normalization (SenseVoice).</li>
+</ul>
+<h4>Returns:</h4>
+<p>list[dict]: Results for each input sample. Common fields:</p>
+<ul>
+<li>&quot;key&quot; (str): Sample identifier</li>
+<li>&quot;text&quot; (str): Recognized text</li>
+<li>&quot;timestamp&quot; (list): [[start_ms, end_ms], ...] per character/word</li>
+<li>&quot;sentence_info&quot; (list): [{text, start, end, spk, timestamp}, ...] when spk enabled</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def generate(self, input, input_len=None, progress_callback=None, **cfg):
+ &quot;&quot;&quot;Run speech recognition on input audio.
+
+ This is the primary user-facing method. It automatically routes to:
+ - inference() if no vad_model is configured (single utterance)
+ - inference_with_vad() if vad_model is configured (long audio with segmentation)
+
+ Args:
+ input: Audio input. Accepts:
+ - File path (str): &quot;audio.wav&quot;, &quot;audio.mp3&quot;
+ - URL (str): &quot;https://...&quot;
+ - numpy array: raw audio samples (float32, 16kHz)
+ - list: batch of file paths or arrays
+ - bytes: raw audio bytes
+ input_len (tensor, optional): Length of each input sample.
+ progress_callback (callable, optional): fn(current, total) called during processing.
+ **cfg: Runtime parameters:
+ - cache (dict): State cache for streaming mode. Pass {} for first call.
+ - hotword (str/list): Keywords to boost recognition accuracy.
+ - language (str): Language hint (&quot;auto&quot;, &quot;zh&quot;, &quot;en&quot;, &quot;Chinese&quot;, etc.)
+ - batch_size_s (int): Dynamic batch total duration in seconds.
+ - is_final (bool): Last chunk flag for streaming mode.
+ - return_spk_res (bool): Return speaker diarization results.
+ - sentence_timestamp (bool): Return sentence-level timestamps.
+ - use_itn (bool): Apply inverse text normalization (SenseVoice).
+
+ Returns:
+ list[dict]: Results for each input sample. Common fields:
+ - &quot;key&quot; (str): Sample identifier
+ - &quot;text&quot; (str): Recognized text
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/auto/auto_model.py#L436" target="_blank" class="src-more">View full source on GitHub →</a></details></div>
+<div class="api-detail" id="e6"><span class="dtag badge-m">method</span><h2><code>AutoModel.inference(input, input_len, model, kwargs, key, progress_callback, **cfg)</code></h2><div class="dmod">funasr.auto.auto_model.AutoModel · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/auto/auto_model.py#L490" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Run model inference on input data (internal method).</p>
+<p>Handles batching, timing, and progress reporting. Called by generate()</p>
+<p>and inference_with_vad(). Typically not called directly by users.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>input</code> — Audio data, file path, or text (for punc model).</li>
+<li><code>input_len (tensor, optional)</code> — Input lengths for batch.</li>
+<li><code>model (nn.Module, optional)</code> — Override model (used for VAD/PUNC/SPK sub-models).</li>
+<li><code>kwargs (dict, optional)</code> — Override kwargs (used for sub-model configs).</li>
+<li><code>key (list, optional)</code> — Sample identifiers.</li>
+<li><code>progress_callback (callable, optional)</code> — Progress reporting function.</li>
+<li><code>**cfg</code> — Additional config merged into kwargs.</li>
+</ul>
+<h4>Returns:</h4>
+<p>list[dict]: Model inference results.</p>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def inference(
+ self,
+ input,
+ input_len=None,
+ model=None,
+ kwargs=None,
+ key=None,
+ progress_callback=None,
+ **cfg,
+ ):
+ &quot;&quot;&quot;Run model inference on input data (internal method).
+
+ Handles batching, timing, and progress reporting. Called by generate()
+ and inference_with_vad(). Typically not called directly by users.
+
+ Args:
+ input: Audio data, file path, or text (for punc model).
+ input_len (tensor, optional): Input lengths for batch.
+ model (nn.Module, optional): Override model (used for VAD/PUNC/SPK sub-models).
+ kwargs (dict, optional): Override kwargs (used for sub-model configs).
+ key (list, optional): Sample identifiers.
+ progress_callback (callable, optional): Progress reporting function.
+ **cfg: Additional config merged into kwargs.
+
+ Returns:
+ list[dict]: Model inference results.
+ &quot;&quot;&quot;
+ if kwargs is None:
+ self._reset_runtime_configs()
+ kwargs = self.kwargs if kwargs is None else kwargs
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/auto/auto_model.py#L490" target="_blank" class="src-more">View full source on GitHub →</a></details></div>
+<div class="api-detail" id="e7"><span class="dtag badge-m">method</span><h2><code>AutoModel.inference_with_vad(input, input_len, **cfg)</code></h2><div class="dmod">funasr.auto.auto_model.AutoModel · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/auto/auto_model.py#L592" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Run ASR with VAD segmentation, punctuation, and optional speaker diarization.</p>
+<p>Pipeline:</p>
+<p>1. VAD: Segment audio into speech regions</p>
+<p>2. ASR: Recognize each segment (sorted by length for efficient batching)</p>
+<p>3. Timestamp merge: Combine per-segment timestamps with VAD offsets</p>
+<p>4. Punctuation: Add punctuation to combined text (if punc_model configured)</p>
+<p>5. Speaker diarization: Cluster speaker embeddings and assign labels (if spk_model configured)</p>
+<h4>Args:</h4>
+<ul>
+<li><code>input</code> — Audio file path, URL, or numpy array.</li>
+<li><code>input_len</code> — Not used (kept for interface consistency).</li>
+<li><code>**cfg</code> — Runtime parameters (same as generate()).</li>
+</ul>
+<h4>Returns:</h4>
+<p>list[dict]: Results with fields: key, text, timestamp, sentence_info, raw_text.</p>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def inference_with_vad(self, input, input_len=None, **cfg):
+ &quot;&quot;&quot;Run ASR with VAD segmentation, punctuation, and optional speaker diarization.
+
+ Pipeline:
+ 1. VAD: Segment audio into speech regions
+ 2. ASR: Recognize each segment (sorted by length for efficient batching)
+ 3. Timestamp merge: Combine per-segment timestamps with VAD offsets
+ 4. Punctuation: Add punctuation to combined text (if punc_model configured)
+ 5. Speaker diarization: Cluster speaker embeddings and assign labels (if spk_model configured)
+
+ Args:
+ input: Audio file path, URL, or numpy array.
+ input_len: Not used (kept for interface consistency).
+ **cfg: Runtime parameters (same as generate()).
+
+ Returns:
+ list[dict]: Results with fields: key, text, timestamp, sentence_info, raw_text.
+ &quot;&quot;&quot;
+ self._reset_runtime_configs()
+ if self.spk_model is not None and &quot;output_timestamp&quot; not in cfg:
+ cfg[&quot;output_timestamp&quot;] = True
+ cfg[&quot;return_time_stamps&quot;] = True
+ kwargs = self.kwargs
+ # step.1: compute the vad model
+ deep_update(self.vad_kwargs, cfg)
+ beg_vad = time.time()
+ res = self.inference(
+ input, input_len=input_len, model=self.vad_model, kwargs=self.vad_kwargs, **cfg
+ )
+ end_vad = time.time()
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/auto/auto_model.py#L592" target="_blank" class="src-more">View full source on GitHub →</a></details></div>
+<div class="api-detail" id="e8"><span class="dtag badge-m">method</span><h2><code>AutoModel.export(input, **cfg)</code></h2><div class="dmod">funasr.auto.auto_model.AutoModel · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/auto/auto_model.py#L911" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Export model to ONNX format.</p>
+<p>Creates a deep copy of the model to isolate ONNX operator monkey-patching,</p>
+<p>then runs torch.onnx.export. The original model remains usable after export.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>input</code> — Sample input for tracing (auto-generated if None).</li>
+<li><code>**cfg</code> — Export parameters:</li>
+<li>type (str): Export format, &quot;onnx&quot; (default).</li>
+<li>quantize (bool): Whether to quantize the model.</li>
+<li>device (str): Device for export.</li>
+</ul>
+<h4>Returns:</h4>
+<ul>
+<li><code>str</code> — Path to the exported model directory.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def export(self, input=None, **cfg):
+ &quot;&quot;&quot;Export model to ONNX format.
+
+ Creates a deep copy of the model to isolate ONNX operator monkey-patching,
+ then runs torch.onnx.export. The original model remains usable after export.
+
+ Args:
+ input: Sample input for tracing (auto-generated if None).
+ **cfg: Export parameters:
+ - type (str): Export format, &quot;onnx&quot; (default).
+ - quantize (bool): Whether to quantize the model.
+ - device (str): Device for export.
+
+ Returns:
+ str: Path to the exported model directory.
+ &quot;&quot;&quot;
+ &quot;&quot;&quot;
+
+ :param input:
+ :param type:
+ :param quantize:
+ :param fallback_num:
+ :param calib_num:
+ :param opset_version:
+ :param cfg:
+ :return:
+ &quot;&quot;&quot;
+
+ device = cfg.get(&quot;device&quot;, &quot;cpu&quot;)
+
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/auto/auto_model.py#L911" target="_blank" class="src-more">View full source on GitHub →</a></details></div>
+<div class="api-detail" id="e9"><span class="dtag badge-c">class</span><h2>RegisterTables</h2><div class="dmod">funasr.register.register · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/register.py#L8" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Registry system for classes.</p>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code>class RegisterTables:
+ &quot;&quot;&quot;Registry system for classes.&quot;&quot;&quot;
+
+ model_classes = {}
+ frontend_classes = {}
+ specaug_classes = {}
+ normalize_classes = {}
+ encoder_classes = {}
+ decoder_classes = {}
+ joint_network_classes = {}
+ predictor_classes = {}
+ stride_conv_classes = {}
+ tokenizer_classes = {}
+ dataloader_classes = {}
+ batch_sampler_classes = {}
+ dataset_classes = {}
+ index_ds_classes = {}
+
+ def print(self, key: str = None) -&gt; None:
+ &quot;&quot;&quot;Print registered classes.&quot;&quot;&quot;
+ print(&quot;\ntables: \n&quot;)
+ fields = vars(self)
+ headers = [&quot;register name&quot;, &quot;class name&quot;, &quot;class location&quot;]
+ for classes_key, classes_dict in fields.items():
+ if classes_key.endswith(&quot;_meta&quot;) and (key is None or key in classes_key):
+ print(f&quot;----------- ** {classes_key.replace(&#x27;_meta&#x27;, &#x27;&#x27;)} ** --------------&quot;)
+ metas = []
+ for register_key, meta in classes_dict.items():
+ metas.append(meta)
+ metas.sort(key=lambda x: x[0])
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/register.py#L8" target="_blank" class="src-more">View full source on GitHub →</a></details><h3 class="mt">Methods</h3><div class="mblk"><code>.print(key)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/register.py#L26" target="_blank" class="gh-link">L26</a><div class="mdoc"><p>Print registered classes.</p>
+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def print(self, key: str = None) -&gt; None:
+ &quot;&quot;&quot;Print registered classes.&quot;&quot;&quot;
+ print(&quot;\ntables: \n&quot;)
+ fields = vars(self)
+ headers = [&quot;register name&quot;, &quot;class name&quot;, &quot;class location&quot;]
+ for classes_key, classes_dict in fields.items():
+ if classes_key.endswith(&quot;_meta&quot;) and (key is None or key in classes_key):
+ print(f&quot;----------- ** {classes_key.replace(&#x27;_meta&#x27;, &#x27;&#x27;)} ** --------------&quot;)
+ metas = []
+ for register_key, meta in classes_dict.items():
+ metas.append(meta)
+ metas.sort(key=lambda x: x[0])
+ data = [headers] + metas
+ col_widths = [max(len(str(item)) for item in col) for col in zip(*data)]
+
+ for row in data:
+ print(
+ &quot;| &quot;
+ + &quot; | &quot;.join(str(item).ljust(width) for item, width in zip(row, col_widths))
+ + &quot; |&quot;
+ )
+ print(&quot;\n&quot;)
+</code></pre></details></div><div class="mblk"><code>.register(register_tables_key, key)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/register.py#L49" target="_blank" class="gh-link">L49</a><div class="mdoc"><p>Decorator to register a class.</p>
+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def register(self, register_tables_key: str, key: str = None) -&gt; callable:
+ &quot;&quot;&quot;Decorator to register a class.&quot;&quot;&quot;
+
+ def decorator(target_class):
+ &quot;&quot;&quot;Decorator.
+
+ Args:
+ target_class: TODO.
+ &quot;&quot;&quot;
+ if not hasattr(self, register_tables_key):
+ setattr(self, register_tables_key, {})
+ logging.debug(f&quot;New registry table added: {register_tables_key}&quot;)
+
+ registry = getattr(self, register_tables_key)
+ registry_key = key if key is not None else target_class.__name__
+
+ if registry_key in registry:
+ logging.debug(
+ f&quot;Key {registry_key} already exists in {register_tables_key}, re-register&quot;
+ )
+
+ registry[registry_key] = target_class
+
+ register_tables_key_meta = register_tables_key + &quot;_meta&quot;
+ if not hasattr(self, register_tables_key_meta):
+ setattr(self, register_tables_key_meta, {})
+ registry_meta = getattr(self, register_tables_key_meta)
+
+ class_file = inspect.getfile(target_class)
+ class_line = inspect.getsourcelines(target_class)[1]
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/register.py#L49" target="_blank" class="src-more">View full source →</a></details></div></div>
+<div class="api-detail" id="e10"><span class="dtag badge-m">method</span><h2><code>RegisterTables.print(key)</code></h2><div class="dmod">funasr.register.register.RegisterTables · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/register.py#L26" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Print registered classes.</p>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def print(self, key: str = None) -&gt; None:
+ &quot;&quot;&quot;Print registered classes.&quot;&quot;&quot;
+ print(&quot;\ntables: \n&quot;)
+ fields = vars(self)
+ headers = [&quot;register name&quot;, &quot;class name&quot;, &quot;class location&quot;]
+ for classes_key, classes_dict in fields.items():
+ if classes_key.endswith(&quot;_meta&quot;) and (key is None or key in classes_key):
+ print(f&quot;----------- ** {classes_key.replace(&#x27;_meta&#x27;, &#x27;&#x27;)} ** --------------&quot;)
+ metas = []
+ for register_key, meta in classes_dict.items():
+ metas.append(meta)
+ metas.sort(key=lambda x: x[0])
+ data = [headers] + metas
+ col_widths = [max(len(str(item)) for item in col) for col in zip(*data)]
+
+ for row in data:
+ print(
+ &quot;| &quot;
+ + &quot; | &quot;.join(str(item).ljust(width) for item, width in zip(row, col_widths))
+ + &quot; |&quot;
+ )
+ print(&quot;\n&quot;)
+</code></pre></details></div>
+<div class="api-detail" id="e11"><span class="dtag badge-m">method</span><h2><code>RegisterTables.register(register_tables_key, key)</code></h2><div class="dmod">funasr.register.register.RegisterTables · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/register.py#L49" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Decorator to register a class.</p>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def register(self, register_tables_key: str, key: str = None) -&gt; callable:
+ &quot;&quot;&quot;Decorator to register a class.&quot;&quot;&quot;
+
+ def decorator(target_class):
+ &quot;&quot;&quot;Decorator.
+
+ Args:
+ target_class: TODO.
+ &quot;&quot;&quot;
+ if not hasattr(self, register_tables_key):
+ setattr(self, register_tables_key, {})
+ logging.debug(f&quot;New registry table added: {register_tables_key}&quot;)
+
+ registry = getattr(self, register_tables_key)
+ registry_key = key if key is not None else target_class.__name__
+
+ if registry_key in registry:
+ logging.debug(
+ f&quot;Key {registry_key} already exists in {register_tables_key}, re-register&quot;
+ )
+
+ registry[registry_key] = target_class
+
+ register_tables_key_meta = register_tables_key + &quot;_meta&quot;
+ if not hasattr(self, register_tables_key_meta):
+ setattr(self, register_tables_key_meta, {})
+ registry_meta = getattr(self, register_tables_key_meta)
+
+ class_file = inspect.getfile(target_class)
+ class_line = inspect.getsourcelines(target_class)[1]
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/register.py#L49" target="_blank" class="src-more">View full source on GitHub →</a></details></div>
+<div class="api-detail" id="e12"><span class="dtag badge-c">class</span><h2>BAT</h2><div class="dmod">funasr.models.bat · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/bat/model.py#L41" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><ul>
+<li><code>BAT (Boundary-Aware Transducer)</code> — Low-latency RNN-T model with boundary detection.</li>
+</ul>
+<p>Inherits from Transducer. Designed for streaming ASR with reduced latency</p>
+<p>by predicting token boundaries explicitly.</p>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code>class BAT(Transducer):
+ &quot;&quot;&quot;BAT (Boundary-Aware Transducer): Low-latency RNN-T model with boundary detection.
+
+ Inherits from Transducer. Designed for streaming ASR with reduced latency
+ by predicting token boundaries explicitly.
+ &quot;&quot;&quot;
+
+ pass
+</code></pre></details></div>
+<div class="api-detail" id="e13"><span class="dtag badge-c">class</span><h2>BiCifParaformer</h2><div class="dmod">funasr.models.bicif_paraformer · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/bicif_paraformer/model.py#L43" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><ul>
+<li><code>BiCifParaformer</code> — Paraformer with Bidirectional CIF for Timestamp Prediction.</li>
+</ul>
+<p>Extends Paraformer with a second CIF predictor that provides accurate</p>
+<ul>
+<li><code>character</code> — level timestamp prediction alongside ASR. Uses bidirectional</li>
+</ul>
+<p>information flow for better alignment between audio frames and text tokens.</p>
+<p>Reference:</p>
+<ul>
+<li>FunASR: A Fundamental End-to-End Speech Recognition Toolkit (https://arxiv.org/abs/2305.11013)</li>
+<li>Achieving timestamp prediction while recognizing with non-autoregressive end-to-end ASR model</li>
+</ul>
+<p>(https://arxiv.org/abs/2301.12343)</p>
+<h4>Output:</h4>
+<p>{&quot;key&quot;: str, &quot;text&quot;: str, &quot;timestamp&quot;: [[start_ms, end_ms], ...]}</p>
+<ul>
+<li><code>Author</code> — Speech Lab of DAMO Academy, Alibaba Group</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code>class BiCifParaformer(Paraformer):
+ &quot;&quot;&quot;BiCifParaformer: Paraformer with Bidirectional CIF for Timestamp Prediction.
+
+ Extends Paraformer with a second CIF predictor that provides accurate
+ character-level timestamp prediction alongside ASR. Uses bidirectional
+ information flow for better alignment between audio frames and text tokens.
+
+ Reference:
+ - FunASR: A Fundamental End-to-End Speech Recognition Toolkit (https://arxiv.org/abs/2305.11013)
+ - Achieving timestamp prediction while recognizing with non-autoregressive end-to-end ASR model
+ (https://arxiv.org/abs/2301.12343)
+
+ Output:
+ {&quot;key&quot;: str, &quot;text&quot;: str, &quot;timestamp&quot;: [[start_ms, end_ms], ...]}
+
+ Author: Speech Lab of DAMO Academy, Alibaba Group
+ &quot;&quot;&quot;
+
+ def __init__(
+ self,
+ *args,
+ **kwargs,
+ ):
+ &quot;&quot;&quot;Initialize BiCifParaformer.
+
+ Args:
+ *args: Variable positional arguments.
+ **kwargs: Additional keyword arguments.
+ &quot;&quot;&quot;
+ super().__init__(*args, **kwargs)
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/bicif_paraformer/model.py#L43" target="_blank" class="src-more">View full source on GitHub →</a></details><h3 class="mt">Methods</h3><div class="mblk"><code>.calc_predictor(encoder_out, encoder_out_lens)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/bicif_paraformer/model.py#L162" target="_blank" class="gh-link">L162</a><div class="mdoc"><p>Calc predictor.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>encoder_out</code> — Encoder output tensor.</li>
+<li><code>encoder_out_lens</code> — Encoder output lengths.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def calc_predictor(self, encoder_out, encoder_out_lens):
+ &quot;&quot;&quot;Calc predictor.
+
+ Args:
+ encoder_out: Encoder output tensor.
+ encoder_out_lens: Encoder output lengths.
+ &quot;&quot;&quot;
+ encoder_out_mask = (
+ ~make_pad_mask(encoder_out_lens, maxlen=encoder_out.size(1))[:, None, :]
+ ).to(encoder_out.device)
+ pre_acoustic_embeds, pre_token_length, alphas, pre_peak_index, pre_token_length2 = (
+ self.predictor(encoder_out, None, encoder_out_mask, ignore_id=self.ignore_id)
+ )
+ return pre_acoustic_embeds, pre_token_length, alphas, pre_peak_index
+</code></pre></details></div><div class="mblk"><code>.calc_predictor_timestamp(encoder_out, encoder_out_lens, token_num)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/bicif_paraformer/model.py#L177" target="_blank" class="gh-link">L177</a><div class="mdoc"><p>Calc predictor timestamp.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>encoder_out</code> — Encoder output tensor.</li>
+<li><code>encoder_out_lens</code> — Encoder output lengths.</li>
+<li><code>token_num</code> — TODO.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def calc_predictor_timestamp(self, encoder_out, encoder_out_lens, token_num):
+ &quot;&quot;&quot;Calc predictor timestamp.
+
+ Args:
+ encoder_out: Encoder output tensor.
+ encoder_out_lens: Encoder output lengths.
+ token_num: TODO.
+ &quot;&quot;&quot;
+ encoder_out_mask = (
+ ~make_pad_mask(encoder_out_lens, maxlen=encoder_out.size(1))[:, None, :]
+ ).to(encoder_out.device)
+ ds_alphas, ds_cif_peak, us_alphas, us_peaks = self.predictor.get_upsample_timestamp(
+ encoder_out, encoder_out_mask, token_num
+ )
+ return ds_alphas, ds_cif_peak, us_alphas, us_peaks
+</code></pre></details></div><div class="mblk"><code>.forward(speech, speech_lengths, text, text_lengths, **kwargs)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/bicif_paraformer/model.py#L193" target="_blank" class="gh-link">L193</a><div class="mdoc"><p>Frontend + Encoder + Decoder + Calc loss</p>
+<h4>Args:</h4>
+<ul>
+<li><code>speech</code> — (Batch, Length, ...)</li>
+<li><code>speech_lengths</code> — (Batch, )</li>
+<li><code>text</code> — (Batch, Length)</li>
+<li><code>text_lengths</code> — (Batch,)</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def forward(
+ self,
+ speech: torch.Tensor,
+ speech_lengths: torch.Tensor,
+ text: torch.Tensor,
+ text_lengths: torch.Tensor,
+ **kwargs,
+ ) -&gt; Tuple[torch.Tensor, Dict[str, torch.Tensor], torch.Tensor]:
+ &quot;&quot;&quot;Frontend + Encoder + Decoder + Calc loss
+ Args:
+ speech: (Batch, Length, ...)
+ speech_lengths: (Batch, )
+ text: (Batch, Length)
+ text_lengths: (Batch,)
+ &quot;&quot;&quot;
+ if len(text_lengths.size()) &gt; 1:
+ text_lengths = text_lengths[:, 0]
+ if len(speech_lengths.size()) &gt; 1:
+ speech_lengths = speech_lengths[:, 0]
+
+ batch_size = speech.shape[0]
+
+ # Encoder
+ encoder_out, encoder_out_lens = self.encode(speech, speech_lengths)
+
+ loss_ctc, cer_ctc = None, None
+ loss_pre = None
+ stats = dict()
+
+ # decoder: CTC branch
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/bicif_paraformer/model.py#L193" target="_blank" class="src-more">View full source →</a></details></div><div class="mblk"><code>.inference(data_in, data_lengths, key, tokenizer, frontend, **kwargs)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/bicif_paraformer/model.py#L271" target="_blank" class="gh-link">L271</a><div class="mdoc"><p>Run inference on input data.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>data_in</code> — Input data (audio samples, file paths, or text).</li>
+<li><code>data_lengths</code> — Lengths of each input sample in the batch.</li>
+<li><code>key</code> — Sample identifiers.</li>
+<li><code>tokenizer</code> — Tokenizer instance for text encoding/decoding.</li>
+<li><code>frontend</code> — Audio frontend for feature extraction.</li>
+<li><code>**kwargs</code> — Additional keyword arguments.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def inference(
+ self,
+ data_in,
+ data_lengths=None,
+ key: list = None,
+ tokenizer=None,
+ frontend=None,
+ **kwargs,
+ ):
+
+ # init beamsearch
+ &quot;&quot;&quot;Run inference on input data.
+
+ Args:
+ data_in: Input data (audio samples, file paths, or text).
+ data_lengths: Lengths of each input sample in the batch.
+ key: Sample identifiers.
+ tokenizer: Tokenizer instance for text encoding/decoding.
+ frontend: Audio frontend for feature extraction.
+ **kwargs: Additional keyword arguments.
+ &quot;&quot;&quot;
+ is_use_ctc = kwargs.get(&quot;decoding_ctc_weight&quot;, 0.0) &gt; 0.00001 and self.ctc != None
+ is_use_lm = (
+ kwargs.get(&quot;lm_weight&quot;, 0.0) &gt; 0.00001 and kwargs.get(&quot;lm_file&quot;, None) is not None
+ )
+ if self.beam_search is None and (is_use_lm or is_use_ctc):
+ logging.info(&quot;enable beam_search&quot;)
+ self.init_beam_search(**kwargs)
+ self.nbest = kwargs.get(&quot;nbest&quot;, 1)
+
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/bicif_paraformer/model.py#L271" target="_blank" class="src-more">View full source →</a></details></div><div class="mblk"><code>.export(**kwargs)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/bicif_paraformer/model.py#L430" target="_blank" class="gh-link">L430</a><div class="mdoc"><p>Export.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>**kwargs</code> — Additional keyword arguments.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def export(self, **kwargs):
+ &quot;&quot;&quot;Export.
+
+ Args:
+ **kwargs: Additional keyword arguments.
+ &quot;&quot;&quot;
+ from .export_meta import export_rebuild_model
+
+ if &quot;max_seq_len&quot; not in kwargs:
+ kwargs[&quot;max_seq_len&quot;] = 512
+ models = export_rebuild_model(model=self, **kwargs)
+ return models
+</code></pre></details></div></div>
+<div class="api-detail" id="e14"><span class="dtag badge-m">method</span><h2><code>BiCifParaformer.calc_predictor(encoder_out, encoder_out_lens)</code></h2><div class="dmod">funasr.models.bicif_paraformer.BiCifParaformer · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/bicif_paraformer/model.py#L162" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Calc predictor.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>encoder_out</code> — Encoder output tensor.</li>
+<li><code>encoder_out_lens</code> — Encoder output lengths.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def calc_predictor(self, encoder_out, encoder_out_lens):
+ &quot;&quot;&quot;Calc predictor.
+
+ Args:
+ encoder_out: Encoder output tensor.
+ encoder_out_lens: Encoder output lengths.
+ &quot;&quot;&quot;
+ encoder_out_mask = (
+ ~make_pad_mask(encoder_out_lens, maxlen=encoder_out.size(1))[:, None, :]
+ ).to(encoder_out.device)
+ pre_acoustic_embeds, pre_token_length, alphas, pre_peak_index, pre_token_length2 = (
+ self.predictor(encoder_out, None, encoder_out_mask, ignore_id=self.ignore_id)
+ )
+ return pre_acoustic_embeds, pre_token_length, alphas, pre_peak_index
+</code></pre></details></div>
+<div class="api-detail" id="e15"><span class="dtag badge-m">method</span><h2><code>BiCifParaformer.calc_predictor_timestamp(encoder_out, encoder_out_lens, token_num)</code></h2><div class="dmod">funasr.models.bicif_paraformer.BiCifParaformer · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/bicif_paraformer/model.py#L177" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Calc predictor timestamp.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>encoder_out</code> — Encoder output tensor.</li>
+<li><code>encoder_out_lens</code> — Encoder output lengths.</li>
+<li><code>token_num</code> — TODO.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def calc_predictor_timestamp(self, encoder_out, encoder_out_lens, token_num):
+ &quot;&quot;&quot;Calc predictor timestamp.
+
+ Args:
+ encoder_out: Encoder output tensor.
+ encoder_out_lens: Encoder output lengths.
+ token_num: TODO.
+ &quot;&quot;&quot;
+ encoder_out_mask = (
+ ~make_pad_mask(encoder_out_lens, maxlen=encoder_out.size(1))[:, None, :]
+ ).to(encoder_out.device)
+ ds_alphas, ds_cif_peak, us_alphas, us_peaks = self.predictor.get_upsample_timestamp(
+ encoder_out, encoder_out_mask, token_num
+ )
+ return ds_alphas, ds_cif_peak, us_alphas, us_peaks
+</code></pre></details></div>
+<div class="api-detail" id="e16"><span class="dtag badge-m">method</span><h2><code>BiCifParaformer.forward(speech, speech_lengths, text, text_lengths, **kwargs)</code></h2><div class="dmod">funasr.models.bicif_paraformer.BiCifParaformer · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/bicif_paraformer/model.py#L193" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Frontend + Encoder + Decoder + Calc loss</p>
+<h4>Args:</h4>
+<ul>
+<li><code>speech</code> — (Batch, Length, ...)</li>
+<li><code>speech_lengths</code> — (Batch, )</li>
+<li><code>text</code> — (Batch, Length)</li>
+<li><code>text_lengths</code> — (Batch,)</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def forward(
+ self,
+ speech: torch.Tensor,
+ speech_lengths: torch.Tensor,
+ text: torch.Tensor,
+ text_lengths: torch.Tensor,
+ **kwargs,
+ ) -&gt; Tuple[torch.Tensor, Dict[str, torch.Tensor], torch.Tensor]:
+ &quot;&quot;&quot;Frontend + Encoder + Decoder + Calc loss
+ Args:
+ speech: (Batch, Length, ...)
+ speech_lengths: (Batch, )
+ text: (Batch, Length)
+ text_lengths: (Batch,)
+ &quot;&quot;&quot;
+ if len(text_lengths.size()) &gt; 1:
+ text_lengths = text_lengths[:, 0]
+ if len(speech_lengths.size()) &gt; 1:
+ speech_lengths = speech_lengths[:, 0]
+
+ batch_size = speech.shape[0]
+
+ # Encoder
+ encoder_out, encoder_out_lens = self.encode(speech, speech_lengths)
+
+ loss_ctc, cer_ctc = None, None
+ loss_pre = None
+ stats = dict()
+
+ # decoder: CTC branch
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/bicif_paraformer/model.py#L193" target="_blank" class="src-more">View full source on GitHub →</a></details></div>
+<div class="api-detail" id="e17"><span class="dtag badge-m">method</span><h2><code>BiCifParaformer.inference(data_in, data_lengths, key, tokenizer, frontend, **kwargs)</code></h2><div class="dmod">funasr.models.bicif_paraformer.BiCifParaformer · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/bicif_paraformer/model.py#L271" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Run inference on input data.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>data_in</code> — Input data (audio samples, file paths, or text).</li>
+<li><code>data_lengths</code> — Lengths of each input sample in the batch.</li>
+<li><code>key</code> — Sample identifiers.</li>
+<li><code>tokenizer</code> — Tokenizer instance for text encoding/decoding.</li>
+<li><code>frontend</code> — Audio frontend for feature extraction.</li>
+<li><code>**kwargs</code> — Additional keyword arguments.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def inference(
+ self,
+ data_in,
+ data_lengths=None,
+ key: list = None,
+ tokenizer=None,
+ frontend=None,
+ **kwargs,
+ ):
+
+ # init beamsearch
+ &quot;&quot;&quot;Run inference on input data.
+
+ Args:
+ data_in: Input data (audio samples, file paths, or text).
+ data_lengths: Lengths of each input sample in the batch.
+ key: Sample identifiers.
+ tokenizer: Tokenizer instance for text encoding/decoding.
+ frontend: Audio frontend for feature extraction.
+ **kwargs: Additional keyword arguments.
+ &quot;&quot;&quot;
+ is_use_ctc = kwargs.get(&quot;decoding_ctc_weight&quot;, 0.0) &gt; 0.00001 and self.ctc != None
+ is_use_lm = (
+ kwargs.get(&quot;lm_weight&quot;, 0.0) &gt; 0.00001 and kwargs.get(&quot;lm_file&quot;, None) is not None
+ )
+ if self.beam_search is None and (is_use_lm or is_use_ctc):
+ logging.info(&quot;enable beam_search&quot;)
+ self.init_beam_search(**kwargs)
+ self.nbest = kwargs.get(&quot;nbest&quot;, 1)
+
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/bicif_paraformer/model.py#L271" target="_blank" class="src-more">View full source on GitHub →</a></details></div>
+<div class="api-detail" id="e18"><span class="dtag badge-m">method</span><h2><code>BiCifParaformer.export(**kwargs)</code></h2><div class="dmod">funasr.models.bicif_paraformer.BiCifParaformer · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/bicif_paraformer/model.py#L430" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Export.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>**kwargs</code> — Additional keyword arguments.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def export(self, **kwargs):
+ &quot;&quot;&quot;Export.
+
+ Args:
+ **kwargs: Additional keyword arguments.
+ &quot;&quot;&quot;
+ from .export_meta import export_rebuild_model
+
+ if &quot;max_seq_len&quot; not in kwargs:
+ kwargs[&quot;max_seq_len&quot;] = 512
+ models = export_rebuild_model(model=self, **kwargs)
+ return models
+</code></pre></details></div>
+<div class="api-detail" id="e19"><span class="dtag badge-c">class</span><h2>Branchformer</h2><div class="dmod">funasr.models.branchformer · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/branchformer/model.py#L8" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><ul>
+<li><code>Branchformer</code> — Parallel branch encoder architecture.</li>
+</ul>
+<p>Uses parallel branches of self-attention and convolution that are</p>
+<p>merged via concatenation. Alternative to Conformer with similar accuracy.</p>
+<p>Inherits Transformer pipeline for training and inference.</p>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code>class Branchformer(Transformer):
+ &quot;&quot;&quot;Branchformer: Parallel branch encoder architecture.
+
+ Uses parallel branches of self-attention and convolution that are
+ merged via concatenation. Alternative to Conformer with similar accuracy.
+
+ Inherits Transformer pipeline for training and inference.
+ &quot;&quot;&quot;
+
+ def __init__(
+ self,
+ *args,
+ **kwargs,
+ ):
+
+ &quot;&quot;&quot;Initialize Branchformer.
+
+ Args:
+ *args: Variable positional arguments.
+ **kwargs: Additional keyword arguments.
+ &quot;&quot;&quot;
+ super().__init__(*args, **kwargs)
+</code></pre></details></div>
+<div class="api-detail" id="e20"><span class="dtag badge-c">class</span><h2>CAMPPlus</h2><div class="dmod">funasr.models.campplus · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/campplus/model.py#L43" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>CAM++ Speaker Verification Model.</p>
+<p>Extracts fixed-dimensional speaker embeddings from variable-length audio.</p>
+<p>Used for speaker verification and speaker diarization pipelines.</p>
+<ul>
+<li><code>Output</code> — 192-dimensional speaker embedding per utterance.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code>class CAMPPlus(torch.nn.Module):
+ &quot;&quot;&quot;CAM++ Speaker Verification Model.
+
+ Extracts fixed-dimensional speaker embeddings from variable-length audio.
+ Used for speaker verification and speaker diarization pipelines.
+
+ Output: 192-dimensional speaker embedding per utterance.
+ &quot;&quot;&quot;
+
+ def __init__(
+ self,
+ feat_dim=80,
+ embedding_size=192,
+ growth_rate=32,
+ bn_size=4,
+ init_channels=128,
+ config_str=&quot;batchnorm-relu&quot;,
+ memory_efficient=True,
+ output_level=&quot;segment&quot;,
+ **kwargs,
+ ):
+ &quot;&quot;&quot;Initialize CAMPPlus.
+
+ Args:
+ feat_dim: Size/dimension parameter.
+ embedding_size: Size/dimension parameter.
+ growth_rate: TODO.
+ bn_size: Size/dimension parameter.
+ init_channels: TODO.
+ config_str: TODO.
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/campplus/model.py#L43" target="_blank" class="src-more">View full source on GitHub →</a></details><h3 class="mt">Methods</h3><div class="mblk"><code>.forward(x)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/campplus/model.py#L141" target="_blank" class="gh-link">L141</a><div class="mdoc"><p>Extract speaker embedding from fbank features.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>x (Tensor)</code> — Input fbank features, shape (batch, time, feat_dim).</li>
+</ul>
+<h4>Returns:</h4>
+<ul>
+<li><code>Tensor</code> — Speaker embedding, shape (batch, embedding_size) for segment level,</li>
+<li class="sub">or (batch, time, channels) for frame level.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def forward(self, x):
+ &quot;&quot;&quot;Extract speaker embedding from fbank features.
+
+ Args:
+ x (Tensor): Input fbank features, shape (batch, time, feat_dim).
+
+ Returns:
+ Tensor: Speaker embedding, shape (batch, embedding_size) for segment level,
+ or (batch, time, channels) for frame level.
+ &quot;&quot;&quot;
+ x = x.permute(0, 2, 1) # (B,T,F) =&gt; (B,F,T)
+ x = self.head(x)
+ x = self.xvector(x)
+ if self.output_level == &quot;frame&quot;:
+ x = x.transpose(1, 2)
+ return x
+</code></pre></details></div><div class="mblk"><code>.inference(data_in, data_lengths, key, tokenizer, frontend, **kwargs)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/campplus/model.py#L158" target="_blank" class="gh-link">L158</a><div class="mdoc"><p>Run speaker embedding extraction on audio input.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>data_in</code> — Audio input (file path, numpy array, or list).</li>
+<li><code>data_lengths</code> — Not used.</li>
+<li><code>key (list)</code> — Sample identifiers.</li>
+<li><code>tokenizer</code> — Not used.</li>
+<li><code>frontend</code> — Not used.</li>
+<li><code>**kwargs</code> — Must include &#x27;device&#x27; (str) and optional &#x27;fs&#x27; (int, default 16000).</li>
+</ul>
+<h4>Returns:</h4>
+<ul>
+<li><code>tuple</code> — (results, meta_data) where results is</li>
+<li class="sub">[{&quot;spk_embedding&quot;: Tensor of shape (1, 192)}]</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def inference(
+ self,
+ data_in,
+ data_lengths=None,
+ key: list = None,
+ tokenizer=None,
+ frontend=None,
+ **kwargs,
+ ):
+ &quot;&quot;&quot;Run speaker embedding extraction on audio input.
+
+ Args:
+ data_in: Audio input (file path, numpy array, or list).
+ data_lengths: Not used.
+ key (list): Sample identifiers.
+ tokenizer: Not used.
+ frontend: Not used.
+ **kwargs: Must include &#x27;device&#x27; (str) and optional &#x27;fs&#x27; (int, default 16000).
+
+ Returns:
+ tuple: (results, meta_data) where results is
+ [{&quot;spk_embedding&quot;: Tensor of shape (1, 192)}]
+ &quot;&quot;&quot;
+ # extract fbank feats
+ meta_data = {}
+ time1 = time.perf_counter()
+ audio_sample_list = load_audio_text_image_video(
+ data_in, fs=16000, audio_fs=kwargs.get(&quot;fs&quot;, 16000), data_type=&quot;sound&quot;
+ )
+ time2 = time.perf_counter()
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/campplus/model.py#L158" target="_blank" class="src-more">View full source →</a></details></div></div>
+<div class="api-detail" id="e21"><span class="dtag badge-m">method</span><h2><code>CAMPPlus.forward(x)</code></h2><div class="dmod">funasr.models.campplus.CAMPPlus · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/campplus/model.py#L141" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Extract speaker embedding from fbank features.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>x (Tensor)</code> — Input fbank features, shape (batch, time, feat_dim).</li>
+</ul>
+<h4>Returns:</h4>
+<ul>
+<li><code>Tensor</code> — Speaker embedding, shape (batch, embedding_size) for segment level,</li>
+<li class="sub">or (batch, time, channels) for frame level.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def forward(self, x):
+ &quot;&quot;&quot;Extract speaker embedding from fbank features.
+
+ Args:
+ x (Tensor): Input fbank features, shape (batch, time, feat_dim).
+
+ Returns:
+ Tensor: Speaker embedding, shape (batch, embedding_size) for segment level,
+ or (batch, time, channels) for frame level.
+ &quot;&quot;&quot;
+ x = x.permute(0, 2, 1) # (B,T,F) =&gt; (B,F,T)
+ x = self.head(x)
+ x = self.xvector(x)
+ if self.output_level == &quot;frame&quot;:
+ x = x.transpose(1, 2)
+ return x
+</code></pre></details></div>
+<div class="api-detail" id="e22"><span class="dtag badge-m">method</span><h2><code>CAMPPlus.inference(data_in, data_lengths, key, tokenizer, frontend, **kwargs)</code></h2><div class="dmod">funasr.models.campplus.CAMPPlus · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/campplus/model.py#L158" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Run speaker embedding extraction on audio input.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>data_in</code> — Audio input (file path, numpy array, or list).</li>
+<li><code>data_lengths</code> — Not used.</li>
+<li><code>key (list)</code> — Sample identifiers.</li>
+<li><code>tokenizer</code> — Not used.</li>
+<li><code>frontend</code> — Not used.</li>
+<li><code>**kwargs</code> — Must include &#x27;device&#x27; (str) and optional &#x27;fs&#x27; (int, default 16000).</li>
+</ul>
+<h4>Returns:</h4>
+<ul>
+<li><code>tuple</code> — (results, meta_data) where results is</li>
+<li class="sub">[{&quot;spk_embedding&quot;: Tensor of shape (1, 192)}]</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def inference(
+ self,
+ data_in,
+ data_lengths=None,
+ key: list = None,
+ tokenizer=None,
+ frontend=None,
+ **kwargs,
+ ):
+ &quot;&quot;&quot;Run speaker embedding extraction on audio input.
+
+ Args:
+ data_in: Audio input (file path, numpy array, or list).
+ data_lengths: Not used.
+ key (list): Sample identifiers.
+ tokenizer: Not used.
+ frontend: Not used.
+ **kwargs: Must include &#x27;device&#x27; (str) and optional &#x27;fs&#x27; (int, default 16000).
+
+ Returns:
+ tuple: (results, meta_data) where results is
+ [{&quot;spk_embedding&quot;: Tensor of shape (1, 192)}]
+ &quot;&quot;&quot;
+ # extract fbank feats
+ meta_data = {}
+ time1 = time.perf_counter()
+ audio_sample_list = load_audio_text_image_video(
+ data_in, fs=16000, audio_fs=kwargs.get(&quot;fs&quot;, 16000), data_type=&quot;sound&quot;
+ )
+ time2 = time.perf_counter()
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/campplus/model.py#L158" target="_blank" class="src-more">View full source on GitHub →</a></details></div>
+<div class="api-detail" id="e23"><span class="dtag badge-c">class</span><h2>Conformer</h2><div class="dmod">funasr.models.conformer · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/conformer/model.py#L10" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><ul>
+<li><code>Conformer</code> — CTC-attention hybrid encoder-decoder model.</li>
+</ul>
+<p>Combines convolution and self-attention in the encoder for better</p>
+<p>local and global context modeling. Inherits full Transformer pipeline</p>
+<p>(CTC + attention decoder + beam search).</p>
+<ul>
+<li><code>Output</code> — {&quot;key&quot;: str, &quot;text&quot;: str}</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code>class Conformer(Transformer):
+ &quot;&quot;&quot;Conformer: CTC-attention hybrid encoder-decoder model.
+
+ Combines convolution and self-attention in the encoder for better
+ local and global context modeling. Inherits full Transformer pipeline
+ (CTC + attention decoder + beam search).
+
+ Output: {&quot;key&quot;: str, &quot;text&quot;: str}
+ &quot;&quot;&quot;
+
+ def __init__(
+ self,
+ *args,
+ **kwargs,
+ ):
+
+ &quot;&quot;&quot;Initialize Conformer.
+
+ Args:
+ *args: Variable positional arguments.
+ **kwargs: Additional keyword arguments.
+ &quot;&quot;&quot;
+ super().__init__(*args, **kwargs)
+</code></pre></details></div>
+<div class="api-detail" id="e24"><span class="dtag badge-c">class</span><h2>Conformer</h2><div class="dmod">funasr.models.conformer_rwkv · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/conformer_rwkv/model.py#L10" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><ul>
+<li><code>CTC</code> — attention hybrid Encoder-Decoder model</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code>class Conformer(Transformer):
+ &quot;&quot;&quot;CTC-attention hybrid Encoder-Decoder model&quot;&quot;&quot;
+
+ def __init__(
+ self,
+ *args,
+ **kwargs,
+ ):
+
+ &quot;&quot;&quot;Initialize Conformer.
+
+ Args:
+ *args: Variable positional arguments.
+ **kwargs: Additional keyword arguments.
+ &quot;&quot;&quot;
+ super().__init__(*args, **kwargs)
+</code></pre></details></div>
+<div class="api-detail" id="e25"><span class="dtag badge-c">class</span><h2>ContextualParaformer</h2><div class="dmod">funasr.models.contextual_paraformer · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/contextual_paraformer/model.py#L46" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><ul>
+<li><code>ContextualParaformer</code> — Paraformer with hotword/context biasing.</li>
+</ul>
+<p>Extends Paraformer with a context encoder that incorporates user-defined</p>
+<p>hotwords/keywords to boost recognition of domain-specific terms.</p>
+<ul>
+<li><code>Usage</code> — Pass hotwords via generate(hotword=&#x27;term1 term2&#x27;).</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code>class ContextualParaformer(Paraformer):
+ &quot;&quot;&quot;ContextualParaformer: Paraformer with hotword/context biasing.
+
+ Extends Paraformer with a context encoder that incorporates user-defined
+ hotwords/keywords to boost recognition of domain-specific terms.
+
+ Usage: Pass hotwords via generate(hotword=&#x27;term1 term2&#x27;).
+ &quot;&quot;&quot;
+
+ def __init__(
+ self,
+ *args,
+ **kwargs,
+ ):
+ &quot;&quot;&quot;Initialize ContextualParaformer.
+
+ Args:
+ *args: Variable positional arguments.
+ **kwargs: Additional keyword arguments.
+ &quot;&quot;&quot;
+ super().__init__(*args, **kwargs)
+
+ self.target_buffer_length = kwargs.get(&quot;target_buffer_length&quot;, -1)
+ inner_dim = kwargs.get(&quot;inner_dim&quot;, 256)
+ bias_encoder_type = kwargs.get(&quot;bias_encoder_type&quot;, &quot;lstm&quot;)
+ use_decoder_embedding = kwargs.get(&quot;use_decoder_embedding&quot;, False)
+ crit_attn_weight = kwargs.get(&quot;crit_attn_weight&quot;, 0.0)
+ crit_attn_smooth = kwargs.get(&quot;crit_attn_smooth&quot;, 0.0)
+ bias_encoder_dropout_rate = kwargs.get(&quot;bias_encoder_dropout_rate&quot;, 0.0)
+
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/contextual_paraformer/model.py#L46" target="_blank" class="src-more">View full source on GitHub →</a></details><h3 class="mt">Methods</h3><div class="mblk"><code>.forward(speech, speech_lengths, text, text_lengths, **kwargs)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/contextual_paraformer/model.py#L96" target="_blank" class="gh-link">L96</a><div class="mdoc"><p>Frontend + Encoder + Decoder + Calc loss</p>
+<h4>Args:</h4>
+<ul>
+<li><code>speech</code> — (Batch, Length, ...)</li>
+<li><code>speech_lengths</code> — (Batch, )</li>
+<li><code>text</code> — (Batch, Length)</li>
+<li><code>text_lengths</code> — (Batch,)</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def forward(
+ self,
+ speech: torch.Tensor,
+ speech_lengths: torch.Tensor,
+ text: torch.Tensor,
+ text_lengths: torch.Tensor,
+ **kwargs,
+ ) -&gt; Tuple[torch.Tensor, Dict[str, torch.Tensor], torch.Tensor]:
+ &quot;&quot;&quot;Frontend + Encoder + Decoder + Calc loss
+
+ Args:
+ speech: (Batch, Length, ...)
+ speech_lengths: (Batch, )
+ text: (Batch, Length)
+ text_lengths: (Batch,)
+ &quot;&quot;&quot;
+ text_lengths = text_lengths.squeeze()
+ speech_lengths = speech_lengths.squeeze()
+
+ batch_size = speech.shape[0]
+
+ hotword_pad = kwargs.get(&quot;hotword_pad&quot;)
+ hotword_lengths = kwargs.get(&quot;hotword_lengths&quot;)
+ # dha_pad = kwargs.get(&quot;dha_pad&quot;)
+
+ # 1. Encoder
+ encoder_out, encoder_out_lens = self.encode(speech, speech_lengths)
+
+ loss_ctc, cer_ctc = None, None
+
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/contextual_paraformer/model.py#L96" target="_blank" class="src-more">View full source →</a></details></div><div class="mblk"><code>.sampler(encoder_out, encoder_out_lens, ys_pad, ys_pad_lens, pre_acoustic_embeds, contextual_info)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/contextual_paraformer/model.py#L268" target="_blank" class="gh-link">L268</a><div class="mdoc"><p>Sampler.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>encoder_out</code> — Encoder output tensor.</li>
+<li><code>encoder_out_lens</code> — Encoder output lengths.</li>
+<li><code>ys_pad</code> — TODO.</li>
+<li><code>ys_pad_lens</code> — Lengths of ys_pad.</li>
+<li><code>pre_acoustic_embeds</code> — TODO.</li>
+<li><code>contextual_info</code> — TODO.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def sampler(
+ self,
+ encoder_out,
+ encoder_out_lens,
+ ys_pad,
+ ys_pad_lens,
+ pre_acoustic_embeds,
+ contextual_info,
+ ):
+ &quot;&quot;&quot;Sampler.
+
+ Args:
+ encoder_out: Encoder output tensor.
+ encoder_out_lens: Encoder output lengths.
+ ys_pad: TODO.
+ ys_pad_lens: Lengths of ys_pad.
+ pre_acoustic_embeds: TODO.
+ contextual_info: TODO.
+ &quot;&quot;&quot;
+ tgt_mask = (~make_pad_mask(ys_pad_lens, maxlen=ys_pad_lens.max())[:, :, None]).to(
+ ys_pad.device
+ )
+ ys_pad = ys_pad * tgt_mask[:, :, 0]
+ if self.share_embedding:
+ ys_pad_embed = self.decoder.output_layer.weight[ys_pad]
+ else:
+ ys_pad_embed = self.decoder.embed(ys_pad)
+ with torch.no_grad():
+ decoder_outs = self.decoder(
+ encoder_out,
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/contextual_paraformer/model.py#L268" target="_blank" class="src-more">View full source →</a></details></div><div class="mblk"><code>.cal_decoder_with_predictor(encoder_out, encoder_out_lens, sematic_embeds, ys_pad_lens, hw_list, clas_scale)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/contextual_paraformer/model.py#L331" target="_blank" class="gh-link">L331</a><div class="mdoc"><p>Cal decoder with predictor.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>encoder_out</code> — Encoder output tensor.</li>
+<li><code>encoder_out_lens</code> — Encoder output lengths.</li>
+<li><code>sematic_embeds</code> — TODO.</li>
+<li><code>ys_pad_lens</code> — Lengths of ys_pad.</li>
+<li><code>hw_list</code> — TODO.</li>
+<li><code>clas_scale</code> — TODO.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def cal_decoder_with_predictor(
+ self,
+ encoder_out,
+ encoder_out_lens,
+ sematic_embeds,
+ ys_pad_lens,
+ hw_list=None,
+ clas_scale=1.0,
+ ):
+ &quot;&quot;&quot;Cal decoder with predictor.
+
+ Args:
+ encoder_out: Encoder output tensor.
+ encoder_out_lens: Encoder output lengths.
+ sematic_embeds: TODO.
+ ys_pad_lens: Lengths of ys_pad.
+ hw_list: TODO.
+ clas_scale: TODO.
+ &quot;&quot;&quot;
+ if hw_list is None:
+ hw_list = [torch.Tensor([1]).long().to(encoder_out.device)] # empty hotword list
+ hw_list_pad = pad_list(hw_list, 0)
+ if self.use_decoder_embedding:
+ hw_embed = self.decoder.embed(hw_list_pad)
+ else:
+ hw_embed = self.bias_embed(hw_list_pad)
+ hw_embed, (h_n, _) = self.bias_encoder(hw_embed)
+ hw_embed = h_n.repeat(encoder_out.shape[0], 1, 1)
+ else:
+ hw_lengths = [len(i) for i in hw_list]
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/contextual_paraformer/model.py#L331" target="_blank" class="src-more">View full source →</a></details></div><div class="mblk"><code>.inference(data_in, data_lengths, key, tokenizer, frontend, **kwargs)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/contextual_paraformer/model.py#L387" target="_blank" class="gh-link">L387</a><div class="mdoc"><p>Run inference on input data.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>data_in</code> — Input data (audio samples, file paths, or text).</li>
+<li><code>data_lengths</code> — Lengths of each input sample in the batch.</li>
+<li><code>key</code> — Sample identifiers.</li>
+<li><code>tokenizer</code> — Tokenizer instance for text encoding/decoding.</li>
+<li><code>frontend</code> — Audio frontend for feature extraction.</li>
+<li><code>**kwargs</code> — Additional keyword arguments.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def inference(
+ self,
+ data_in,
+ data_lengths=None,
+ key: list = None,
+ tokenizer=None,
+ frontend=None,
+ **kwargs,
+ ):
+ # init beamsearch
+
+ &quot;&quot;&quot;Run inference on input data.
+
+ Args:
+ data_in: Input data (audio samples, file paths, or text).
+ data_lengths: Lengths of each input sample in the batch.
+ key: Sample identifiers.
+ tokenizer: Tokenizer instance for text encoding/decoding.
+ frontend: Audio frontend for feature extraction.
+ **kwargs: Additional keyword arguments.
+ &quot;&quot;&quot;
+ is_use_ctc = kwargs.get(&quot;decoding_ctc_weight&quot;, 0.0) &gt; 0.00001 and self.ctc != None
+ is_use_lm = (
+ kwargs.get(&quot;lm_weight&quot;, 0.0) &gt; 0.00001 and kwargs.get(&quot;lm_file&quot;, None) is not None
+ )
+ if self.beam_search is None and (is_use_lm or is_use_ctc):
+ logging.info(&quot;enable beam_search&quot;)
+ self.init_beam_search(**kwargs)
+ self.nbest = kwargs.get(&quot;nbest&quot;, 1)
+
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/contextual_paraformer/model.py#L387" target="_blank" class="src-more">View full source →</a></details></div><div class="mblk"><code>.generate_hotwords_list(hotword_list_or_file, tokenizer, frontend)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/contextual_paraformer/model.py#L534" target="_blank" class="gh-link">L534</a><div class="mdoc"><p>Generate hotwords list.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>hotword_list_or_file</code> — TODO.</li>
+<li><code>tokenizer</code> — Tokenizer instance for text encoding/decoding.</li>
+<li><code>frontend</code> — Audio frontend for feature extraction.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def generate_hotwords_list(self, hotword_list_or_file, tokenizer=None, frontend=None):
+ &quot;&quot;&quot;Generate hotwords list.
+
+ Args:
+ hotword_list_or_file: TODO.
+ tokenizer: Tokenizer instance for text encoding/decoding.
+ frontend: Audio frontend for feature extraction.
+ &quot;&quot;&quot;
+ def load_seg_dict(seg_dict_file):
+ &quot;&quot;&quot;Load seg dict.
+
+ Args:
+ seg_dict_file: TODO.
+ &quot;&quot;&quot;
+ seg_dict = {}
+ assert isinstance(seg_dict_file, str)
+ with open(seg_dict_file, &quot;r&quot;, encoding=&quot;utf8&quot;) as f:
+ lines = f.readlines()
+ for line in lines:
+ s = line.strip().split()
+ key = s[0]
+ value = s[1:]
+ seg_dict[key] = &quot; &quot;.join(value)
+ return seg_dict
+
+ def seg_tokenize(txt, seg_dict):
+ &quot;&quot;&quot;Seg tokenize.
+
+ Args:
+ txt: TODO.
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/contextual_paraformer/model.py#L534" target="_blank" class="src-more">View full source →</a></details></div><div class="mblk"><code>.export(**kwargs)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/contextual_paraformer/model.py#L659" target="_blank" class="gh-link">L659</a><div class="mdoc"><p>Export.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>**kwargs</code> — Additional keyword arguments.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def export(
+ self,
+ **kwargs,
+ ):
+ &quot;&quot;&quot;Export.
+
+ Args:
+ **kwargs: Additional keyword arguments.
+ &quot;&quot;&quot;
+ if &quot;max_seq_len&quot; not in kwargs:
+ kwargs[&quot;max_seq_len&quot;] = 512
+ from .export_meta import export_rebuild_model
+
+ models = export_rebuild_model(model=self, **kwargs)
+ return models
+</code></pre></details></div></div>
+<div class="api-detail" id="e26"><span class="dtag badge-m">method</span><h2><code>ContextualParaformer.forward(speech, speech_lengths, text, text_lengths, **kwargs)</code></h2><div class="dmod">funasr.models.contextual_paraformer.ContextualParaformer · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/contextual_paraformer/model.py#L96" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Frontend + Encoder + Decoder + Calc loss</p>
+<h4>Args:</h4>
+<ul>
+<li><code>speech</code> — (Batch, Length, ...)</li>
+<li><code>speech_lengths</code> — (Batch, )</li>
+<li><code>text</code> — (Batch, Length)</li>
+<li><code>text_lengths</code> — (Batch,)</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def forward(
+ self,
+ speech: torch.Tensor,
+ speech_lengths: torch.Tensor,
+ text: torch.Tensor,
+ text_lengths: torch.Tensor,
+ **kwargs,
+ ) -&gt; Tuple[torch.Tensor, Dict[str, torch.Tensor], torch.Tensor]:
+ &quot;&quot;&quot;Frontend + Encoder + Decoder + Calc loss
+
+ Args:
+ speech: (Batch, Length, ...)
+ speech_lengths: (Batch, )
+ text: (Batch, Length)
+ text_lengths: (Batch,)
+ &quot;&quot;&quot;
+ text_lengths = text_lengths.squeeze()
+ speech_lengths = speech_lengths.squeeze()
+
+ batch_size = speech.shape[0]
+
+ hotword_pad = kwargs.get(&quot;hotword_pad&quot;)
+ hotword_lengths = kwargs.get(&quot;hotword_lengths&quot;)
+ # dha_pad = kwargs.get(&quot;dha_pad&quot;)
+
+ # 1. Encoder
+ encoder_out, encoder_out_lens = self.encode(speech, speech_lengths)
+
+ loss_ctc, cer_ctc = None, None
+
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/contextual_paraformer/model.py#L96" target="_blank" class="src-more">View full source on GitHub →</a></details></div>
+<div class="api-detail" id="e27"><span class="dtag badge-m">method</span><h2><code>ContextualParaformer.sampler(encoder_out, encoder_out_lens, ys_pad, ys_pad_lens, pre_acoustic_embeds, contextual_info)</code></h2><div class="dmod">funasr.models.contextual_paraformer.ContextualParaformer · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/contextual_paraformer/model.py#L268" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Sampler.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>encoder_out</code> — Encoder output tensor.</li>
+<li><code>encoder_out_lens</code> — Encoder output lengths.</li>
+<li><code>ys_pad</code> — TODO.</li>
+<li><code>ys_pad_lens</code> — Lengths of ys_pad.</li>
+<li><code>pre_acoustic_embeds</code> — TODO.</li>
+<li><code>contextual_info</code> — TODO.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def sampler(
+ self,
+ encoder_out,
+ encoder_out_lens,
+ ys_pad,
+ ys_pad_lens,
+ pre_acoustic_embeds,
+ contextual_info,
+ ):
+ &quot;&quot;&quot;Sampler.
+
+ Args:
+ encoder_out: Encoder output tensor.
+ encoder_out_lens: Encoder output lengths.
+ ys_pad: TODO.
+ ys_pad_lens: Lengths of ys_pad.
+ pre_acoustic_embeds: TODO.
+ contextual_info: TODO.
+ &quot;&quot;&quot;
+ tgt_mask = (~make_pad_mask(ys_pad_lens, maxlen=ys_pad_lens.max())[:, :, None]).to(
+ ys_pad.device
+ )
+ ys_pad = ys_pad * tgt_mask[:, :, 0]
+ if self.share_embedding:
+ ys_pad_embed = self.decoder.output_layer.weight[ys_pad]
+ else:
+ ys_pad_embed = self.decoder.embed(ys_pad)
+ with torch.no_grad():
+ decoder_outs = self.decoder(
+ encoder_out,
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/contextual_paraformer/model.py#L268" target="_blank" class="src-more">View full source on GitHub →</a></details></div>
+<div class="api-detail" id="e28"><span class="dtag badge-m">method</span><h2><code>ContextualParaformer.cal_decoder_with_predictor(encoder_out, encoder_out_lens, sematic_embeds, ys_pad_lens, hw_list, clas_scale)</code></h2><div class="dmod">funasr.models.contextual_paraformer.ContextualParaformer · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/contextual_paraformer/model.py#L331" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Cal decoder with predictor.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>encoder_out</code> — Encoder output tensor.</li>
+<li><code>encoder_out_lens</code> — Encoder output lengths.</li>
+<li><code>sematic_embeds</code> — TODO.</li>
+<li><code>ys_pad_lens</code> — Lengths of ys_pad.</li>
+<li><code>hw_list</code> — TODO.</li>
+<li><code>clas_scale</code> — TODO.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def cal_decoder_with_predictor(
+ self,
+ encoder_out,
+ encoder_out_lens,
+ sematic_embeds,
+ ys_pad_lens,
+ hw_list=None,
+ clas_scale=1.0,
+ ):
+ &quot;&quot;&quot;Cal decoder with predictor.
+
+ Args:
+ encoder_out: Encoder output tensor.
+ encoder_out_lens: Encoder output lengths.
+ sematic_embeds: TODO.
+ ys_pad_lens: Lengths of ys_pad.
+ hw_list: TODO.
+ clas_scale: TODO.
+ &quot;&quot;&quot;
+ if hw_list is None:
+ hw_list = [torch.Tensor([1]).long().to(encoder_out.device)] # empty hotword list
+ hw_list_pad = pad_list(hw_list, 0)
+ if self.use_decoder_embedding:
+ hw_embed = self.decoder.embed(hw_list_pad)
+ else:
+ hw_embed = self.bias_embed(hw_list_pad)
+ hw_embed, (h_n, _) = self.bias_encoder(hw_embed)
+ hw_embed = h_n.repeat(encoder_out.shape[0], 1, 1)
+ else:
+ hw_lengths = [len(i) for i in hw_list]
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/contextual_paraformer/model.py#L331" target="_blank" class="src-more">View full source on GitHub →</a></details></div>
+<div class="api-detail" id="e29"><span class="dtag badge-m">method</span><h2><code>ContextualParaformer.inference(data_in, data_lengths, key, tokenizer, frontend, **kwargs)</code></h2><div class="dmod">funasr.models.contextual_paraformer.ContextualParaformer · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/contextual_paraformer/model.py#L387" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Run inference on input data.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>data_in</code> — Input data (audio samples, file paths, or text).</li>
+<li><code>data_lengths</code> — Lengths of each input sample in the batch.</li>
+<li><code>key</code> — Sample identifiers.</li>
+<li><code>tokenizer</code> — Tokenizer instance for text encoding/decoding.</li>
+<li><code>frontend</code> — Audio frontend for feature extraction.</li>
+<li><code>**kwargs</code> — Additional keyword arguments.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def inference(
+ self,
+ data_in,
+ data_lengths=None,
+ key: list = None,
+ tokenizer=None,
+ frontend=None,
+ **kwargs,
+ ):
+ # init beamsearch
+
+ &quot;&quot;&quot;Run inference on input data.
+
+ Args:
+ data_in: Input data (audio samples, file paths, or text).
+ data_lengths: Lengths of each input sample in the batch.
+ key: Sample identifiers.
+ tokenizer: Tokenizer instance for text encoding/decoding.
+ frontend: Audio frontend for feature extraction.
+ **kwargs: Additional keyword arguments.
+ &quot;&quot;&quot;
+ is_use_ctc = kwargs.get(&quot;decoding_ctc_weight&quot;, 0.0) &gt; 0.00001 and self.ctc != None
+ is_use_lm = (
+ kwargs.get(&quot;lm_weight&quot;, 0.0) &gt; 0.00001 and kwargs.get(&quot;lm_file&quot;, None) is not None
+ )
+ if self.beam_search is None and (is_use_lm or is_use_ctc):
+ logging.info(&quot;enable beam_search&quot;)
+ self.init_beam_search(**kwargs)
+ self.nbest = kwargs.get(&quot;nbest&quot;, 1)
+
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/contextual_paraformer/model.py#L387" target="_blank" class="src-more">View full source on GitHub →</a></details></div>
+<div class="api-detail" id="e30"><span class="dtag badge-m">method</span><h2><code>ContextualParaformer.generate_hotwords_list(hotword_list_or_file, tokenizer, frontend)</code></h2><div class="dmod">funasr.models.contextual_paraformer.ContextualParaformer · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/contextual_paraformer/model.py#L534" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Generate hotwords list.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>hotword_list_or_file</code> — TODO.</li>
+<li><code>tokenizer</code> — Tokenizer instance for text encoding/decoding.</li>
+<li><code>frontend</code> — Audio frontend for feature extraction.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def generate_hotwords_list(self, hotword_list_or_file, tokenizer=None, frontend=None):
+ &quot;&quot;&quot;Generate hotwords list.
+
+ Args:
+ hotword_list_or_file: TODO.
+ tokenizer: Tokenizer instance for text encoding/decoding.
+ frontend: Audio frontend for feature extraction.
+ &quot;&quot;&quot;
+ def load_seg_dict(seg_dict_file):
+ &quot;&quot;&quot;Load seg dict.
+
+ Args:
+ seg_dict_file: TODO.
+ &quot;&quot;&quot;
+ seg_dict = {}
+ assert isinstance(seg_dict_file, str)
+ with open(seg_dict_file, &quot;r&quot;, encoding=&quot;utf8&quot;) as f:
+ lines = f.readlines()
+ for line in lines:
+ s = line.strip().split()
+ key = s[0]
+ value = s[1:]
+ seg_dict[key] = &quot; &quot;.join(value)
+ return seg_dict
+
+ def seg_tokenize(txt, seg_dict):
+ &quot;&quot;&quot;Seg tokenize.
+
+ Args:
+ txt: TODO.
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/contextual_paraformer/model.py#L534" target="_blank" class="src-more">View full source on GitHub →</a></details></div>
+<div class="api-detail" id="e31"><span class="dtag badge-m">method</span><h2><code>ContextualParaformer.export(**kwargs)</code></h2><div class="dmod">funasr.models.contextual_paraformer.ContextualParaformer · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/contextual_paraformer/model.py#L659" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Export.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>**kwargs</code> — Additional keyword arguments.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def export(
+ self,
+ **kwargs,
+ ):
+ &quot;&quot;&quot;Export.
+
+ Args:
+ **kwargs: Additional keyword arguments.
+ &quot;&quot;&quot;
+ if &quot;max_seq_len&quot; not in kwargs:
+ kwargs[&quot;max_seq_len&quot;] = 512
+ from .export_meta import export_rebuild_model
+
+ models = export_rebuild_model(model=self, **kwargs)
+ return models
+</code></pre></details></div>
+<div class="api-detail" id="e32"><span class="dtag badge-c">class</span><h2>CTTransformer</h2><div class="dmod">funasr.models.ct_transformer · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/ct_transformer/model.py#L40" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><ul>
+<li><code>CT</code> — Transformer: Punctuation Restoration Model.</li>
+</ul>
+<p>Adds punctuation (comma, period, question mark) to unpunctuated text.</p>
+<p>Supports Chinese and English. Used as punc_model in the ASR pipeline.</p>
+<ul>
+<li><code>Output</code> — {&quot;key&quot;: &quot;...&quot;, &quot;text&quot;: &quot;punctuated text&quot;, &quot;punc_array&quot;: Tensor}</li>
+</ul>
+<p>punc_array encoding: 1=none, 2=comma(,), 3=period(。), 4=question(?)</p>
+<ul>
+<li><code>Note</code> — Not needed for Fun-ASR-Nano/SenseVoice/Qwen3-ASR (they output punctuation natively).</li>
+</ul>
+<p>Only required for Paraformer models.</p>
+<ul>
+<li><code>Author</code> — Speech Lab of DAMO Academy, Alibaba Group</li>
+<li><code>CT</code> — Transformer: Controllable time-delay transformer for real-time punctuation prediction and disfluency detection</li>
+</ul>
+<p>https://arxiv.org/pdf/2003.01309.pdf</p>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code>class CTTransformer(torch.nn.Module):
+ &quot;&quot;&quot;CT-Transformer: Punctuation Restoration Model.
+
+ Adds punctuation (comma, period, question mark) to unpunctuated text.
+ Supports Chinese and English. Used as punc_model in the ASR pipeline.
+
+ Output: {&quot;key&quot;: &quot;...&quot;, &quot;text&quot;: &quot;punctuated text&quot;, &quot;punc_array&quot;: Tensor}
+ punc_array encoding: 1=none, 2=comma(,), 3=period(。), 4=question(?)
+
+ Note: Not needed for Fun-ASR-Nano/SenseVoice/Qwen3-ASR (they output punctuation natively).
+ Only required for Paraformer models.
+
+ Author: Speech Lab of DAMO Academy, Alibaba Group
+ CT-Transformer: Controllable time-delay transformer for real-time punctuation prediction and disfluency detection
+ https://arxiv.org/pdf/2003.01309.pdf
+ &quot;&quot;&quot;
+
+ def __init__(
+ self,
+ encoder: str = None,
+ encoder_conf: dict = None,
+ vocab_size: int = -1,
+ punc_list: list = None,
+ punc_weight: list = None,
+ embed_unit: int = 128,
+ att_unit: int = 256,
+ dropout_rate: float = 0.5,
+ ignore_id: int = -1,
+ sos: int = 1,
+ eos: int = 2,
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/ct_transformer/model.py#L40" target="_blank" class="src-more">View full source on GitHub →</a></details><h3 class="mt">Methods</h3><div class="mblk"><code>.punc_forward(text, text_lengths, **kwargs)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/ct_transformer/model.py#L113" target="_blank" class="gh-link">L113</a><div class="mdoc"><p>Compute loss value from buffer sequences.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>input (torch.Tensor)</code> — Input ids. (batch, len)</li>
+<li><code>hidden (torch.Tensor)</code> — Target ids. (batch, len)</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def punc_forward(self, text: torch.Tensor, text_lengths: torch.Tensor, **kwargs):
+ &quot;&quot;&quot;Compute loss value from buffer sequences.
+
+ Args:
+ input (torch.Tensor): Input ids. (batch, len)
+ hidden (torch.Tensor): Target ids. (batch, len)
+
+ &quot;&quot;&quot;
+ x = self.embed(text)
+ # mask = self._target_mask(input)
+ h, _, _ = self.encoder(x, text_lengths)
+ y = self.decoder(h)
+ return y, None
+</code></pre></details></div><div class="mblk"><code>.with_vad()</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/ct_transformer/model.py#L127" target="_blank" class="gh-link">L127</a><div class="mdoc"><p>With vad.</p>
+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def with_vad(self):
+ &quot;&quot;&quot;With vad.&quot;&quot;&quot;
+ return False
+</code></pre></details></div><div class="mblk"><code>.score(y, state, x)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/ct_transformer/model.py#L131" target="_blank" class="gh-link">L131</a><div class="mdoc"><p>Score new token.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>y (torch.Tensor)</code> — 1D torch.int64 prefix tokens.</li>
+<li><code>state</code> — Scorer state for prefix tokens</li>
+<li><code>x (torch.Tensor)</code> — encoder feature that generates ys.</li>
+</ul>
+<h4>Returns:</h4>
+<p>tuple[torch.Tensor, Any]: Tuple of</p>
+<p>torch.float32 scores for next token (vocab_size)</p>
+<p>and next state for ys</p>
+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def score(self, y: torch.Tensor, state: Any, x: torch.Tensor) -&gt; Tuple[torch.Tensor, Any]:
+ &quot;&quot;&quot;Score new token.
+
+ Args:
+ y (torch.Tensor): 1D torch.int64 prefix tokens.
+ state: Scorer state for prefix tokens
+ x (torch.Tensor): encoder feature that generates ys.
+
+ Returns:
+ tuple[torch.Tensor, Any]: Tuple of
+ torch.float32 scores for next token (vocab_size)
+ and next state for ys
+
+ &quot;&quot;&quot;
+ y = y.unsqueeze(0)
+ h, _, cache = self.encoder.forward_one_step(
+ self.embed(y), self._target_mask(y), cache=state
+ )
+ h = self.decoder(h[:, -1])
+ logp = h.log_softmax(dim=-1).squeeze(0)
+ return logp, cache
+</code></pre></details></div><div class="mblk"><code>.batch_score(ys, states, xs)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/ct_transformer/model.py#L153" target="_blank" class="gh-link">L153</a><div class="mdoc"><p>Score new token batch.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>ys (torch.Tensor)</code> — torch.int64 prefix tokens (n_batch, ylen).</li>
+<li><code>states (List[Any])</code> — Scorer states for prefix tokens.</li>
+</ul>
+<p>xs (torch.Tensor):</p>
+<p>The encoder feature that generates ys (n_batch, xlen, n_feat).</p>
+<h4>Returns:</h4>
+<p>tuple[torch.Tensor, List[Any]]: Tuple of</p>
+<p>batchfied scores for next token with shape of `(n_batch, vocab_size)`</p>
+<p>and next state list for ys.</p>
+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def batch_score(
+ self, ys: torch.Tensor, states: List[Any], xs: torch.Tensor
+ ) -&gt; Tuple[torch.Tensor, List[Any]]:
+ &quot;&quot;&quot;Score new token batch.
+
+ Args:
+ ys (torch.Tensor): torch.int64 prefix tokens (n_batch, ylen).
+ states (List[Any]): Scorer states for prefix tokens.
+ xs (torch.Tensor):
+ The encoder feature that generates ys (n_batch, xlen, n_feat).
+
+ Returns:
+ tuple[torch.Tensor, List[Any]]: Tuple of
+ batchfied scores for next token with shape of `(n_batch, vocab_size)`
+ and next state list for ys.
+
+ &quot;&quot;&quot;
+ # merge states
+ n_batch = len(ys)
+ n_layers = len(self.encoder.encoders)
+ if states[0] is None:
+ batch_state = None
+ else:
+ # transpose state of [batch, layer] into [layer, batch]
+ batch_state = [
+ torch.stack([states[b][i] for b in range(n_batch)]) for i in range(n_layers)
+ ]
+
+ # batch decoding
+ h, _, states = self.encoder.forward_one_step(
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/ct_transformer/model.py#L153" target="_blank" class="src-more">View full source →</a></details></div><div class="mblk"><code>.nll(text, punc, text_lengths, punc_lengths, max_length, vad_indexes, vad_indexes_lengths)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/ct_transformer/model.py#L192" target="_blank" class="gh-link">L192</a><div class="mdoc"><p>Compute negative log likelihood(nll)</p>
+<p>Normally, this function is called in batchify_nll.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>text</code> — (Batch, Length)</li>
+<li><code>punc</code> — (Batch, Length)</li>
+<li><code>text_lengths</code> — (Batch,)</li>
+<li><code>max_lengths</code> — int</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def nll(
+ self,
+ text: torch.Tensor,
+ punc: torch.Tensor,
+ text_lengths: torch.Tensor,
+ punc_lengths: torch.Tensor,
+ max_length: Optional[int] = None,
+ vad_indexes: Optional[torch.Tensor] = None,
+ vad_indexes_lengths: Optional[torch.Tensor] = None,
+ ) -&gt; Tuple[torch.Tensor, torch.Tensor]:
+ &quot;&quot;&quot;Compute negative log likelihood(nll)
+
+ Normally, this function is called in batchify_nll.
+ Args:
+ text: (Batch, Length)
+ punc: (Batch, Length)
+ text_lengths: (Batch,)
+ max_lengths: int
+ &quot;&quot;&quot;
+ batch_size = text.size(0)
+ # For data parallel
+ if max_length is None:
+ text = text[:, : text_lengths.max()]
+ punc = punc[:, : text_lengths.max()]
+ else:
+ text = text[:, :max_length]
+ punc = punc[:, :max_length]
+
+ if self.with_vad():
+ # Should be VadRealtimeTransformer
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/ct_transformer/model.py#L192" target="_blank" class="src-more">View full source →</a></details></div><div class="mblk"><code>.forward(text, punc, text_lengths, punc_lengths, vad_indexes, vad_indexes_lengths)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/ct_transformer/model.py#L262" target="_blank" class="gh-link">L262</a><div class="mdoc"><p>Forward pass for training.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>text</code> — Text tensor or string input.</li>
+<li><code>punc</code> — TODO.</li>
+<li><code>text_lengths</code> — Length of each text sample.</li>
+<li><code>punc_lengths</code> — Lengths of punc.</li>
+<li><code>vad_indexes</code> — TODO.</li>
+<li><code>vad_indexes_lengths</code> — Lengths of vad_indexes.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def forward(
+ self,
+ text: torch.Tensor,
+ punc: torch.Tensor,
+ text_lengths: torch.Tensor,
+ punc_lengths: torch.Tensor,
+ vad_indexes: Optional[torch.Tensor] = None,
+ vad_indexes_lengths: Optional[torch.Tensor] = None,
+ ):
+ &quot;&quot;&quot;Forward pass for training.
+
+ Args:
+ text: Text tensor or string input.
+ punc: TODO.
+ text_lengths: Length of each text sample.
+ punc_lengths: Lengths of punc.
+ vad_indexes: TODO.
+ vad_indexes_lengths: Lengths of vad_indexes.
+ &quot;&quot;&quot;
+ nll, y_lengths = self.nll(text, punc, text_lengths, punc_lengths, vad_indexes=vad_indexes)
+ ntokens = y_lengths.sum()
+ loss = nll.sum() / ntokens
+ stats = dict(loss=loss.detach())
+
+ # force_gatherable: to-device and to-tensor if scalar for DataParallel
+ loss, stats, weight = force_gatherable((loss, stats, ntokens), loss.device)
+ return loss, stats, weight
+</code></pre></details></div><div class="mblk"><code>.inference(data_in, data_lengths, key, tokenizer, frontend, **kwargs)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/ct_transformer/model.py#L290" target="_blank" class="gh-link">L290</a><div class="mdoc"><p>Run inference on input data.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>data_in</code> — Input data (audio samples, file paths, or text).</li>
+<li><code>data_lengths</code> — Lengths of each input sample in the batch.</li>
+<li><code>key</code> — Sample identifiers.</li>
+<li><code>tokenizer</code> — Tokenizer instance for text encoding/decoding.</li>
+<li><code>frontend</code> — Audio frontend for feature extraction.</li>
+<li><code>**kwargs</code> — Additional keyword arguments.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def inference(
+ self,
+ data_in,
+ data_lengths=None,
+ key: list = None,
+ tokenizer=None,
+ frontend=None,
+ **kwargs,
+ ):
+ &quot;&quot;&quot;Run inference on input data.
+
+ Args:
+ data_in: Input data (audio samples, file paths, or text).
+ data_lengths: Lengths of each input sample in the batch.
+ key: Sample identifiers.
+ tokenizer: Tokenizer instance for text encoding/decoding.
+ frontend: Audio frontend for feature extraction.
+ **kwargs: Additional keyword arguments.
+ &quot;&quot;&quot;
+ assert len(data_in) == 1
+ if not data_in[0] or (isinstance(data_in[0], str) and not data_in[0].strip()):
+ meta_data = {&quot;batch_data_time&quot;: -1}
+ return [{&quot;key&quot;: key[0] if key else &quot;&quot;, &quot;text&quot;: &quot;&quot;, &quot;punc_array&quot;: None}], meta_data
+ text = load_audio_text_image_video(data_in, data_type=kwargs.get(&quot;kwargs&quot;, &quot;text&quot;))[0]
+ vad_indexes = kwargs.get(&quot;vad_indexes&quot;, None)
+ # text = data_in[0]
+ # text_lengths = data_lengths[0] if data_lengths is not None else None
+ split_size = kwargs.get(&quot;split_size&quot;, 20)
+
+ tokens = split_words(text, jieba_usr_dict=self.jieba_usr_dict)
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/ct_transformer/model.py#L290" target="_blank" class="src-more">View full source →</a></details></div><div class="mblk"><code>.export(**kwargs)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/ct_transformer/model.py#L471" target="_blank" class="gh-link">L471</a><div class="mdoc"><p>Export.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>**kwargs</code> — Additional keyword arguments.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def export(self, **kwargs):
+
+ &quot;&quot;&quot;Export.
+
+ Args:
+ **kwargs: Additional keyword arguments.
+ &quot;&quot;&quot;
+ from .export_meta import export_rebuild_model
+
+ models = export_rebuild_model(model=self, **kwargs)
+ return models
+</code></pre></details></div></div>
+<div class="api-detail" id="e33"><span class="dtag badge-m">method</span><h2><code>CTTransformer.punc_forward(text, text_lengths, **kwargs)</code></h2><div class="dmod">funasr.models.ct_transformer.CTTransformer · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/ct_transformer/model.py#L113" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Compute loss value from buffer sequences.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>input (torch.Tensor)</code> — Input ids. (batch, len)</li>
+<li><code>hidden (torch.Tensor)</code> — Target ids. (batch, len)</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def punc_forward(self, text: torch.Tensor, text_lengths: torch.Tensor, **kwargs):
+ &quot;&quot;&quot;Compute loss value from buffer sequences.
+
+ Args:
+ input (torch.Tensor): Input ids. (batch, len)
+ hidden (torch.Tensor): Target ids. (batch, len)
+
+ &quot;&quot;&quot;
+ x = self.embed(text)
+ # mask = self._target_mask(input)
+ h, _, _ = self.encoder(x, text_lengths)
+ y = self.decoder(h)
+ return y, None
+</code></pre></details></div>
+<div class="api-detail" id="e34"><span class="dtag badge-m">method</span><h2><code>CTTransformer.with_vad()</code></h2><div class="dmod">funasr.models.ct_transformer.CTTransformer · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/ct_transformer/model.py#L127" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>With vad.</p>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def with_vad(self):
+ &quot;&quot;&quot;With vad.&quot;&quot;&quot;
+ return False
+</code></pre></details></div>
+<div class="api-detail" id="e35"><span class="dtag badge-m">method</span><h2><code>CTTransformer.score(y, state, x)</code></h2><div class="dmod">funasr.models.ct_transformer.CTTransformer · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/ct_transformer/model.py#L131" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Score new token.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>y (torch.Tensor)</code> — 1D torch.int64 prefix tokens.</li>
+<li><code>state</code> — Scorer state for prefix tokens</li>
+<li><code>x (torch.Tensor)</code> — encoder feature that generates ys.</li>
+</ul>
+<h4>Returns:</h4>
+<p>tuple[torch.Tensor, Any]: Tuple of</p>
+<p>torch.float32 scores for next token (vocab_size)</p>
+<p>and next state for ys</p>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def score(self, y: torch.Tensor, state: Any, x: torch.Tensor) -&gt; Tuple[torch.Tensor, Any]:
+ &quot;&quot;&quot;Score new token.
+
+ Args:
+ y (torch.Tensor): 1D torch.int64 prefix tokens.
+ state: Scorer state for prefix tokens
+ x (torch.Tensor): encoder feature that generates ys.
+
+ Returns:
+ tuple[torch.Tensor, Any]: Tuple of
+ torch.float32 scores for next token (vocab_size)
+ and next state for ys
+
+ &quot;&quot;&quot;
+ y = y.unsqueeze(0)
+ h, _, cache = self.encoder.forward_one_step(
+ self.embed(y), self._target_mask(y), cache=state
+ )
+ h = self.decoder(h[:, -1])
+ logp = h.log_softmax(dim=-1).squeeze(0)
+ return logp, cache
+</code></pre></details></div>
+<div class="api-detail" id="e36"><span class="dtag badge-m">method</span><h2><code>CTTransformer.batch_score(ys, states, xs)</code></h2><div class="dmod">funasr.models.ct_transformer.CTTransformer · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/ct_transformer/model.py#L153" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Score new token batch.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>ys (torch.Tensor)</code> — torch.int64 prefix tokens (n_batch, ylen).</li>
+<li><code>states (List[Any])</code> — Scorer states for prefix tokens.</li>
+</ul>
+<p>xs (torch.Tensor):</p>
+<p>The encoder feature that generates ys (n_batch, xlen, n_feat).</p>
+<h4>Returns:</h4>
+<p>tuple[torch.Tensor, List[Any]]: Tuple of</p>
+<p>batchfied scores for next token with shape of `(n_batch, vocab_size)`</p>
+<p>and next state list for ys.</p>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def batch_score(
+ self, ys: torch.Tensor, states: List[Any], xs: torch.Tensor
+ ) -&gt; Tuple[torch.Tensor, List[Any]]:
+ &quot;&quot;&quot;Score new token batch.
+
+ Args:
+ ys (torch.Tensor): torch.int64 prefix tokens (n_batch, ylen).
+ states (List[Any]): Scorer states for prefix tokens.
+ xs (torch.Tensor):
+ The encoder feature that generates ys (n_batch, xlen, n_feat).
+
+ Returns:
+ tuple[torch.Tensor, List[Any]]: Tuple of
+ batchfied scores for next token with shape of `(n_batch, vocab_size)`
+ and next state list for ys.
+
+ &quot;&quot;&quot;
+ # merge states
+ n_batch = len(ys)
+ n_layers = len(self.encoder.encoders)
+ if states[0] is None:
+ batch_state = None
+ else:
+ # transpose state of [batch, layer] into [layer, batch]
+ batch_state = [
+ torch.stack([states[b][i] for b in range(n_batch)]) for i in range(n_layers)
+ ]
+
+ # batch decoding
+ h, _, states = self.encoder.forward_one_step(
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/ct_transformer/model.py#L153" target="_blank" class="src-more">View full source on GitHub →</a></details></div>
+<div class="api-detail" id="e37"><span class="dtag badge-m">method</span><h2><code>CTTransformer.nll(text, punc, text_lengths, punc_lengths, max_length, vad_indexes, vad_indexes_lengths)</code></h2><div class="dmod">funasr.models.ct_transformer.CTTransformer · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/ct_transformer/model.py#L192" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Compute negative log likelihood(nll)</p>
+<p>Normally, this function is called in batchify_nll.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>text</code> — (Batch, Length)</li>
+<li><code>punc</code> — (Batch, Length)</li>
+<li><code>text_lengths</code> — (Batch,)</li>
+<li><code>max_lengths</code> — int</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def nll(
+ self,
+ text: torch.Tensor,
+ punc: torch.Tensor,
+ text_lengths: torch.Tensor,
+ punc_lengths: torch.Tensor,
+ max_length: Optional[int] = None,
+ vad_indexes: Optional[torch.Tensor] = None,
+ vad_indexes_lengths: Optional[torch.Tensor] = None,
+ ) -&gt; Tuple[torch.Tensor, torch.Tensor]:
+ &quot;&quot;&quot;Compute negative log likelihood(nll)
+
+ Normally, this function is called in batchify_nll.
+ Args:
+ text: (Batch, Length)
+ punc: (Batch, Length)
+ text_lengths: (Batch,)
+ max_lengths: int
+ &quot;&quot;&quot;
+ batch_size = text.size(0)
+ # For data parallel
+ if max_length is None:
+ text = text[:, : text_lengths.max()]
+ punc = punc[:, : text_lengths.max()]
+ else:
+ text = text[:, :max_length]
+ punc = punc[:, :max_length]
+
+ if self.with_vad():
+ # Should be VadRealtimeTransformer
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/ct_transformer/model.py#L192" target="_blank" class="src-more">View full source on GitHub →</a></details></div>
+<div class="api-detail" id="e38"><span class="dtag badge-m">method</span><h2><code>CTTransformer.forward(text, punc, text_lengths, punc_lengths, vad_indexes, vad_indexes_lengths)</code></h2><div class="dmod">funasr.models.ct_transformer.CTTransformer · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/ct_transformer/model.py#L262" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Forward pass for training.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>text</code> — Text tensor or string input.</li>
+<li><code>punc</code> — TODO.</li>
+<li><code>text_lengths</code> — Length of each text sample.</li>
+<li><code>punc_lengths</code> — Lengths of punc.</li>
+<li><code>vad_indexes</code> — TODO.</li>
+<li><code>vad_indexes_lengths</code> — Lengths of vad_indexes.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def forward(
+ self,
+ text: torch.Tensor,
+ punc: torch.Tensor,
+ text_lengths: torch.Tensor,
+ punc_lengths: torch.Tensor,
+ vad_indexes: Optional[torch.Tensor] = None,
+ vad_indexes_lengths: Optional[torch.Tensor] = None,
+ ):
+ &quot;&quot;&quot;Forward pass for training.
+
+ Args:
+ text: Text tensor or string input.
+ punc: TODO.
+ text_lengths: Length of each text sample.
+ punc_lengths: Lengths of punc.
+ vad_indexes: TODO.
+ vad_indexes_lengths: Lengths of vad_indexes.
+ &quot;&quot;&quot;
+ nll, y_lengths = self.nll(text, punc, text_lengths, punc_lengths, vad_indexes=vad_indexes)
+ ntokens = y_lengths.sum()
+ loss = nll.sum() / ntokens
+ stats = dict(loss=loss.detach())
+
+ # force_gatherable: to-device and to-tensor if scalar for DataParallel
+ loss, stats, weight = force_gatherable((loss, stats, ntokens), loss.device)
+ return loss, stats, weight
+</code></pre></details></div>
+<div class="api-detail" id="e39"><span class="dtag badge-m">method</span><h2><code>CTTransformer.inference(data_in, data_lengths, key, tokenizer, frontend, **kwargs)</code></h2><div class="dmod">funasr.models.ct_transformer.CTTransformer · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/ct_transformer/model.py#L290" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Run inference on input data.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>data_in</code> — Input data (audio samples, file paths, or text).</li>
+<li><code>data_lengths</code> — Lengths of each input sample in the batch.</li>
+<li><code>key</code> — Sample identifiers.</li>
+<li><code>tokenizer</code> — Tokenizer instance for text encoding/decoding.</li>
+<li><code>frontend</code> — Audio frontend for feature extraction.</li>
+<li><code>**kwargs</code> — Additional keyword arguments.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def inference(
+ self,
+ data_in,
+ data_lengths=None,
+ key: list = None,
+ tokenizer=None,
+ frontend=None,
+ **kwargs,
+ ):
+ &quot;&quot;&quot;Run inference on input data.
+
+ Args:
+ data_in: Input data (audio samples, file paths, or text).
+ data_lengths: Lengths of each input sample in the batch.
+ key: Sample identifiers.
+ tokenizer: Tokenizer instance for text encoding/decoding.
+ frontend: Audio frontend for feature extraction.
+ **kwargs: Additional keyword arguments.
+ &quot;&quot;&quot;
+ assert len(data_in) == 1
+ if not data_in[0] or (isinstance(data_in[0], str) and not data_in[0].strip()):
+ meta_data = {&quot;batch_data_time&quot;: -1}
+ return [{&quot;key&quot;: key[0] if key else &quot;&quot;, &quot;text&quot;: &quot;&quot;, &quot;punc_array&quot;: None}], meta_data
+ text = load_audio_text_image_video(data_in, data_type=kwargs.get(&quot;kwargs&quot;, &quot;text&quot;))[0]
+ vad_indexes = kwargs.get(&quot;vad_indexes&quot;, None)
+ # text = data_in[0]
+ # text_lengths = data_lengths[0] if data_lengths is not None else None
+ split_size = kwargs.get(&quot;split_size&quot;, 20)
+
+ tokens = split_words(text, jieba_usr_dict=self.jieba_usr_dict)
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/ct_transformer/model.py#L290" target="_blank" class="src-more">View full source on GitHub →</a></details></div>
+<div class="api-detail" id="e40"><span class="dtag badge-m">method</span><h2><code>CTTransformer.export(**kwargs)</code></h2><div class="dmod">funasr.models.ct_transformer.CTTransformer · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/ct_transformer/model.py#L471" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Export.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>**kwargs</code> — Additional keyword arguments.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def export(self, **kwargs):
+
+ &quot;&quot;&quot;Export.
+
+ Args:
+ **kwargs: Additional keyword arguments.
+ &quot;&quot;&quot;
+ from .export_meta import export_rebuild_model
+
+ models = export_rebuild_model(model=self, **kwargs)
+ return models
+</code></pre></details></div>
+<div class="api-detail" id="e41"><span class="dtag badge-c">class</span><h2>CTTransformerStreaming</h2><div class="dmod">funasr.models.ct_transformer_streaming · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/ct_transformer_streaming/model.py#L33" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><ul>
+<li><code>CT</code> — Transformer Streaming: Online punctuation restoration.</li>
+</ul>
+<p>Processes text incrementally with a sliding window, maintaining cache</p>
+<p>of previous context for consistent punctuation decisions across chunks.</p>
+<p>Used as punc_model in streaming ASR pipelines.</p>
+<p>Supports VAD-aware punctuation: uses VAD boundaries to improve sentence segmentation.</p>
+<ul>
+<li><code>Reference</code> — https://arxiv.org/pdf/2003.01309.pdf</li>
+<li><code>Output</code> — {&quot;key&quot;: str, &quot;text&quot;: str, &quot;punc_array&quot;: Tensor}</li>
+<li><code>Author</code> — Speech Lab of DAMO Academy, Alibaba Group</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code>class CTTransformerStreaming(CTTransformer):
+ &quot;&quot;&quot;CT-Transformer Streaming: Online punctuation restoration.
+
+ Processes text incrementally with a sliding window, maintaining cache
+ of previous context for consistent punctuation decisions across chunks.
+ Used as punc_model in streaming ASR pipelines.
+
+ Supports VAD-aware punctuation: uses VAD boundaries to improve sentence segmentation.
+
+ Reference: https://arxiv.org/pdf/2003.01309.pdf
+ Output: {&quot;key&quot;: str, &quot;text&quot;: str, &quot;punc_array&quot;: Tensor}
+
+ Author: Speech Lab of DAMO Academy, Alibaba Group
+ &quot;&quot;&quot;
+
+ def __init__(
+ self,
+ *args,
+ **kwargs,
+ ):
+ &quot;&quot;&quot;Initialize CTTransformerStreaming.
+
+ Args:
+ *args: Variable positional arguments.
+ **kwargs: Additional keyword arguments.
+ &quot;&quot;&quot;
+ super().__init__(*args, **kwargs)
+
+ def punc_forward(
+ self, text: torch.Tensor, text_lengths: torch.Tensor, vad_indexes: torch.Tensor, **kwargs
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/ct_transformer_streaming/model.py#L33" target="_blank" class="src-more">View full source on GitHub →</a></details><h3 class="mt">Methods</h3><div class="mblk"><code>.punc_forward(text, text_lengths, vad_indexes, **kwargs)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/ct_transformer_streaming/model.py#L61" target="_blank" class="gh-link">L61</a><div class="mdoc"><p>Compute loss value from buffer sequences.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>input (torch.Tensor)</code> — Input ids. (batch, len)</li>
+<li><code>hidden (torch.Tensor)</code> — Target ids. (batch, len)</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def punc_forward(
+ self, text: torch.Tensor, text_lengths: torch.Tensor, vad_indexes: torch.Tensor, **kwargs
+ ):
+ &quot;&quot;&quot;Compute loss value from buffer sequences.
+
+ Args:
+ input (torch.Tensor): Input ids. (batch, len)
+ hidden (torch.Tensor): Target ids. (batch, len)
+
+ &quot;&quot;&quot;
+ x = self.embed(text)
+ # mask = self._target_mask(input)
+ h, _, _ = self.encoder(x, text_lengths, vad_indexes=vad_indexes)
+ y = self.decoder(h)
+ return y, None
+</code></pre></details></div><div class="mblk"><code>.with_vad()</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/ct_transformer_streaming/model.py#L77" target="_blank" class="gh-link">L77</a><div class="mdoc"><p>With vad.</p>
+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def with_vad(self):
+ &quot;&quot;&quot;With vad.&quot;&quot;&quot;
+ return True
+</code></pre></details></div><div class="mblk"><code>.inference(data_in, data_lengths, key, tokenizer, frontend, cache, **kwargs)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/ct_transformer_streaming/model.py#L81" target="_blank" class="gh-link">L81</a><div class="mdoc"><p>Run inference on input data.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>data_in</code> — Input data (audio samples, file paths, or text).</li>
+<li><code>data_lengths</code> — Lengths of each input sample in the batch.</li>
+<li><code>key</code> — Sample identifiers.</li>
+<li><code>tokenizer</code> — Tokenizer instance for text encoding/decoding.</li>
+<li><code>frontend</code> — Audio frontend for feature extraction.</li>
+<li><code>cache</code> — State cache dict for streaming inference.</li>
+<li><code>**kwargs</code> — Additional keyword arguments.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def inference(
+ self,
+ data_in,
+ data_lengths=None,
+ key: list = None,
+ tokenizer=None,
+ frontend=None,
+ cache: dict = None,
+ **kwargs,
+ ):
+ &quot;&quot;&quot;Run inference on input data.
+
+ Args:
+ data_in: Input data (audio samples, file paths, or text).
+ data_lengths: Lengths of each input sample in the batch.
+ key: Sample identifiers.
+ tokenizer: Tokenizer instance for text encoding/decoding.
+ frontend: Audio frontend for feature extraction.
+ cache: State cache dict for streaming inference.
+ **kwargs: Additional keyword arguments.
+ &quot;&quot;&quot;
+ if cache is None:
+ cache = {}
+ assert len(data_in) == 1
+
+ if len(cache) == 0:
+ cache[&quot;pre_text&quot;] = []
+ text = load_audio_text_image_video(data_in, data_type=kwargs.get(&quot;kwargs&quot;, &quot;text&quot;))[0]
+ text = &quot;&quot;.join(cache[&quot;pre_text&quot;]) + &quot; &quot; + text
+
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/ct_transformer_streaming/model.py#L81" target="_blank" class="src-more">View full source →</a></details></div><div class="mblk"><code>.export(**kwargs)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/ct_transformer_streaming/model.py#L221" target="_blank" class="gh-link">L221</a><div class="mdoc"><p>Export.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>**kwargs</code> — Additional keyword arguments.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def export(self, **kwargs):
+
+ &quot;&quot;&quot;Export.
+
+ Args:
+ **kwargs: Additional keyword arguments.
+ &quot;&quot;&quot;
+ from .export_meta import export_rebuild_model
+
+ models = export_rebuild_model(model=self, **kwargs)
+ return models
+</code></pre></details></div></div>
+<div class="api-detail" id="e42"><span class="dtag badge-m">method</span><h2><code>CTTransformerStreaming.punc_forward(text, text_lengths, vad_indexes, **kwargs)</code></h2><div class="dmod">funasr.models.ct_transformer_streaming.CTTransformerStreaming · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/ct_transformer_streaming/model.py#L61" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Compute loss value from buffer sequences.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>input (torch.Tensor)</code> — Input ids. (batch, len)</li>
+<li><code>hidden (torch.Tensor)</code> — Target ids. (batch, len)</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def punc_forward(
+ self, text: torch.Tensor, text_lengths: torch.Tensor, vad_indexes: torch.Tensor, **kwargs
+ ):
+ &quot;&quot;&quot;Compute loss value from buffer sequences.
+
+ Args:
+ input (torch.Tensor): Input ids. (batch, len)
+ hidden (torch.Tensor): Target ids. (batch, len)
+
+ &quot;&quot;&quot;
+ x = self.embed(text)
+ # mask = self._target_mask(input)
+ h, _, _ = self.encoder(x, text_lengths, vad_indexes=vad_indexes)
+ y = self.decoder(h)
+ return y, None
+</code></pre></details></div>
+<div class="api-detail" id="e43"><span class="dtag badge-m">method</span><h2><code>CTTransformerStreaming.with_vad()</code></h2><div class="dmod">funasr.models.ct_transformer_streaming.CTTransformerStreaming · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/ct_transformer_streaming/model.py#L77" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>With vad.</p>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def with_vad(self):
+ &quot;&quot;&quot;With vad.&quot;&quot;&quot;
+ return True
+</code></pre></details></div>
+<div class="api-detail" id="e44"><span class="dtag badge-m">method</span><h2><code>CTTransformerStreaming.inference(data_in, data_lengths, key, tokenizer, frontend, cache, **kwargs)</code></h2><div class="dmod">funasr.models.ct_transformer_streaming.CTTransformerStreaming · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/ct_transformer_streaming/model.py#L81" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Run inference on input data.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>data_in</code> — Input data (audio samples, file paths, or text).</li>
+<li><code>data_lengths</code> — Lengths of each input sample in the batch.</li>
+<li><code>key</code> — Sample identifiers.</li>
+<li><code>tokenizer</code> — Tokenizer instance for text encoding/decoding.</li>
+<li><code>frontend</code> — Audio frontend for feature extraction.</li>
+<li><code>cache</code> — State cache dict for streaming inference.</li>
+<li><code>**kwargs</code> — Additional keyword arguments.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def inference(
+ self,
+ data_in,
+ data_lengths=None,
+ key: list = None,
+ tokenizer=None,
+ frontend=None,
+ cache: dict = None,
+ **kwargs,
+ ):
+ &quot;&quot;&quot;Run inference on input data.
+
+ Args:
+ data_in: Input data (audio samples, file paths, or text).
+ data_lengths: Lengths of each input sample in the batch.
+ key: Sample identifiers.
+ tokenizer: Tokenizer instance for text encoding/decoding.
+ frontend: Audio frontend for feature extraction.
+ cache: State cache dict for streaming inference.
+ **kwargs: Additional keyword arguments.
+ &quot;&quot;&quot;
+ if cache is None:
+ cache = {}
+ assert len(data_in) == 1
+
+ if len(cache) == 0:
+ cache[&quot;pre_text&quot;] = []
+ text = load_audio_text_image_video(data_in, data_type=kwargs.get(&quot;kwargs&quot;, &quot;text&quot;))[0]
+ text = &quot;&quot;.join(cache[&quot;pre_text&quot;]) + &quot; &quot; + text
+
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/ct_transformer_streaming/model.py#L81" target="_blank" class="src-more">View full source on GitHub →</a></details></div>
+<div class="api-detail" id="e45"><span class="dtag badge-m">method</span><h2><code>CTTransformerStreaming.export(**kwargs)</code></h2><div class="dmod">funasr.models.ct_transformer_streaming.CTTransformerStreaming · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/ct_transformer_streaming/model.py#L221" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Export.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>**kwargs</code> — Additional keyword arguments.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def export(self, **kwargs):
+
+ &quot;&quot;&quot;Export.
+
+ Args:
+ **kwargs: Additional keyword arguments.
+ &quot;&quot;&quot;
+ from .export_meta import export_rebuild_model
+
+ models = export_rebuild_model(model=self, **kwargs)
+ return models
+</code></pre></details></div>
+<div class="api-detail" id="e46"><span class="dtag badge-c">class</span><h2>Transformer</h2><div class="dmod">funasr.models.ctc · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/ctc/model.py#L18" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><ul>
+<li><code>CTC</code> — attention hybrid Encoder-Decoder model</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code>class Transformer(nn.Module):
+ &quot;&quot;&quot;CTC-attention hybrid Encoder-Decoder model&quot;&quot;&quot;
+
+ def __init__(
+ self,
+ specaug: str = None,
+ specaug_conf: dict = None,
+ normalize: str = None,
+ normalize_conf: dict = None,
+ encoder: str = None,
+ encoder_conf: dict = None,
+ ctc_conf: dict = None,
+ input_size: int = 80,
+ vocab_size: int = -1,
+ ignore_id: int = -1,
+ blank_id: int = 0,
+ sos: int = 1,
+ eos: int = 2,
+ length_normalized_loss: bool = False,
+ **kwargs,
+ ):
+
+ &quot;&quot;&quot;Initialize Transformer.
+
+ Args:
+ specaug: TODO.
+ specaug_conf: Configuration dict for specaug.
+ normalize: TODO.
+ normalize_conf: Configuration dict for normalize.
+ encoder: TODO.
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/ctc/model.py#L18" target="_blank" class="src-more">View full source on GitHub →</a></details><h3 class="mt">Methods</h3><div class="mblk"><code>.forward(speech, speech_lengths, text, text_lengths, **kwargs)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/ctc/model.py#L89" target="_blank" class="gh-link">L89</a><div class="mdoc"><p>Encoder + Decoder + Calc loss</p>
+<h4>Args:</h4>
+<ul>
+<li><code>speech</code> — (Batch, Length, ...)</li>
+<li><code>speech_lengths</code> — (Batch, )</li>
+<li><code>text</code> — (Batch, Length)</li>
+<li><code>text_lengths</code> — (Batch,)</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def forward(
+ self,
+ speech: torch.Tensor,
+ speech_lengths: torch.Tensor,
+ text: torch.Tensor,
+ text_lengths: torch.Tensor,
+ **kwargs,
+ ) -&gt; Tuple[torch.Tensor, Dict[str, torch.Tensor], torch.Tensor]:
+ &quot;&quot;&quot;Encoder + Decoder + Calc loss
+ Args:
+ speech: (Batch, Length, ...)
+ speech_lengths: (Batch, )
+ text: (Batch, Length)
+ text_lengths: (Batch,)
+ &quot;&quot;&quot;
+ # import pdb;
+ # pdb.set_trace()
+ if len(text_lengths.size()) &gt; 1:
+ text_lengths = text_lengths[:, 0]
+ if len(speech_lengths.size()) &gt; 1:
+ speech_lengths = speech_lengths[:, 0]
+
+ batch_size = speech.shape[0]
+
+ # 1. Encoder
+ encoder_out, encoder_out_lens = self.encode(speech, speech_lengths)
+
+ loss_ctc, cer_ctc = None, None
+ stats = dict()
+
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/ctc/model.py#L89" target="_blank" class="src-more">View full source →</a></details></div><div class="mblk"><code>.encode(speech, speech_lengths, **kwargs)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/ctc/model.py#L134" target="_blank" class="gh-link">L134</a><div class="mdoc"><p>Frontend + Encoder. Note that this method is used by asr_inference.py</p>
+<h4>Args:</h4>
+<ul>
+<li><code>speech</code> — (Batch, Length, ...)</li>
+<li><code>speech_lengths</code> — (Batch, )</li>
+<li><code>ind</code> — int</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def encode(
+ self,
+ speech: torch.Tensor,
+ speech_lengths: torch.Tensor,
+ **kwargs,
+ ) -&gt; Tuple[torch.Tensor, torch.Tensor]:
+ &quot;&quot;&quot;Frontend + Encoder. Note that this method is used by asr_inference.py
+ Args:
+ speech: (Batch, Length, ...)
+ speech_lengths: (Batch, )
+ ind: int
+ &quot;&quot;&quot;
+
+ # Data augmentation
+ if self.specaug is not None and self.training:
+ speech, speech_lengths = self.specaug(speech, speech_lengths)
+
+ # Normalization for feature: e.g. Global-CMVN, Utterance-CMVN
+ if self.normalize is not None:
+ speech, speech_lengths = self.normalize(speech, speech_lengths)
+
+ # Forward encoder
+ # feats: (Batch, Length, Dim)
+ # -&gt; encoder_out: (Batch, Length2, Dim2)
+ encoder_out, encoder_out_lens = self.encoder(speech, speech_lengths)
+
+ return encoder_out, encoder_out_lens
+</code></pre></details></div><div class="mblk"><code>.inference(data_in, data_lengths, key, tokenizer, frontend, **kwargs)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/ctc/model.py#L189" target="_blank" class="gh-link">L189</a><div class="mdoc"><p>Run inference on input data.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>data_in</code> — Input data (audio samples, file paths, or text).</li>
+<li><code>data_lengths</code> — Lengths of each input sample in the batch.</li>
+<li><code>key</code> — Sample identifiers.</li>
+<li><code>tokenizer</code> — Tokenizer instance for text encoding/decoding.</li>
+<li><code>frontend</code> — Audio frontend for feature extraction.</li>
+<li><code>**kwargs</code> — Additional keyword arguments.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def inference(
+ self,
+ data_in,
+ data_lengths=None,
+ key: list = None,
+ tokenizer=None,
+ frontend=None,
+ **kwargs,
+ ):
+
+ &quot;&quot;&quot;Run inference on input data.
+
+ Args:
+ data_in: Input data (audio samples, file paths, or text).
+ data_lengths: Lengths of each input sample in the batch.
+ key: Sample identifiers.
+ tokenizer: Tokenizer instance for text encoding/decoding.
+ frontend: Audio frontend for feature extraction.
+ **kwargs: Additional keyword arguments.
+ &quot;&quot;&quot;
+ if kwargs.get(&quot;batch_size&quot;, 1) &gt; 1:
+ raise NotImplementedError(&quot;batch decoding is not implemented&quot;)
+
+ meta_data = {}
+ if (
+ isinstance(data_in, torch.Tensor) and kwargs.get(&quot;data_type&quot;, &quot;sound&quot;) == &quot;fbank&quot;
+ ): # fbank
+ speech, speech_lengths = data_in, data_lengths
+ if len(speech.shape) &lt; 3:
+ speech = speech[None, :, :]
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/ctc/model.py#L189" target="_blank" class="src-more">View full source →</a></details></div></div>
+<div class="api-detail" id="e47"><span class="dtag badge-m">method</span><h2><code>Transformer.forward(speech, speech_lengths, text, text_lengths, **kwargs)</code></h2><div class="dmod">funasr.models.ctc.Transformer · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/ctc/model.py#L89" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Encoder + Decoder + Calc loss</p>
+<h4>Args:</h4>
+<ul>
+<li><code>speech</code> — (Batch, Length, ...)</li>
+<li><code>speech_lengths</code> — (Batch, )</li>
+<li><code>text</code> — (Batch, Length)</li>
+<li><code>text_lengths</code> — (Batch,)</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def forward(
+ self,
+ speech: torch.Tensor,
+ speech_lengths: torch.Tensor,
+ text: torch.Tensor,
+ text_lengths: torch.Tensor,
+ **kwargs,
+ ) -&gt; Tuple[torch.Tensor, Dict[str, torch.Tensor], torch.Tensor]:
+ &quot;&quot;&quot;Encoder + Decoder + Calc loss
+ Args:
+ speech: (Batch, Length, ...)
+ speech_lengths: (Batch, )
+ text: (Batch, Length)
+ text_lengths: (Batch,)
+ &quot;&quot;&quot;
+ # import pdb;
+ # pdb.set_trace()
+ if len(text_lengths.size()) &gt; 1:
+ text_lengths = text_lengths[:, 0]
+ if len(speech_lengths.size()) &gt; 1:
+ speech_lengths = speech_lengths[:, 0]
+
+ batch_size = speech.shape[0]
+
+ # 1. Encoder
+ encoder_out, encoder_out_lens = self.encode(speech, speech_lengths)
+
+ loss_ctc, cer_ctc = None, None
+ stats = dict()
+
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/ctc/model.py#L89" target="_blank" class="src-more">View full source on GitHub →</a></details></div>
+<div class="api-detail" id="e48"><span class="dtag badge-m">method</span><h2><code>Transformer.encode(speech, speech_lengths, **kwargs)</code></h2><div class="dmod">funasr.models.ctc.Transformer · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/ctc/model.py#L134" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Frontend + Encoder. Note that this method is used by asr_inference.py</p>
+<h4>Args:</h4>
+<ul>
+<li><code>speech</code> — (Batch, Length, ...)</li>
+<li><code>speech_lengths</code> — (Batch, )</li>
+<li><code>ind</code> — int</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def encode(
+ self,
+ speech: torch.Tensor,
+ speech_lengths: torch.Tensor,
+ **kwargs,
+ ) -&gt; Tuple[torch.Tensor, torch.Tensor]:
+ &quot;&quot;&quot;Frontend + Encoder. Note that this method is used by asr_inference.py
+ Args:
+ speech: (Batch, Length, ...)
+ speech_lengths: (Batch, )
+ ind: int
+ &quot;&quot;&quot;
+
+ # Data augmentation
+ if self.specaug is not None and self.training:
+ speech, speech_lengths = self.specaug(speech, speech_lengths)
+
+ # Normalization for feature: e.g. Global-CMVN, Utterance-CMVN
+ if self.normalize is not None:
+ speech, speech_lengths = self.normalize(speech, speech_lengths)
+
+ # Forward encoder
+ # feats: (Batch, Length, Dim)
+ # -&gt; encoder_out: (Batch, Length2, Dim2)
+ encoder_out, encoder_out_lens = self.encoder(speech, speech_lengths)
+
+ return encoder_out, encoder_out_lens
+</code></pre></details></div>
+<div class="api-detail" id="e49"><span class="dtag badge-m">method</span><h2><code>Transformer.inference(data_in, data_lengths, key, tokenizer, frontend, **kwargs)</code></h2><div class="dmod">funasr.models.ctc.Transformer · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/ctc/model.py#L189" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Run inference on input data.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>data_in</code> — Input data (audio samples, file paths, or text).</li>
+<li><code>data_lengths</code> — Lengths of each input sample in the batch.</li>
+<li><code>key</code> — Sample identifiers.</li>
+<li><code>tokenizer</code> — Tokenizer instance for text encoding/decoding.</li>
+<li><code>frontend</code> — Audio frontend for feature extraction.</li>
+<li><code>**kwargs</code> — Additional keyword arguments.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def inference(
+ self,
+ data_in,
+ data_lengths=None,
+ key: list = None,
+ tokenizer=None,
+ frontend=None,
+ **kwargs,
+ ):
+
+ &quot;&quot;&quot;Run inference on input data.
+
+ Args:
+ data_in: Input data (audio samples, file paths, or text).
+ data_lengths: Lengths of each input sample in the batch.
+ key: Sample identifiers.
+ tokenizer: Tokenizer instance for text encoding/decoding.
+ frontend: Audio frontend for feature extraction.
+ **kwargs: Additional keyword arguments.
+ &quot;&quot;&quot;
+ if kwargs.get(&quot;batch_size&quot;, 1) &gt; 1:
+ raise NotImplementedError(&quot;batch decoding is not implemented&quot;)
+
+ meta_data = {}
+ if (
+ isinstance(data_in, torch.Tensor) and kwargs.get(&quot;data_type&quot;, &quot;sound&quot;) == &quot;fbank&quot;
+ ): # fbank
+ speech, speech_lengths = data_in, data_lengths
+ if len(speech.shape) &lt; 3:
+ speech = speech[None, :, :]
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/ctc/model.py#L189" target="_blank" class="src-more">View full source on GitHub →</a></details></div>
+<div class="api-detail" id="e50"><span class="dtag badge-c">class</span><h2>EBranchformer</h2><div class="dmod">funasr.models.e_branchformer · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/e_branchformer/model.py#L8" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><ul>
+<li><code>E</code> — Branchformer: Enhanced Branchformer with improved merging.</li>
+</ul>
+<p>Uses element-wise merging instead of concatenation for parallel branches,</p>
+<p>resulting in better parameter efficiency.</p>
+<p>Inherits Transformer pipeline.</p>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code>class EBranchformer(Transformer):
+ &quot;&quot;&quot;E-Branchformer: Enhanced Branchformer with improved merging.
+
+ Uses element-wise merging instead of concatenation for parallel branches,
+ resulting in better parameter efficiency.
+
+ Inherits Transformer pipeline.
+ &quot;&quot;&quot;
+
+ def __init__(
+ self,
+ *args,
+ **kwargs,
+ ):
+
+ &quot;&quot;&quot;Initialize EBranchformer.
+
+ Args:
+ *args: Variable positional arguments.
+ **kwargs: Additional keyword arguments.
+ &quot;&quot;&quot;
+ super().__init__(*args, **kwargs)
+</code></pre></details></div>
+<div class="api-detail" id="e51"><span class="dtag badge-c">class</span><h2>EParaformer</h2><div class="dmod">funasr.models.e_paraformer · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/e_paraformer/model.py#L31" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><ul>
+<li><code>E</code> — Paraformer: Enhanced Paraformer with streaming support.</li>
+</ul>
+<p>Extended Paraformer supporting both offline and streaming modes</p>
+<p>through dynamic masking in the encoder. Used for 2-pass decoding</p>
+<p>where first pass provides streaming results and second pass refines.</p>
+<ul>
+<li><code>Output</code> — {&quot;key&quot;: str, &quot;text&quot;: str, &quot;timestamp&quot;: [[start_ms, end_ms], ...]}</li>
+<li><code>Author</code> — Speech Lab of DAMO Academy, Alibaba Group</li>
+<li><code>Paraformer</code> — Fast and Accurate Parallel Transformer for Non-autoregressive End-to-End Speech Recognition</li>
+</ul>
+<p>https://arxiv.org/abs/2206.08317</p>
+<ul>
+<li><code>Author</code> — Kun Zou, [email protected]</li>
+<li><code>E</code> — Paraformer: A Faster and Better Parallel Transformer for Non-autoregressive End-to-End Mandarin Speech Recognition</li>
+</ul>
+<p>https://www.isca-archive.org/interspeech_2024/zou24_interspeech.pdf</p>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code>class EParaformer(torch.nn.Module):
+ &quot;&quot;&quot;E-Paraformer: Enhanced Paraformer with streaming support.
+
+ Extended Paraformer supporting both offline and streaming modes
+ through dynamic masking in the encoder. Used for 2-pass decoding
+ where first pass provides streaming results and second pass refines.
+
+ Output: {&quot;key&quot;: str, &quot;text&quot;: str, &quot;timestamp&quot;: [[start_ms, end_ms], ...]}
+
+ Author: Speech Lab of DAMO Academy, Alibaba Group
+ Paraformer: Fast and Accurate Parallel Transformer for Non-autoregressive End-to-End Speech Recognition
+ https://arxiv.org/abs/2206.08317
+ Author: Kun Zou, [email protected]
+ E-Paraformer: A Faster and Better Parallel Transformer for Non-autoregressive End-to-End Mandarin Speech Recognition
+ https://www.isca-archive.org/interspeech_2024/zou24_interspeech.pdf
+ &quot;&quot;&quot;
+
+ def __init__(
+ self,
+ specaug: Optional[str] = None,
+ specaug_conf: Optional[Dict] = None,
+ normalize: str = None,
+ normalize_conf: Optional[Dict] = None,
+ encoder: str = None,
+ encoder_conf: Optional[Dict] = None,
+ decoder: str = None,
+ decoder_conf: Optional[Dict] = None,
+ ctc: str = None,
+ ctc_conf: Optional[Dict] = None,
+ predictor: str = None,
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/e_paraformer/model.py#L31" target="_blank" class="src-more">View full source on GitHub →</a></details><h3 class="mt">Methods</h3><div class="mblk"><code>.forward(speech, speech_lengths, text, text_lengths, **kwargs)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/e_paraformer/model.py#L220" target="_blank" class="gh-link">L220</a><div class="mdoc"><p>Encoder + Decoder + Calc loss</p>
+<h4>Args:</h4>
+<ul>
+<li><code>speech</code> — (Batch, Length, ...)</li>
+<li><code>speech_lengths</code> — (Batch, )</li>
+<li><code>text</code> — (Batch, Length)</li>
+<li><code>text_lengths</code> — (Batch,)</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def forward(
+ self,
+ speech: torch.Tensor,
+ speech_lengths: torch.Tensor,
+ text: torch.Tensor,
+ text_lengths: torch.Tensor,
+ **kwargs,
+ ) -&gt; Tuple[torch.Tensor, Dict[str, torch.Tensor], torch.Tensor]:
+ &quot;&quot;&quot;Encoder + Decoder + Calc loss
+ Args:
+ speech: (Batch, Length, ...)
+ speech_lengths: (Batch, )
+ text: (Batch, Length)
+ text_lengths: (Batch,)
+ &quot;&quot;&quot;
+ if len(text_lengths.size()) &gt; 1:
+ text_lengths = text_lengths[:, 0]
+ if len(speech_lengths.size()) &gt; 1:
+ speech_lengths = speech_lengths[:, 0]
+
+ batch_size = speech.shape[0]
+
+ # Encoder
+ encoder_out, encoder_out_lens = self.encode(speech, speech_lengths)
+
+ loss_ctc, cer_ctc = None, None
+ loss_pre = None
+ stats = dict()
+
+ # decoder: CTC branch
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/e_paraformer/model.py#L220" target="_blank" class="src-more">View full source →</a></details></div><div class="mblk"><code>.encode(speech, speech_lengths, **kwargs)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/e_paraformer/model.py#L292" target="_blank" class="gh-link">L292</a><div class="mdoc"><p>Encoder. Note that this method is used by asr_inference.py</p>
+<h4>Args:</h4>
+<ul>
+<li><code>speech</code> — (Batch, Length, ...)</li>
+<li><code>speech_lengths</code> — (Batch, )</li>
+<li><code>ind</code> — int</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def encode(
+ self,
+ speech: torch.Tensor,
+ speech_lengths: torch.Tensor,
+ **kwargs,
+ ) -&gt; Tuple[torch.Tensor, torch.Tensor]:
+ &quot;&quot;&quot;Encoder. Note that this method is used by asr_inference.py
+ Args:
+ speech: (Batch, Length, ...)
+ speech_lengths: (Batch, )
+ ind: int
+ &quot;&quot;&quot;
+ with autocast(False):
+
+ # Data augmentation
+ if self.specaug is not None and self.training:
+ speech, speech_lengths = self.specaug(speech, speech_lengths)
+
+ # Normalization for feature: e.g. Global-CMVN, Utterance-CMVN
+ if self.normalize is not None:
+ speech, speech_lengths = self.normalize(speech, speech_lengths)
+
+ # Forward encoder
+ encoder_out, encoder_out_lens, _ = self.encoder(speech, speech_lengths)
+ if isinstance(encoder_out, tuple):
+ encoder_out = encoder_out[0]
+
+ return encoder_out, encoder_out_lens
+</code></pre></details></div><div class="mblk"><code>.calc_predictor(encoder_out, encoder_out_lens)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/e_paraformer/model.py#L321" target="_blank" class="gh-link">L321</a><div class="mdoc"><p>Calc predictor.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>encoder_out</code> — Encoder output tensor.</li>
+<li><code>encoder_out_lens</code> — Encoder output lengths.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def calc_predictor(self, encoder_out, encoder_out_lens):
+
+ &quot;&quot;&quot;Calc predictor.
+
+ Args:
+ encoder_out: Encoder output tensor.
+ encoder_out_lens: Encoder output lengths.
+ &quot;&quot;&quot;
+ encoder_out_mask = (
+ ~make_pad_mask(encoder_out_lens, maxlen=encoder_out.size(1))[:, None, :]
+ ).to(encoder_out.device)
+ pre_acoustic_embeds, pre_token_length, alphas, pre_peak_index = self.predictor(
+ encoder_out, None, encoder_out_mask, ignore_id=self.ignore_id
+ )
+ return pre_acoustic_embeds, pre_token_length, alphas, pre_peak_index
+</code></pre></details></div><div class="mblk"><code>.cal_decoder_with_predictor(encoder_out, encoder_out_lens, sematic_embeds, ys_pad_lens)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/e_paraformer/model.py#L337" target="_blank" class="gh-link">L337</a><div class="mdoc"><p>Cal decoder with predictor.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>encoder_out</code> — Encoder output tensor.</li>
+<li><code>encoder_out_lens</code> — Encoder output lengths.</li>
+<li><code>sematic_embeds</code> — TODO.</li>
+<li><code>ys_pad_lens</code> — Lengths of ys_pad.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def cal_decoder_with_predictor(
+ self, encoder_out, encoder_out_lens, sematic_embeds, ys_pad_lens
+ ):
+
+ &quot;&quot;&quot;Cal decoder with predictor.
+
+ Args:
+ encoder_out: Encoder output tensor.
+ encoder_out_lens: Encoder output lengths.
+ sematic_embeds: TODO.
+ ys_pad_lens: Lengths of ys_pad.
+ &quot;&quot;&quot;
+ decoder_outs = self.decoder(encoder_out, encoder_out_lens, sematic_embeds, ys_pad_lens)
+ decoder_out = decoder_outs[0]
+ decoder_out = torch.log_softmax(decoder_out, dim=-1)
+ return decoder_out, ys_pad_lens
+</code></pre></details></div><div class="mblk"><code>.sampler(encoder_out, encoder_out_lens, ys_pad, ys_pad_lens, pre_acoustic_embeds)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/e_paraformer/model.py#L425" target="_blank" class="gh-link">L425</a><div class="mdoc"><p>Sampler.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>encoder_out</code> — Encoder output tensor.</li>
+<li><code>encoder_out_lens</code> — Encoder output lengths.</li>
+<li><code>ys_pad</code> — TODO.</li>
+<li><code>ys_pad_lens</code> — Lengths of ys_pad.</li>
+<li><code>pre_acoustic_embeds</code> — TODO.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def sampler(self, encoder_out, encoder_out_lens, ys_pad, ys_pad_lens, pre_acoustic_embeds):
+
+ &quot;&quot;&quot;Sampler.
+
+ Args:
+ encoder_out: Encoder output tensor.
+ encoder_out_lens: Encoder output lengths.
+ ys_pad: TODO.
+ ys_pad_lens: Lengths of ys_pad.
+ pre_acoustic_embeds: TODO.
+ &quot;&quot;&quot;
+ tgt_mask = (~make_pad_mask(ys_pad_lens, maxlen=ys_pad_lens.max())[:, :, None]).to(
+ ys_pad.device
+ )
+ ys_pad_masked = ys_pad * tgt_mask[:, :, 0]
+ if self.share_embedding:
+ ys_pad_embed = self.decoder.output_layer.weight[ys_pad_masked]
+ else:
+ ys_pad_embed = self.decoder.embed(ys_pad_masked)
+ with torch.no_grad():
+ decoder_outs = self.decoder(
+ encoder_out, encoder_out_lens, pre_acoustic_embeds, ys_pad_lens
+ )
+ decoder_out, _ = decoder_outs[0], decoder_outs[1]
+ pred_tokens = decoder_out.argmax(-1)
+ nonpad_positions = ys_pad.ne(self.ignore_id)
+ seq_lens = (nonpad_positions).sum(1)
+ same_num = ((pred_tokens == ys_pad) &amp; nonpad_positions).sum(1)
+ input_mask = torch.ones_like(nonpad_positions)
+ bsz, seq_len = ys_pad.size()
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/e_paraformer/model.py#L425" target="_blank" class="src-more">View full source →</a></details></div><div class="mblk"><code>.sampler_with_grad(encoder_out, encoder_out_lens, ys_pad, ys_pad_lens, pre_acoustic_embeds)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/e_paraformer/model.py#L474" target="_blank" class="gh-link">L474</a><div class="mdoc"><p>Sampler with grad.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>encoder_out</code> — Encoder output tensor.</li>
+<li><code>encoder_out_lens</code> — Encoder output lengths.</li>
+<li><code>ys_pad</code> — TODO.</li>
+<li><code>ys_pad_lens</code> — Lengths of ys_pad.</li>
+<li><code>pre_acoustic_embeds</code> — TODO.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def sampler_with_grad(self, encoder_out, encoder_out_lens, ys_pad, ys_pad_lens, pre_acoustic_embeds):
+
+ &quot;&quot;&quot;Sampler with grad.
+
+ Args:
+ encoder_out: Encoder output tensor.
+ encoder_out_lens: Encoder output lengths.
+ ys_pad: TODO.
+ ys_pad_lens: Lengths of ys_pad.
+ pre_acoustic_embeds: TODO.
+ &quot;&quot;&quot;
+ tgt_mask = (~make_pad_mask(ys_pad_lens, maxlen=ys_pad_lens.max())[:, :, None]).to(
+ ys_pad.device
+ )
+ ys_pad_masked = ys_pad * tgt_mask[:, :, 0]
+ if self.share_embedding:
+ ys_pad_embed = self.decoder.output_layer.weight[ys_pad_masked]
+ else:
+ ys_pad_embed = self.decoder.embed(ys_pad_masked)
+ decoder_outs = self.decoder(
+ encoder_out, encoder_out_lens, pre_acoustic_embeds, ys_pad_lens
+ )
+ decoder_out, _ = decoder_outs[0], decoder_outs[1]
+ pred_tokens = decoder_out.argmax(-1)
+ nonpad_positions = ys_pad.ne(self.ignore_id)
+ seq_lens = (nonpad_positions).sum(1)
+ same_num = ((pred_tokens == ys_pad) &amp; nonpad_positions).sum(1)
+ input_mask = torch.ones_like(nonpad_positions)
+ bsz, seq_len = ys_pad.size()
+ for li in range(bsz):
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/e_paraformer/model.py#L474" target="_blank" class="src-more">View full source →</a></details></div><div class="mblk"><code>.init_beam_search(**kwargs)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/e_paraformer/model.py#L548" target="_blank" class="gh-link">L548</a><div class="mdoc"><p>Init beam search.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>**kwargs</code> — Additional keyword arguments.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def init_beam_search(
+ self,
+ **kwargs,
+ ):
+ &quot;&quot;&quot;Init beam search.
+
+ Args:
+ **kwargs: Additional keyword arguments.
+ &quot;&quot;&quot;
+ from funasr.models.paraformer.search import BeamSearchPara
+ from funasr.models.transformer.scorers.ctc import CTCPrefixScorer
+ from funasr.models.transformer.scorers.length_bonus import LengthBonus
+
+ # 1. Build ASR model
+ scorers = {}
+
+ if self.ctc != None:
+ ctc = CTCPrefixScorer(ctc=self.ctc, eos=self.eos)
+ scorers.update(ctc=ctc)
+ token_list = kwargs.get(&quot;token_list&quot;)
+ scorers.update(
+ length_bonus=LengthBonus(len(token_list)),
+ )
+
+ # 3. Build ngram model
+ # ngram is not supported now
+ ngram = None
+ scorers[&quot;ngram&quot;] = ngram
+
+ weights = dict(
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/e_paraformer/model.py#L548" target="_blank" class="src-more">View full source →</a></details></div><div class="mblk"><code>.inference(data_in, data_lengths, key, tokenizer, frontend, **kwargs)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/e_paraformer/model.py#L600" target="_blank" class="gh-link">L600</a><div class="mdoc"><p>Run inference on input data.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>data_in</code> — Input data (audio samples, file paths, or text).</li>
+<li><code>data_lengths</code> — Lengths of each input sample in the batch.</li>
+<li><code>key</code> — Sample identifiers.</li>
+<li><code>tokenizer</code> — Tokenizer instance for text encoding/decoding.</li>
+<li><code>frontend</code> — Audio frontend for feature extraction.</li>
+<li><code>**kwargs</code> — Additional keyword arguments.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def inference(
+ self,
+ data_in,
+ data_lengths=None,
+ key: list = None,
+ tokenizer=None,
+ frontend=None,
+ **kwargs,
+ ):
+ # init beamsearch
+ &quot;&quot;&quot;Run inference on input data.
+
+ Args:
+ data_in: Input data (audio samples, file paths, or text).
+ data_lengths: Lengths of each input sample in the batch.
+ key: Sample identifiers.
+ tokenizer: Tokenizer instance for text encoding/decoding.
+ frontend: Audio frontend for feature extraction.
+ **kwargs: Additional keyword arguments.
+ &quot;&quot;&quot;
+ is_use_ctc = kwargs.get(&quot;decoding_ctc_weight&quot;, 0.0) &gt; 0.00001 and self.ctc != None
+ is_use_lm = (
+ kwargs.get(&quot;lm_weight&quot;, 0.0) &gt; 0.00001 and kwargs.get(&quot;lm_file&quot;, None) is not None
+ )
+ pred_timestamp = kwargs.get(&quot;pred_timestamp&quot;, False)
+ if self.beam_search is None and (is_use_lm or is_use_ctc):
+ logging.info(&quot;enable beam_search&quot;)
+ self.init_beam_search(**kwargs)
+ self.nbest = kwargs.get(&quot;nbest&quot;, 1)
+
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/e_paraformer/model.py#L600" target="_blank" class="src-more">View full source →</a></details></div><div class="mblk"><code>.export(**kwargs)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/e_paraformer/model.py#L765" target="_blank" class="gh-link">L765</a><div class="mdoc"><p>Export.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>**kwargs</code> — Additional keyword arguments.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def export(self, **kwargs):
+ &quot;&quot;&quot;Export.
+
+ Args:
+ **kwargs: Additional keyword arguments.
+ &quot;&quot;&quot;
+ from .export_meta import export_rebuild_model
+
+ if &quot;max_seq_len&quot; not in kwargs:
+ kwargs[&quot;max_seq_len&quot;] = 512
+ models = export_rebuild_model(model=self, **kwargs)
+ return models
+</code></pre></details></div></div>
+<div class="api-detail" id="e52"><span class="dtag badge-m">method</span><h2><code>EParaformer.forward(speech, speech_lengths, text, text_lengths, **kwargs)</code></h2><div class="dmod">funasr.models.e_paraformer.EParaformer · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/e_paraformer/model.py#L220" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Encoder + Decoder + Calc loss</p>
+<h4>Args:</h4>
+<ul>
+<li><code>speech</code> — (Batch, Length, ...)</li>
+<li><code>speech_lengths</code> — (Batch, )</li>
+<li><code>text</code> — (Batch, Length)</li>
+<li><code>text_lengths</code> — (Batch,)</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def forward(
+ self,
+ speech: torch.Tensor,
+ speech_lengths: torch.Tensor,
+ text: torch.Tensor,
+ text_lengths: torch.Tensor,
+ **kwargs,
+ ) -&gt; Tuple[torch.Tensor, Dict[str, torch.Tensor], torch.Tensor]:
+ &quot;&quot;&quot;Encoder + Decoder + Calc loss
+ Args:
+ speech: (Batch, Length, ...)
+ speech_lengths: (Batch, )
+ text: (Batch, Length)
+ text_lengths: (Batch,)
+ &quot;&quot;&quot;
+ if len(text_lengths.size()) &gt; 1:
+ text_lengths = text_lengths[:, 0]
+ if len(speech_lengths.size()) &gt; 1:
+ speech_lengths = speech_lengths[:, 0]
+
+ batch_size = speech.shape[0]
+
+ # Encoder
+ encoder_out, encoder_out_lens = self.encode(speech, speech_lengths)
+
+ loss_ctc, cer_ctc = None, None
+ loss_pre = None
+ stats = dict()
+
+ # decoder: CTC branch
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/e_paraformer/model.py#L220" target="_blank" class="src-more">View full source on GitHub →</a></details></div>
+<div class="api-detail" id="e53"><span class="dtag badge-m">method</span><h2><code>EParaformer.encode(speech, speech_lengths, **kwargs)</code></h2><div class="dmod">funasr.models.e_paraformer.EParaformer · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/e_paraformer/model.py#L292" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Encoder. Note that this method is used by asr_inference.py</p>
+<h4>Args:</h4>
+<ul>
+<li><code>speech</code> — (Batch, Length, ...)</li>
+<li><code>speech_lengths</code> — (Batch, )</li>
+<li><code>ind</code> — int</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def encode(
+ self,
+ speech: torch.Tensor,
+ speech_lengths: torch.Tensor,
+ **kwargs,
+ ) -&gt; Tuple[torch.Tensor, torch.Tensor]:
+ &quot;&quot;&quot;Encoder. Note that this method is used by asr_inference.py
+ Args:
+ speech: (Batch, Length, ...)
+ speech_lengths: (Batch, )
+ ind: int
+ &quot;&quot;&quot;
+ with autocast(False):
+
+ # Data augmentation
+ if self.specaug is not None and self.training:
+ speech, speech_lengths = self.specaug(speech, speech_lengths)
+
+ # Normalization for feature: e.g. Global-CMVN, Utterance-CMVN
+ if self.normalize is not None:
+ speech, speech_lengths = self.normalize(speech, speech_lengths)
+
+ # Forward encoder
+ encoder_out, encoder_out_lens, _ = self.encoder(speech, speech_lengths)
+ if isinstance(encoder_out, tuple):
+ encoder_out = encoder_out[0]
+
+ return encoder_out, encoder_out_lens
+</code></pre></details></div>
+<div class="api-detail" id="e54"><span class="dtag badge-m">method</span><h2><code>EParaformer.calc_predictor(encoder_out, encoder_out_lens)</code></h2><div class="dmod">funasr.models.e_paraformer.EParaformer · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/e_paraformer/model.py#L321" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Calc predictor.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>encoder_out</code> — Encoder output tensor.</li>
+<li><code>encoder_out_lens</code> — Encoder output lengths.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def calc_predictor(self, encoder_out, encoder_out_lens):
+
+ &quot;&quot;&quot;Calc predictor.
+
+ Args:
+ encoder_out: Encoder output tensor.
+ encoder_out_lens: Encoder output lengths.
+ &quot;&quot;&quot;
+ encoder_out_mask = (
+ ~make_pad_mask(encoder_out_lens, maxlen=encoder_out.size(1))[:, None, :]
+ ).to(encoder_out.device)
+ pre_acoustic_embeds, pre_token_length, alphas, pre_peak_index = self.predictor(
+ encoder_out, None, encoder_out_mask, ignore_id=self.ignore_id
+ )
+ return pre_acoustic_embeds, pre_token_length, alphas, pre_peak_index
+</code></pre></details></div>
+<div class="api-detail" id="e55"><span class="dtag badge-m">method</span><h2><code>EParaformer.cal_decoder_with_predictor(encoder_out, encoder_out_lens, sematic_embeds, ys_pad_lens)</code></h2><div class="dmod">funasr.models.e_paraformer.EParaformer · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/e_paraformer/model.py#L337" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Cal decoder with predictor.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>encoder_out</code> — Encoder output tensor.</li>
+<li><code>encoder_out_lens</code> — Encoder output lengths.</li>
+<li><code>sematic_embeds</code> — TODO.</li>
+<li><code>ys_pad_lens</code> — Lengths of ys_pad.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def cal_decoder_with_predictor(
+ self, encoder_out, encoder_out_lens, sematic_embeds, ys_pad_lens
+ ):
+
+ &quot;&quot;&quot;Cal decoder with predictor.
+
+ Args:
+ encoder_out: Encoder output tensor.
+ encoder_out_lens: Encoder output lengths.
+ sematic_embeds: TODO.
+ ys_pad_lens: Lengths of ys_pad.
+ &quot;&quot;&quot;
+ decoder_outs = self.decoder(encoder_out, encoder_out_lens, sematic_embeds, ys_pad_lens)
+ decoder_out = decoder_outs[0]
+ decoder_out = torch.log_softmax(decoder_out, dim=-1)
+ return decoder_out, ys_pad_lens
+</code></pre></details></div>
+<div class="api-detail" id="e56"><span class="dtag badge-m">method</span><h2><code>EParaformer.sampler(encoder_out, encoder_out_lens, ys_pad, ys_pad_lens, pre_acoustic_embeds)</code></h2><div class="dmod">funasr.models.e_paraformer.EParaformer · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/e_paraformer/model.py#L425" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Sampler.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>encoder_out</code> — Encoder output tensor.</li>
+<li><code>encoder_out_lens</code> — Encoder output lengths.</li>
+<li><code>ys_pad</code> — TODO.</li>
+<li><code>ys_pad_lens</code> — Lengths of ys_pad.</li>
+<li><code>pre_acoustic_embeds</code> — TODO.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def sampler(self, encoder_out, encoder_out_lens, ys_pad, ys_pad_lens, pre_acoustic_embeds):
+
+ &quot;&quot;&quot;Sampler.
+
+ Args:
+ encoder_out: Encoder output tensor.
+ encoder_out_lens: Encoder output lengths.
+ ys_pad: TODO.
+ ys_pad_lens: Lengths of ys_pad.
+ pre_acoustic_embeds: TODO.
+ &quot;&quot;&quot;
+ tgt_mask = (~make_pad_mask(ys_pad_lens, maxlen=ys_pad_lens.max())[:, :, None]).to(
+ ys_pad.device
+ )
+ ys_pad_masked = ys_pad * tgt_mask[:, :, 0]
+ if self.share_embedding:
+ ys_pad_embed = self.decoder.output_layer.weight[ys_pad_masked]
+ else:
+ ys_pad_embed = self.decoder.embed(ys_pad_masked)
+ with torch.no_grad():
+ decoder_outs = self.decoder(
+ encoder_out, encoder_out_lens, pre_acoustic_embeds, ys_pad_lens
+ )
+ decoder_out, _ = decoder_outs[0], decoder_outs[1]
+ pred_tokens = decoder_out.argmax(-1)
+ nonpad_positions = ys_pad.ne(self.ignore_id)
+ seq_lens = (nonpad_positions).sum(1)
+ same_num = ((pred_tokens == ys_pad) &amp; nonpad_positions).sum(1)
+ input_mask = torch.ones_like(nonpad_positions)
+ bsz, seq_len = ys_pad.size()
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/e_paraformer/model.py#L425" target="_blank" class="src-more">View full source on GitHub →</a></details></div>
+<div class="api-detail" id="e57"><span class="dtag badge-m">method</span><h2><code>EParaformer.sampler_with_grad(encoder_out, encoder_out_lens, ys_pad, ys_pad_lens, pre_acoustic_embeds)</code></h2><div class="dmod">funasr.models.e_paraformer.EParaformer · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/e_paraformer/model.py#L474" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Sampler with grad.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>encoder_out</code> — Encoder output tensor.</li>
+<li><code>encoder_out_lens</code> — Encoder output lengths.</li>
+<li><code>ys_pad</code> — TODO.</li>
+<li><code>ys_pad_lens</code> — Lengths of ys_pad.</li>
+<li><code>pre_acoustic_embeds</code> — TODO.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def sampler_with_grad(self, encoder_out, encoder_out_lens, ys_pad, ys_pad_lens, pre_acoustic_embeds):
+
+ &quot;&quot;&quot;Sampler with grad.
+
+ Args:
+ encoder_out: Encoder output tensor.
+ encoder_out_lens: Encoder output lengths.
+ ys_pad: TODO.
+ ys_pad_lens: Lengths of ys_pad.
+ pre_acoustic_embeds: TODO.
+ &quot;&quot;&quot;
+ tgt_mask = (~make_pad_mask(ys_pad_lens, maxlen=ys_pad_lens.max())[:, :, None]).to(
+ ys_pad.device
+ )
+ ys_pad_masked = ys_pad * tgt_mask[:, :, 0]
+ if self.share_embedding:
+ ys_pad_embed = self.decoder.output_layer.weight[ys_pad_masked]
+ else:
+ ys_pad_embed = self.decoder.embed(ys_pad_masked)
+ decoder_outs = self.decoder(
+ encoder_out, encoder_out_lens, pre_acoustic_embeds, ys_pad_lens
+ )
+ decoder_out, _ = decoder_outs[0], decoder_outs[1]
+ pred_tokens = decoder_out.argmax(-1)
+ nonpad_positions = ys_pad.ne(self.ignore_id)
+ seq_lens = (nonpad_positions).sum(1)
+ same_num = ((pred_tokens == ys_pad) &amp; nonpad_positions).sum(1)
+ input_mask = torch.ones_like(nonpad_positions)
+ bsz, seq_len = ys_pad.size()
+ for li in range(bsz):
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/e_paraformer/model.py#L474" target="_blank" class="src-more">View full source on GitHub →</a></details></div>
+<div class="api-detail" id="e58"><span class="dtag badge-m">method</span><h2><code>EParaformer.init_beam_search(**kwargs)</code></h2><div class="dmod">funasr.models.e_paraformer.EParaformer · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/e_paraformer/model.py#L548" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Init beam search.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>**kwargs</code> — Additional keyword arguments.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def init_beam_search(
+ self,
+ **kwargs,
+ ):
+ &quot;&quot;&quot;Init beam search.
+
+ Args:
+ **kwargs: Additional keyword arguments.
+ &quot;&quot;&quot;
+ from funasr.models.paraformer.search import BeamSearchPara
+ from funasr.models.transformer.scorers.ctc import CTCPrefixScorer
+ from funasr.models.transformer.scorers.length_bonus import LengthBonus
+
+ # 1. Build ASR model
+ scorers = {}
+
+ if self.ctc != None:
+ ctc = CTCPrefixScorer(ctc=self.ctc, eos=self.eos)
+ scorers.update(ctc=ctc)
+ token_list = kwargs.get(&quot;token_list&quot;)
+ scorers.update(
+ length_bonus=LengthBonus(len(token_list)),
+ )
+
+ # 3. Build ngram model
+ # ngram is not supported now
+ ngram = None
+ scorers[&quot;ngram&quot;] = ngram
+
+ weights = dict(
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/e_paraformer/model.py#L548" target="_blank" class="src-more">View full source on GitHub →</a></details></div>
+<div class="api-detail" id="e59"><span class="dtag badge-m">method</span><h2><code>EParaformer.inference(data_in, data_lengths, key, tokenizer, frontend, **kwargs)</code></h2><div class="dmod">funasr.models.e_paraformer.EParaformer · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/e_paraformer/model.py#L600" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Run inference on input data.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>data_in</code> — Input data (audio samples, file paths, or text).</li>
+<li><code>data_lengths</code> — Lengths of each input sample in the batch.</li>
+<li><code>key</code> — Sample identifiers.</li>
+<li><code>tokenizer</code> — Tokenizer instance for text encoding/decoding.</li>
+<li><code>frontend</code> — Audio frontend for feature extraction.</li>
+<li><code>**kwargs</code> — Additional keyword arguments.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def inference(
+ self,
+ data_in,
+ data_lengths=None,
+ key: list = None,
+ tokenizer=None,
+ frontend=None,
+ **kwargs,
+ ):
+ # init beamsearch
+ &quot;&quot;&quot;Run inference on input data.
+
+ Args:
+ data_in: Input data (audio samples, file paths, or text).
+ data_lengths: Lengths of each input sample in the batch.
+ key: Sample identifiers.
+ tokenizer: Tokenizer instance for text encoding/decoding.
+ frontend: Audio frontend for feature extraction.
+ **kwargs: Additional keyword arguments.
+ &quot;&quot;&quot;
+ is_use_ctc = kwargs.get(&quot;decoding_ctc_weight&quot;, 0.0) &gt; 0.00001 and self.ctc != None
+ is_use_lm = (
+ kwargs.get(&quot;lm_weight&quot;, 0.0) &gt; 0.00001 and kwargs.get(&quot;lm_file&quot;, None) is not None
+ )
+ pred_timestamp = kwargs.get(&quot;pred_timestamp&quot;, False)
+ if self.beam_search is None and (is_use_lm or is_use_ctc):
+ logging.info(&quot;enable beam_search&quot;)
+ self.init_beam_search(**kwargs)
+ self.nbest = kwargs.get(&quot;nbest&quot;, 1)
+
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/e_paraformer/model.py#L600" target="_blank" class="src-more">View full source on GitHub →</a></details></div>
+<div class="api-detail" id="e60"><span class="dtag badge-m">method</span><h2><code>EParaformer.export(**kwargs)</code></h2><div class="dmod">funasr.models.e_paraformer.EParaformer · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/e_paraformer/model.py#L765" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Export.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>**kwargs</code> — Additional keyword arguments.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def export(self, **kwargs):
+ &quot;&quot;&quot;Export.
+
+ Args:
+ **kwargs: Additional keyword arguments.
+ &quot;&quot;&quot;
+ from .export_meta import export_rebuild_model
+
+ if &quot;max_seq_len&quot; not in kwargs:
+ kwargs[&quot;max_seq_len&quot;] = 512
+ models = export_rebuild_model(model=self, **kwargs)
+ return models
+</code></pre></details></div>
+<div class="api-detail" id="e61"><span class="dtag badge-c">class</span><h2>Emotion2vec</h2><div class="dmod">funasr.models.emotion2vec · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/emotion2vec/model.py#L40" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><ul>
+<li><code>Author</code> — Ziyang Ma, Zhisheng Zheng, Jiaxin Ye, Jinchao Li, Zhifu Gao, Shiliang Zhang, Xie Chen</li>
+<li><code>emotion2vec</code> — Self-Supervised Pre-Training for Speech Emotion Representation</li>
+</ul>
+<p>https://arxiv.org/abs/2312.15185</p>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code>class Emotion2vec(torch.nn.Module):
+ &quot;&quot;&quot;
+ Author: Ziyang Ma, Zhisheng Zheng, Jiaxin Ye, Jinchao Li, Zhifu Gao, Shiliang Zhang, Xie Chen
+ emotion2vec: Self-Supervised Pre-Training for Speech Emotion Representation
+ https://arxiv.org/abs/2312.15185
+ &quot;&quot;&quot;
+
+ def __init__(self, **kwargs):
+ &quot;&quot;&quot;Initialize Emotion2vec.
+
+ Args:
+ **kwargs: Additional keyword arguments.
+ &quot;&quot;&quot;
+ super().__init__()
+ # import pdb; pdb.set_trace()
+ cfg = OmegaConf.create(kwargs[&quot;model_conf&quot;])
+ self.cfg = cfg
+
+ make_layer_norm = partial(
+ torch.nn.LayerNorm, eps=cfg.get(&quot;norm_eps&quot;), elementwise_affine=cfg.get(&quot;norm_affine&quot;)
+ )
+
+ def make_block(drop_path, dim=None, heads=None):
+ &quot;&quot;&quot;Make block.
+
+ Args:
+ drop_path: TODO.
+ dim: TODO.
+ heads: TODO.
+ &quot;&quot;&quot;
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/emotion2vec/model.py#L40" target="_blank" class="src-more">View full source on GitHub →</a></details><h3 class="mt">Methods</h3><div class="mblk"><code>.forward(source, target, id, mode, padding_mask, mask, features_only, force_remove_masked, remove_extra_tokens, precomputed_mask, **kwargs)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/emotion2vec/model.py#L121" target="_blank" class="gh-link">L121</a><div class="mdoc"><p>Forward pass for training.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>source</code> — TODO.</li>
+<li><code>target</code> — TODO.</li>
+<li><code>id</code> — TODO.</li>
+<li><code>mode</code> — TODO.</li>
+<li><code>padding_mask</code> — TODO.</li>
+<li><code>mask</code> — TODO.</li>
+<li><code>features_only</code> — TODO.</li>
+<li><code>force_remove_masked</code> — TODO.</li>
+<li><code>remove_extra_tokens</code> — TODO.</li>
+<li><code>precomputed_mask</code> — TODO.</li>
+<li><code>**kwargs</code> — Additional keyword arguments.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def forward(
+ self,
+ source,
+ target=None,
+ id=None,
+ mode=None,
+ padding_mask=None,
+ mask=True,
+ features_only=False,
+ force_remove_masked=False,
+ remove_extra_tokens=True,
+ precomputed_mask=None,
+ **kwargs,
+ ):
+
+ &quot;&quot;&quot;Forward pass for training.
+
+ Args:
+ source: TODO.
+ target: TODO.
+ id: TODO.
+ mode: TODO.
+ padding_mask: TODO.
+ mask: TODO.
+ features_only: TODO.
+ force_remove_masked: TODO.
+ remove_extra_tokens: TODO.
+ precomputed_mask: TODO.
+ **kwargs: Additional keyword arguments.
+ &quot;&quot;&quot;
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/emotion2vec/model.py#L121" target="_blank" class="src-more">View full source →</a></details></div><div class="mblk"><code>.extract_features(source, mode, padding_mask, mask, remove_extra_tokens)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/emotion2vec/model.py#L212" target="_blank" class="gh-link">L212</a><div class="mdoc"><p>Extract features.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>source</code> — TODO.</li>
+<li><code>mode</code> — TODO.</li>
+<li><code>padding_mask</code> — TODO.</li>
+<li><code>mask</code> — TODO.</li>
+<li><code>remove_extra_tokens</code> — TODO.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def extract_features(
+ self, source, mode=None, padding_mask=None, mask=False, remove_extra_tokens=True
+ ):
+ &quot;&quot;&quot;Extract features.
+
+ Args:
+ source: TODO.
+ mode: TODO.
+ padding_mask: TODO.
+ mask: TODO.
+ remove_extra_tokens: TODO.
+ &quot;&quot;&quot;
+ res = self.forward(
+ source,
+ mode=mode,
+ padding_mask=padding_mask,
+ mask=mask,
+ features_only=True,
+ remove_extra_tokens=remove_extra_tokens,
+ )
+ return res
+</code></pre></details></div><div class="mblk"><code>.inference(data_in, data_lengths, key, tokenizer, frontend, **kwargs)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/emotion2vec/model.py#L234" target="_blank" class="gh-link">L234</a><div class="mdoc"><p>Run inference on input data.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>data_in</code> — Input data (audio samples, file paths, or text).</li>
+<li><code>data_lengths</code> — Lengths of each input sample in the batch.</li>
+<li><code>key</code> — Sample identifiers.</li>
+<li><code>tokenizer</code> — Tokenizer instance for text encoding/decoding.</li>
+<li><code>frontend</code> — Audio frontend for feature extraction.</li>
+<li><code>**kwargs</code> — Additional keyword arguments.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def inference(
+ self,
+ data_in,
+ data_lengths=None,
+ key: list = None,
+ tokenizer=None,
+ frontend=None,
+ **kwargs,
+ ):
+
+ # if source_file.endswith(&#x27;.wav&#x27;):
+ # wav, sr = sf.read(source_file)
+ # channel = sf.info(source_file).channels
+ # assert sr == 16e3, &quot;Sample rate should be 16kHz, but got {}in file {}&quot;.format(sr, source_file)
+ # assert channel == 1, &quot;Channel should be 1, but got {} in file {}&quot;.format(channel, source_file)
+ &quot;&quot;&quot;Run inference on input data.
+
+ Args:
+ data_in: Input data (audio samples, file paths, or text).
+ data_lengths: Lengths of each input sample in the batch.
+ key: Sample identifiers.
+ tokenizer: Tokenizer instance for text encoding/decoding.
+ frontend: Audio frontend for feature extraction.
+ **kwargs: Additional keyword arguments.
+ &quot;&quot;&quot;
+ granularity = kwargs.get(&quot;granularity&quot;, &quot;utterance&quot;)
+ extract_embedding = kwargs.get(&quot;extract_embedding&quot;, True)
+ if self.proj is None:
+ extract_embedding = True
+ meta_data = {}
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/emotion2vec/model.py#L234" target="_blank" class="src-more">View full source →</a></details></div><div class="mblk"><code>.export(**kwargs)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/emotion2vec/model.py#L320" target="_blank" class="gh-link">L320</a><div class="mdoc"><p>Export.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>**kwargs</code> — Additional keyword arguments.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def export(self, **kwargs):
+ &quot;&quot;&quot;Export.
+
+ Args:
+ **kwargs: Additional keyword arguments.
+ &quot;&quot;&quot;
+ from .export_meta import export_rebuild_model
+
+ models = export_rebuild_model(model=self, **kwargs)
+ return models
+</code></pre></details></div></div>
+<div class="api-detail" id="e62"><span class="dtag badge-m">method</span><h2><code>Emotion2vec.forward(source, target, id, mode, padding_mask, mask, features_only, force_remove_masked, remove_extra_tokens, precomputed_mask, **kwargs)</code></h2><div class="dmod">funasr.models.emotion2vec.Emotion2vec · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/emotion2vec/model.py#L121" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Forward pass for training.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>source</code> — TODO.</li>
+<li><code>target</code> — TODO.</li>
+<li><code>id</code> — TODO.</li>
+<li><code>mode</code> — TODO.</li>
+<li><code>padding_mask</code> — TODO.</li>
+<li><code>mask</code> — TODO.</li>
+<li><code>features_only</code> — TODO.</li>
+<li><code>force_remove_masked</code> — TODO.</li>
+<li><code>remove_extra_tokens</code> — TODO.</li>
+<li><code>precomputed_mask</code> — TODO.</li>
+<li><code>**kwargs</code> — Additional keyword arguments.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def forward(
+ self,
+ source,
+ target=None,
+ id=None,
+ mode=None,
+ padding_mask=None,
+ mask=True,
+ features_only=False,
+ force_remove_masked=False,
+ remove_extra_tokens=True,
+ precomputed_mask=None,
+ **kwargs,
+ ):
+
+ &quot;&quot;&quot;Forward pass for training.
+
+ Args:
+ source: TODO.
+ target: TODO.
+ id: TODO.
+ mode: TODO.
+ padding_mask: TODO.
+ mask: TODO.
+ features_only: TODO.
+ force_remove_masked: TODO.
+ remove_extra_tokens: TODO.
+ precomputed_mask: TODO.
+ **kwargs: Additional keyword arguments.
+ &quot;&quot;&quot;
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/emotion2vec/model.py#L121" target="_blank" class="src-more">View full source on GitHub →</a></details></div>
+<div class="api-detail" id="e63"><span class="dtag badge-m">method</span><h2><code>Emotion2vec.extract_features(source, mode, padding_mask, mask, remove_extra_tokens)</code></h2><div class="dmod">funasr.models.emotion2vec.Emotion2vec · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/emotion2vec/model.py#L212" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Extract features.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>source</code> — TODO.</li>
+<li><code>mode</code> — TODO.</li>
+<li><code>padding_mask</code> — TODO.</li>
+<li><code>mask</code> — TODO.</li>
+<li><code>remove_extra_tokens</code> — TODO.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def extract_features(
+ self, source, mode=None, padding_mask=None, mask=False, remove_extra_tokens=True
+ ):
+ &quot;&quot;&quot;Extract features.
+
+ Args:
+ source: TODO.
+ mode: TODO.
+ padding_mask: TODO.
+ mask: TODO.
+ remove_extra_tokens: TODO.
+ &quot;&quot;&quot;
+ res = self.forward(
+ source,
+ mode=mode,
+ padding_mask=padding_mask,
+ mask=mask,
+ features_only=True,
+ remove_extra_tokens=remove_extra_tokens,
+ )
+ return res
+</code></pre></details></div>
+<div class="api-detail" id="e64"><span class="dtag badge-m">method</span><h2><code>Emotion2vec.inference(data_in, data_lengths, key, tokenizer, frontend, **kwargs)</code></h2><div class="dmod">funasr.models.emotion2vec.Emotion2vec · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/emotion2vec/model.py#L234" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Run inference on input data.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>data_in</code> — Input data (audio samples, file paths, or text).</li>
+<li><code>data_lengths</code> — Lengths of each input sample in the batch.</li>
+<li><code>key</code> — Sample identifiers.</li>
+<li><code>tokenizer</code> — Tokenizer instance for text encoding/decoding.</li>
+<li><code>frontend</code> — Audio frontend for feature extraction.</li>
+<li><code>**kwargs</code> — Additional keyword arguments.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def inference(
+ self,
+ data_in,
+ data_lengths=None,
+ key: list = None,
+ tokenizer=None,
+ frontend=None,
+ **kwargs,
+ ):
+
+ # if source_file.endswith(&#x27;.wav&#x27;):
+ # wav, sr = sf.read(source_file)
+ # channel = sf.info(source_file).channels
+ # assert sr == 16e3, &quot;Sample rate should be 16kHz, but got {}in file {}&quot;.format(sr, source_file)
+ # assert channel == 1, &quot;Channel should be 1, but got {} in file {}&quot;.format(channel, source_file)
+ &quot;&quot;&quot;Run inference on input data.
+
+ Args:
+ data_in: Input data (audio samples, file paths, or text).
+ data_lengths: Lengths of each input sample in the batch.
+ key: Sample identifiers.
+ tokenizer: Tokenizer instance for text encoding/decoding.
+ frontend: Audio frontend for feature extraction.
+ **kwargs: Additional keyword arguments.
+ &quot;&quot;&quot;
+ granularity = kwargs.get(&quot;granularity&quot;, &quot;utterance&quot;)
+ extract_embedding = kwargs.get(&quot;extract_embedding&quot;, True)
+ if self.proj is None:
+ extract_embedding = True
+ meta_data = {}
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/emotion2vec/model.py#L234" target="_blank" class="src-more">View full source on GitHub →</a></details></div>
+<div class="api-detail" id="e65"><span class="dtag badge-m">method</span><h2><code>Emotion2vec.export(**kwargs)</code></h2><div class="dmod">funasr.models.emotion2vec.Emotion2vec · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/emotion2vec/model.py#L320" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Export.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>**kwargs</code> — Additional keyword arguments.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def export(self, **kwargs):
+ &quot;&quot;&quot;Export.
+
+ Args:
+ **kwargs: Additional keyword arguments.
+ &quot;&quot;&quot;
+ from .export_meta import export_rebuild_model
+
+ models = export_rebuild_model(model=self, **kwargs)
+ return models
+</code></pre></details></div>
+<div class="api-detail" id="e66"><span class="dtag badge-c">class</span><h2>ERes2NetV2SV</h2><div class="dmod">funasr.models.eres2net · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/eres2net/model.py#L21" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><ul>
+<li><code>ERes2NetV2</code> — Enhanced Res2Net v2 for Speaker Verification.</li>
+</ul>
+<p>Improved speaker embedding model based on Res2Net architecture with</p>
+<ul>
+<li><code>multi</code> — scale feature aggregation. Provides 192-dim speaker embeddings</li>
+</ul>
+<p>for speaker verification and diarization.</p>
+<p>Better than CAM++ for short-duration audio (&lt; 3s) speaker feature extraction.</p>
+<ul>
+<li><code>Output</code> — {&quot;spk_embedding&quot;: Tensor of shape (1, 192)}</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code>class ERes2NetV2SV(torch.nn.Module):
+ &quot;&quot;&quot;ERes2NetV2: Enhanced Res2Net v2 for Speaker Verification.
+
+ Improved speaker embedding model based on Res2Net architecture with
+ multi-scale feature aggregation. Provides 192-dim speaker embeddings
+ for speaker verification and diarization.
+
+ Better than CAM++ for short-duration audio (&lt; 3s) speaker feature extraction.
+
+ Output: {&quot;spk_embedding&quot;: Tensor of shape (1, 192)}
+ &quot;&quot;&quot;
+
+ def __init__(
+ self,
+ feat_dim=80,
+ embedding_size=192,
+ m_channels=64,
+ baseWidth=26,
+ scale=2,
+ expansion=2,
+ num_blocks=[3, 4, 6, 3],
+ pooling_func=&quot;TSTP&quot;,
+ two_emb_layer=False,
+ **kwargs,
+ ):
+ &quot;&quot;&quot;Initialize ERes2NetV2SV.
+
+ Args:
+ feat_dim: Size/dimension parameter.
+ embedding_size: Size/dimension parameter.
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/eres2net/model.py#L21" target="_blank" class="src-more">View full source on GitHub →</a></details><h3 class="mt">Methods</h3><div class="mblk"><code>.forward(x)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/eres2net/model.py#L97" target="_blank" class="gh-link">L97</a><div class="mdoc"><p>Forward pass for training.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>x</code> — TODO.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def forward(self, x):
+ &quot;&quot;&quot;Forward pass for training.
+
+ Args:
+ x: TODO.
+ &quot;&quot;&quot;
+ return self.model(x)
+</code></pre></details></div><div class="mblk"><code>.inference(data_in, data_lengths, key, tokenizer, frontend, **kwargs)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/eres2net/model.py#L105" target="_blank" class="gh-link">L105</a><div class="mdoc"><p>Run inference on input data.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>data_in</code> — Input data (audio samples, file paths, or text).</li>
+<li><code>data_lengths</code> — Lengths of each input sample in the batch.</li>
+<li><code>key</code> — Sample identifiers.</li>
+<li><code>tokenizer</code> — Tokenizer instance for text encoding/decoding.</li>
+<li><code>frontend</code> — Audio frontend for feature extraction.</li>
+<li><code>**kwargs</code> — Additional keyword arguments.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def inference(
+ self,
+ data_in,
+ data_lengths=None,
+ key: list = None,
+ tokenizer=None,
+ frontend=None,
+ **kwargs,
+ ):
+ &quot;&quot;&quot;Run inference on input data.
+
+ Args:
+ data_in: Input data (audio samples, file paths, or text).
+ data_lengths: Lengths of each input sample in the batch.
+ key: Sample identifiers.
+ tokenizer: Tokenizer instance for text encoding/decoding.
+ frontend: Audio frontend for feature extraction.
+ **kwargs: Additional keyword arguments.
+ &quot;&quot;&quot;
+ meta_data = {}
+ time1 = time.perf_counter()
+ audio_sample_list = load_audio_text_image_video(
+ data_in, fs=16000, audio_fs=kwargs.get(&quot;fs&quot;, 16000), data_type=&quot;sound&quot;
+ )
+ time2 = time.perf_counter()
+ meta_data[&quot;load_data&quot;] = f&quot;{time2 - time1:0.3f}&quot;
+ speech, speech_lengths, speech_times = extract_feature(audio_sample_list)
+ speech = speech.to(device=kwargs[&quot;device&quot;])
+ time3 = time.perf_counter()
+ meta_data[&quot;extract_feat&quot;] = f&quot;{time3 - time2:0.3f}&quot;
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/eres2net/model.py#L105" target="_blank" class="src-more">View full source →</a></details></div></div>
+<div class="api-detail" id="e67"><span class="dtag badge-m">method</span><h2><code>ERes2NetV2SV.forward(x)</code></h2><div class="dmod">funasr.models.eres2net.ERes2NetV2SV · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/eres2net/model.py#L97" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Forward pass for training.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>x</code> — TODO.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def forward(self, x):
+ &quot;&quot;&quot;Forward pass for training.
+
+ Args:
+ x: TODO.
+ &quot;&quot;&quot;
+ return self.model(x)
+</code></pre></details></div>
+<div class="api-detail" id="e68"><span class="dtag badge-m">method</span><h2><code>ERes2NetV2SV.inference(data_in, data_lengths, key, tokenizer, frontend, **kwargs)</code></h2><div class="dmod">funasr.models.eres2net.ERes2NetV2SV · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/eres2net/model.py#L105" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Run inference on input data.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>data_in</code> — Input data (audio samples, file paths, or text).</li>
+<li><code>data_lengths</code> — Lengths of each input sample in the batch.</li>
+<li><code>key</code> — Sample identifiers.</li>
+<li><code>tokenizer</code> — Tokenizer instance for text encoding/decoding.</li>
+<li><code>frontend</code> — Audio frontend for feature extraction.</li>
+<li><code>**kwargs</code> — Additional keyword arguments.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def inference(
+ self,
+ data_in,
+ data_lengths=None,
+ key: list = None,
+ tokenizer=None,
+ frontend=None,
+ **kwargs,
+ ):
+ &quot;&quot;&quot;Run inference on input data.
+
+ Args:
+ data_in: Input data (audio samples, file paths, or text).
+ data_lengths: Lengths of each input sample in the batch.
+ key: Sample identifiers.
+ tokenizer: Tokenizer instance for text encoding/decoding.
+ frontend: Audio frontend for feature extraction.
+ **kwargs: Additional keyword arguments.
+ &quot;&quot;&quot;
+ meta_data = {}
+ time1 = time.perf_counter()
+ audio_sample_list = load_audio_text_image_video(
+ data_in, fs=16000, audio_fs=kwargs.get(&quot;fs&quot;, 16000), data_type=&quot;sound&quot;
+ )
+ time2 = time.perf_counter()
+ meta_data[&quot;load_data&quot;] = f&quot;{time2 - time1:0.3f}&quot;
+ speech, speech_lengths, speech_times = extract_feature(audio_sample_list)
+ speech = speech.to(device=kwargs[&quot;device&quot;])
+ time3 = time.perf_counter()
+ meta_data[&quot;extract_feat&quot;] = f&quot;{time3 - time2:0.3f}&quot;
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/eres2net/model.py#L105" target="_blank" class="src-more">View full source on GitHub →</a></details></div>
+<div class="api-detail" id="e69"><span class="dtag badge-c">class</span><h2>FsmnKWS</h2><div class="dmod">funasr.models.fsmn_kws · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/fsmn_kws/model.py#L27" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><ul>
+<li><code>FSMN</code> — KWS: Keyword Spotting model using FSMN architecture.</li>
+</ul>
+<p>Detects predefined keywords/wake words in audio streams.</p>
+<p>Supports both offline and streaming operation.</p>
+<ul>
+<li><code>Output</code> — {&quot;key&quot;: str, &quot;value&quot;: detected_keyword_info}</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code>class FsmnKWS(torch.nn.Module):
+ &quot;&quot;&quot;FSMN-KWS: Keyword Spotting model using FSMN architecture.
+
+ Detects predefined keywords/wake words in audio streams.
+ Supports both offline and streaming operation.
+
+ Output: {&quot;key&quot;: str, &quot;value&quot;: detected_keyword_info}
+ &quot;&quot;&quot;
+
+ def __init__(
+ self,
+ specaug: Optional[str] = None,
+ specaug_conf: Optional[Dict] = None,
+ normalize: str = None,
+ normalize_conf: Optional[Dict] = None,
+ encoder: str = None,
+ encoder_conf: Optional[Dict] = None,
+ ctc: str = None,
+ ctc_conf: Optional[Dict] = None,
+ ctc_weight: float = 1.0,
+ input_size: int = 360,
+ vocab_size: int = -1,
+ ignore_id: int = -1,
+ blank_id: int = 0,
+ **kwargs,
+ ):
+ &quot;&quot;&quot;Initialize FsmnKWS.
+
+ Args:
+ specaug: TODO.
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/fsmn_kws/model.py#L27" target="_blank" class="src-more">View full source on GitHub →</a></details><h3 class="mt">Methods</h3><div class="mblk"><code>.forward(speech, speech_lengths, text, text_lengths, **kwargs)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/fsmn_kws/model.py#L104" target="_blank" class="gh-link">L104</a><div class="mdoc"><p>Encoder + Decoder + Calc loss</p>
+<h4>Args:</h4>
+<ul>
+<li><code>speech</code> — (Batch, Length, ...)</li>
+<li><code>speech_lengths</code> — (Batch, )</li>
+<li><code>text</code> — (Batch, Length)</li>
+<li><code>text_lengths</code> — (Batch,)</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def forward(
+ self,
+ speech: torch.Tensor,
+ speech_lengths: torch.Tensor,
+ text: torch.Tensor,
+ text_lengths: torch.Tensor,
+ **kwargs,
+ ) -&gt; Tuple[torch.Tensor, Dict[str, torch.Tensor], torch.Tensor]:
+ &quot;&quot;&quot;Encoder + Decoder + Calc loss
+ Args:
+ speech: (Batch, Length, ...)
+ speech_lengths: (Batch, )
+ text: (Batch, Length)
+ text_lengths: (Batch,)
+ &quot;&quot;&quot;
+ if len(text_lengths.size()) &gt; 1:
+ text_lengths = text_lengths[:, 0]
+ if len(speech_lengths.size()) &gt; 1:
+ speech_lengths = speech_lengths[:, 0]
+ batch_size = speech.shape[0]
+
+ # Encoder
+ encoder_out, encoder_out_lens = self.encode(speech, speech_lengths)
+ loss_ctc, cer_ctc = self._calc_ctc_loss(
+ encoder_out, encoder_out_lens, text, text_lengths
+ )
+
+ # Collect CTC branch stats
+ stats = dict()
+ stats[&quot;loss_ctc&quot;] = loss_ctc.detach() if loss_ctc is not None else None
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/fsmn_kws/model.py#L104" target="_blank" class="src-more">View full source →</a></details></div><div class="mblk"><code>.encode(speech, speech_lengths, **kwargs)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/fsmn_kws/model.py#L146" target="_blank" class="gh-link">L146</a><div class="mdoc"><p>Encoder. Note that this method is used by asr_inference.py</p>
+<h4>Args:</h4>
+<ul>
+<li><code>speech</code> — (Batch, Length, ...)</li>
+<li><code>speech_lengths</code> — (Batch, )</li>
+<li><code>ind</code> — int</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def encode(
+ self, speech: torch.Tensor, speech_lengths: torch.Tensor, **kwargs,
+ ) -&gt; Tuple[torch.Tensor, torch.Tensor]:
+ &quot;&quot;&quot;Encoder. Note that this method is used by asr_inference.py
+ Args:
+ speech: (Batch, Length, ...)
+ speech_lengths: (Batch, )
+ ind: int
+ &quot;&quot;&quot;
+ with autocast(False):
+ # Data augmentation
+ if self.specaug is not None and self.training:
+ speech, speech_lengths = self.specaug(speech, speech_lengths)
+
+ # Normalization for feature: e.g. Global-CMVN, Utterance-CMVN
+ if self.normalize is not None:
+ speech, speech_lengths = self.normalize(speech, speech_lengths)
+
+ # Forward encoder
+ encoder_out = self.encoder(speech)
+ encoder_out_lens = speech_lengths
+
+ if isinstance(encoder_out, tuple):
+ encoder_out = encoder_out[0]
+
+ return encoder_out, encoder_out_lens
+</code></pre></details></div><div class="mblk"><code>.inference(data_in, data_lengths, key, tokenizer, frontend, **kwargs)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/fsmn_kws/model.py#L199" target="_blank" class="gh-link">L199</a><div class="mdoc"><p>Run inference on input data.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>data_in</code> — Input data (audio samples, file paths, or text).</li>
+<li><code>data_lengths</code> — Lengths of each input sample in the batch.</li>
+<li><code>key</code> — Sample identifiers.</li>
+<li><code>tokenizer</code> — Tokenizer instance for text encoding/decoding.</li>
+<li><code>frontend</code> — Audio frontend for feature extraction.</li>
+<li><code>**kwargs</code> — Additional keyword arguments.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def inference(
+ self,
+ data_in,
+ data_lengths=None,
+ key: list=None,
+ tokenizer=None,
+ frontend=None,
+ **kwargs,
+ ):
+ &quot;&quot;&quot;Run inference on input data.
+
+ Args:
+ data_in: Input data (audio samples, file paths, or text).
+ data_lengths: Lengths of each input sample in the batch.
+ key: Sample identifiers.
+ tokenizer: Tokenizer instance for text encoding/decoding.
+ frontend: Audio frontend for feature extraction.
+ **kwargs: Additional keyword arguments.
+ &quot;&quot;&quot;
+ keywords = kwargs.get(&quot;keywords&quot;)
+ from funasr.utils.kws_utils import KwsCtcPrefixDecoder
+ self.kws_decoder = KwsCtcPrefixDecoder(
+ ctc=self.ctc,
+ keywords=keywords,
+ token_list=tokenizer.token_list,
+ seg_dict=tokenizer.seg_dict,
+ )
+
+ meta_data = {}
+ if isinstance(data_in, torch.Tensor) and kwargs.get(&quot;data_type&quot;, &quot;sound&quot;) == &quot;fbank&quot;: # fbank
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/fsmn_kws/model.py#L199" target="_blank" class="src-more">View full source →</a></details></div></div>
+<div class="api-detail" id="e70"><span class="dtag badge-m">method</span><h2><code>FsmnKWS.forward(speech, speech_lengths, text, text_lengths, **kwargs)</code></h2><div class="dmod">funasr.models.fsmn_kws.FsmnKWS · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/fsmn_kws/model.py#L104" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Encoder + Decoder + Calc loss</p>
+<h4>Args:</h4>
+<ul>
+<li><code>speech</code> — (Batch, Length, ...)</li>
+<li><code>speech_lengths</code> — (Batch, )</li>
+<li><code>text</code> — (Batch, Length)</li>
+<li><code>text_lengths</code> — (Batch,)</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def forward(
+ self,
+ speech: torch.Tensor,
+ speech_lengths: torch.Tensor,
+ text: torch.Tensor,
+ text_lengths: torch.Tensor,
+ **kwargs,
+ ) -&gt; Tuple[torch.Tensor, Dict[str, torch.Tensor], torch.Tensor]:
+ &quot;&quot;&quot;Encoder + Decoder + Calc loss
+ Args:
+ speech: (Batch, Length, ...)
+ speech_lengths: (Batch, )
+ text: (Batch, Length)
+ text_lengths: (Batch,)
+ &quot;&quot;&quot;
+ if len(text_lengths.size()) &gt; 1:
+ text_lengths = text_lengths[:, 0]
+ if len(speech_lengths.size()) &gt; 1:
+ speech_lengths = speech_lengths[:, 0]
+ batch_size = speech.shape[0]
+
+ # Encoder
+ encoder_out, encoder_out_lens = self.encode(speech, speech_lengths)
+ loss_ctc, cer_ctc = self._calc_ctc_loss(
+ encoder_out, encoder_out_lens, text, text_lengths
+ )
+
+ # Collect CTC branch stats
+ stats = dict()
+ stats[&quot;loss_ctc&quot;] = loss_ctc.detach() if loss_ctc is not None else None
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/fsmn_kws/model.py#L104" target="_blank" class="src-more">View full source on GitHub →</a></details></div>
+<div class="api-detail" id="e71"><span class="dtag badge-m">method</span><h2><code>FsmnKWS.encode(speech, speech_lengths, **kwargs)</code></h2><div class="dmod">funasr.models.fsmn_kws.FsmnKWS · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/fsmn_kws/model.py#L146" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Encoder. Note that this method is used by asr_inference.py</p>
+<h4>Args:</h4>
+<ul>
+<li><code>speech</code> — (Batch, Length, ...)</li>
+<li><code>speech_lengths</code> — (Batch, )</li>
+<li><code>ind</code> — int</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def encode(
+ self, speech: torch.Tensor, speech_lengths: torch.Tensor, **kwargs,
+ ) -&gt; Tuple[torch.Tensor, torch.Tensor]:
+ &quot;&quot;&quot;Encoder. Note that this method is used by asr_inference.py
+ Args:
+ speech: (Batch, Length, ...)
+ speech_lengths: (Batch, )
+ ind: int
+ &quot;&quot;&quot;
+ with autocast(False):
+ # Data augmentation
+ if self.specaug is not None and self.training:
+ speech, speech_lengths = self.specaug(speech, speech_lengths)
+
+ # Normalization for feature: e.g. Global-CMVN, Utterance-CMVN
+ if self.normalize is not None:
+ speech, speech_lengths = self.normalize(speech, speech_lengths)
+
+ # Forward encoder
+ encoder_out = self.encoder(speech)
+ encoder_out_lens = speech_lengths
+
+ if isinstance(encoder_out, tuple):
+ encoder_out = encoder_out[0]
+
+ return encoder_out, encoder_out_lens
+</code></pre></details></div>
+<div class="api-detail" id="e72"><span class="dtag badge-m">method</span><h2><code>FsmnKWS.inference(data_in, data_lengths, key, tokenizer, frontend, **kwargs)</code></h2><div class="dmod">funasr.models.fsmn_kws.FsmnKWS · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/fsmn_kws/model.py#L199" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Run inference on input data.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>data_in</code> — Input data (audio samples, file paths, or text).</li>
+<li><code>data_lengths</code> — Lengths of each input sample in the batch.</li>
+<li><code>key</code> — Sample identifiers.</li>
+<li><code>tokenizer</code> — Tokenizer instance for text encoding/decoding.</li>
+<li><code>frontend</code> — Audio frontend for feature extraction.</li>
+<li><code>**kwargs</code> — Additional keyword arguments.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def inference(
+ self,
+ data_in,
+ data_lengths=None,
+ key: list=None,
+ tokenizer=None,
+ frontend=None,
+ **kwargs,
+ ):
+ &quot;&quot;&quot;Run inference on input data.
+
+ Args:
+ data_in: Input data (audio samples, file paths, or text).
+ data_lengths: Lengths of each input sample in the batch.
+ key: Sample identifiers.
+ tokenizer: Tokenizer instance for text encoding/decoding.
+ frontend: Audio frontend for feature extraction.
+ **kwargs: Additional keyword arguments.
+ &quot;&quot;&quot;
+ keywords = kwargs.get(&quot;keywords&quot;)
+ from funasr.utils.kws_utils import KwsCtcPrefixDecoder
+ self.kws_decoder = KwsCtcPrefixDecoder(
+ ctc=self.ctc,
+ keywords=keywords,
+ token_list=tokenizer.token_list,
+ seg_dict=tokenizer.seg_dict,
+ )
+
+ meta_data = {}
+ if isinstance(data_in, torch.Tensor) and kwargs.get(&quot;data_type&quot;, &quot;sound&quot;) == &quot;fbank&quot;: # fbank
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/fsmn_kws/model.py#L199" target="_blank" class="src-more">View full source on GitHub →</a></details></div>
+<div class="api-detail" id="e73"><span class="dtag badge-c">class</span><h2>FsmnKWSConvert</h2><div class="dmod">funasr.models.fsmn_kws · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/fsmn_kws/model.py#L279" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><ul>
+<li><code>Author</code> — Speech Lab of DAMO Academy, Alibaba Group</li>
+<li><code>Deep</code> — FSMN for Large Vocabulary Continuous Speech Recognition</li>
+</ul>
+<p>https://arxiv.org/abs/1803.05030</p>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code>class FsmnKWSConvert(torch.nn.Module):
+ &quot;&quot;&quot;
+ Author: Speech Lab of DAMO Academy, Alibaba Group
+ Deep-FSMN for Large Vocabulary Continuous Speech Recognition
+ https://arxiv.org/abs/1803.05030
+ &quot;&quot;&quot;
+
+ def __init__(
+ self,
+ encoder: str = None,
+ encoder_conf: Optional[Dict] = None,
+ ctc: str = None,
+ ctc_conf: Optional[Dict] = None,
+ ctc_weight: float = 1.0,
+ input_size: int = 360,
+ vocab_size: int = -1,
+ blank_id: int = 0,
+ **kwargs,
+ ):
+ &quot;&quot;&quot;Initialize FsmnKWSConvert.
+
+ Args:
+ encoder: TODO.
+ encoder_conf: Configuration dict for encoder.
+ ctc: TODO.
+ ctc_conf: Configuration dict for ctc.
+ ctc_weight: TODO.
+ input_size: Size/dimension parameter.
+ vocab_size: Size/dimension parameter.
+ blank_id: TODO.
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/fsmn_kws/model.py#L279" target="_blank" class="src-more">View full source on GitHub →</a></details><h3 class="mt">Methods</h3><div class="mblk"><code>.to_kaldi_net()</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/fsmn_kws/model.py#L331" target="_blank" class="gh-link">L331</a><div class="mdoc"><p>To kaldi net.</p>
+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def to_kaldi_net(self):
+ &quot;&quot;&quot;To kaldi net.&quot;&quot;&quot;
+ return self.encoder.to_kaldi_net()
+</code></pre></details></div><div class="mblk"><code>.to_pytorch_net(kaldi_file)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/fsmn_kws/model.py#L336" target="_blank" class="gh-link">L336</a><div class="mdoc"><p>To pytorch net.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>kaldi_file</code> — TODO.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def to_pytorch_net(self, kaldi_file):
+ &quot;&quot;&quot;To pytorch net.
+
+ Args:
+ kaldi_file: TODO.
+ &quot;&quot;&quot;
+ return self.encoder.to_pytorch_net(kaldi_file)
+</code></pre></details></div></div>
+<div class="api-detail" id="e74"><span class="dtag badge-m">method</span><h2><code>FsmnKWSConvert.to_kaldi_net()</code></h2><div class="dmod">funasr.models.fsmn_kws.FsmnKWSConvert · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/fsmn_kws/model.py#L331" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>To kaldi net.</p>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def to_kaldi_net(self):
+ &quot;&quot;&quot;To kaldi net.&quot;&quot;&quot;
+ return self.encoder.to_kaldi_net()
+</code></pre></details></div>
+<div class="api-detail" id="e75"><span class="dtag badge-m">method</span><h2><code>FsmnKWSConvert.to_pytorch_net(kaldi_file)</code></h2><div class="dmod">funasr.models.fsmn_kws.FsmnKWSConvert · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/fsmn_kws/model.py#L336" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>To pytorch net.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>kaldi_file</code> — TODO.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def to_pytorch_net(self, kaldi_file):
+ &quot;&quot;&quot;To pytorch net.
+
+ Args:
+ kaldi_file: TODO.
+ &quot;&quot;&quot;
+ return self.encoder.to_pytorch_net(kaldi_file)
+</code></pre></details></div>
+<div class="api-detail" id="e76"><span class="dtag badge-c">class</span><h2>FsmnKWSMT</h2><div class="dmod">funasr.models.fsmn_kws_mt · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/fsmn_kws_mt/model.py#L27" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><ul>
+<li><code>FSMN</code> — KWS-MT: Multi-Task FSMN Keyword Spotting.</li>
+</ul>
+<p>Keyword spotting with multi-task learning: simultaneously</p>
+<p>detects keywords and performs filler token classification.</p>
+<p>Improves keyword detection robustness through auxiliary tasks.</p>
+<ul>
+<li><code>Output</code> — {&quot;key&quot;: str, &quot;value&quot;: keyword_detection_result}</li>
+<li><code>Author</code> — Speech Lab of DAMO Academy, Alibaba Group</li>
+<li><code>Deep</code> — FSMN for Large Vocabulary Continuous Speech Recognition</li>
+</ul>
+<p>https://arxiv.org/abs/1803.05030</p>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code>class FsmnKWSMT(torch.nn.Module):
+ &quot;&quot;&quot;FSMN-KWS-MT: Multi-Task FSMN Keyword Spotting.
+
+ Keyword spotting with multi-task learning: simultaneously
+ detects keywords and performs filler token classification.
+ Improves keyword detection robustness through auxiliary tasks.
+
+ Output: {&quot;key&quot;: str, &quot;value&quot;: keyword_detection_result}
+
+ Author: Speech Lab of DAMO Academy, Alibaba Group
+ Deep-FSMN for Large Vocabulary Continuous Speech Recognition
+ https://arxiv.org/abs/1803.05030
+ &quot;&quot;&quot;
+
+ def __init__(
+ self,
+ specaug: Optional[str] = None,
+ specaug_conf: Optional[Dict] = None,
+ normalize: str = None,
+ normalize_conf: Optional[Dict] = None,
+ encoder: str = None,
+ encoder_conf: Optional[Dict] = None,
+ ctc_conf: Optional[Dict] = None,
+ input_size: int = 360,
+ vocab_size: list = [],
+ ignore_id: int = -1,
+ blank_id: int = 0,
+ **kwargs,
+ ):
+ &quot;&quot;&quot;Initialize FsmnKWSMT.
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/fsmn_kws_mt/model.py#L27" target="_blank" class="src-more">View full source on GitHub →</a></details><h3 class="mt">Methods</h3><div class="mblk"><code>.forward(speech, speech_lengths, text, text_lengths, text2, text2_lengths, **kwargs)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/fsmn_kws_mt/model.py#L106" target="_blank" class="gh-link">L106</a><div class="mdoc"><p>Encoder + Decoder + Calc loss</p>
+<h4>Args:</h4>
+<ul>
+<li><code>speech</code> — (Batch, Length, ...)</li>
+<li><code>speech_lengths</code> — (Batch, )</li>
+<li><code>text</code> — (Batch, Length)</li>
+<li><code>text_lengths</code> — (Batch,)</li>
+<li><code>text2</code> — (Batch, Length)</li>
+<li><code>text2_lengths</code> — (Batch,)</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def forward(
+ self,
+ speech: torch.Tensor,
+ speech_lengths: torch.Tensor,
+ text: torch.Tensor,
+ text_lengths: torch.Tensor,
+ text2: torch.Tensor,
+ text2_lengths: torch.Tensor,
+ **kwargs,
+ ) -&gt; Tuple[torch.Tensor, Dict[str, torch.Tensor], torch.Tensor]:
+ &quot;&quot;&quot;Encoder + Decoder + Calc loss
+ Args:
+ speech: (Batch, Length, ...)
+ speech_lengths: (Batch, )
+ text: (Batch, Length)
+ text_lengths: (Batch,)
+ text2: (Batch, Length)
+ text2_lengths: (Batch,)
+ &quot;&quot;&quot;
+ if len(text_lengths.size()) &gt; 1:
+ text_lengths = text_lengths[:, 0]
+ if len(speech_lengths.size()) &gt; 1:
+ speech_lengths = speech_lengths[:, 0]
+ batch_size = speech.shape[0]
+
+ # Encoder
+ encoder_out, encoder_out2, encoder_out_lens = self.encode(speech, speech_lengths)
+
+ loss_ctc, cer_ctc = self._calc_ctc_loss(
+ encoder_out, encoder_out_lens, text, text_lengths
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/fsmn_kws_mt/model.py#L106" target="_blank" class="src-more">View full source →</a></details></div><div class="mblk"><code>.encode(speech, speech_lengths, **kwargs)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/fsmn_kws_mt/model.py#L158" target="_blank" class="gh-link">L158</a><div class="mdoc"><p>Encoder. Note that this method is used by asr_inference.py</p>
+<h4>Args:</h4>
+<ul>
+<li><code>speech</code> — (Batch, Length, ...)</li>
+<li><code>speech_lengths</code> — (Batch, )</li>
+<li><code>ind</code> — int</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def encode(
+ self,
+ speech: torch.Tensor,
+ speech_lengths: torch.Tensor,
+ **kwargs,
+ ) -&gt; Tuple[torch.Tensor, torch.Tensor]:
+ &quot;&quot;&quot;Encoder. Note that this method is used by asr_inference.py
+ Args:
+ speech: (Batch, Length, ...)
+ speech_lengths: (Batch, )
+ ind: int
+ &quot;&quot;&quot;
+ with autocast(False):
+ # Data augmentation
+ if self.specaug is not None and self.training:
+ speech, speech_lengths = self.specaug(speech, speech_lengths)
+
+ # Normalization for feature: e.g. Global-CMVN, Utterance-CMVN
+ if self.normalize is not None:
+ speech, speech_lengths = self.normalize(speech, speech_lengths)
+
+ # Forward encoder
+ encoder_out, encoder_out2 = self.encoder(speech)
+ encoder_out_lens = speech_lengths
+
+ if isinstance(encoder_out, tuple):
+ encoder_out = encoder_out[0]
+
+ if isinstance(encoder_out2, tuple):
+ encoder_out2 = encoder_out2[0]
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/fsmn_kws_mt/model.py#L158" target="_blank" class="src-more">View full source →</a></details></div><div class="mblk"><code>.inference(data_in, data_lengths, key, tokenizer, frontend, **kwargs)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/fsmn_kws_mt/model.py#L242" target="_blank" class="gh-link">L242</a><div class="mdoc"><p>Run inference on input data.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>data_in</code> — Input data (audio samples, file paths, or text).</li>
+<li><code>data_lengths</code> — Lengths of each input sample in the batch.</li>
+<li><code>key</code> — Sample identifiers.</li>
+<li><code>tokenizer</code> — Tokenizer instance for text encoding/decoding.</li>
+<li><code>frontend</code> — Audio frontend for feature extraction.</li>
+<li><code>**kwargs</code> — Additional keyword arguments.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def inference(
+ self,
+ data_in,
+ data_lengths=None,
+ key: list=None,
+ tokenizer=None,
+ frontend=None,
+ **kwargs,
+ ):
+ &quot;&quot;&quot;Run inference on input data.
+
+ Args:
+ data_in: Input data (audio samples, file paths, or text).
+ data_lengths: Lengths of each input sample in the batch.
+ key: Sample identifiers.
+ tokenizer: Tokenizer instance for text encoding/decoding.
+ frontend: Audio frontend for feature extraction.
+ **kwargs: Additional keyword arguments.
+ &quot;&quot;&quot;
+ keywords = kwargs.get(&quot;keywords&quot;)
+ from funasr.utils.kws_utils import KwsCtcPrefixDecoder
+ self.kws_decoder = KwsCtcPrefixDecoder(
+ ctc=self.ctc,
+ keywords=keywords,
+ token_list=tokenizer[0].token_list,
+ seg_dict=tokenizer[0].seg_dict,
+ )
+ self.kws_decoder2 = KwsCtcPrefixDecoder(
+ ctc=self.ctc2,
+ keywords=keywords,
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/fsmn_kws_mt/model.py#L242" target="_blank" class="src-more">View full source →</a></details></div></div>
+<div class="api-detail" id="e77"><span class="dtag badge-m">method</span><h2><code>FsmnKWSMT.forward(speech, speech_lengths, text, text_lengths, text2, text2_lengths, **kwargs)</code></h2><div class="dmod">funasr.models.fsmn_kws_mt.FsmnKWSMT · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/fsmn_kws_mt/model.py#L106" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Encoder + Decoder + Calc loss</p>
+<h4>Args:</h4>
+<ul>
+<li><code>speech</code> — (Batch, Length, ...)</li>
+<li><code>speech_lengths</code> — (Batch, )</li>
+<li><code>text</code> — (Batch, Length)</li>
+<li><code>text_lengths</code> — (Batch,)</li>
+<li><code>text2</code> — (Batch, Length)</li>
+<li><code>text2_lengths</code> — (Batch,)</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def forward(
+ self,
+ speech: torch.Tensor,
+ speech_lengths: torch.Tensor,
+ text: torch.Tensor,
+ text_lengths: torch.Tensor,
+ text2: torch.Tensor,
+ text2_lengths: torch.Tensor,
+ **kwargs,
+ ) -&gt; Tuple[torch.Tensor, Dict[str, torch.Tensor], torch.Tensor]:
+ &quot;&quot;&quot;Encoder + Decoder + Calc loss
+ Args:
+ speech: (Batch, Length, ...)
+ speech_lengths: (Batch, )
+ text: (Batch, Length)
+ text_lengths: (Batch,)
+ text2: (Batch, Length)
+ text2_lengths: (Batch,)
+ &quot;&quot;&quot;
+ if len(text_lengths.size()) &gt; 1:
+ text_lengths = text_lengths[:, 0]
+ if len(speech_lengths.size()) &gt; 1:
+ speech_lengths = speech_lengths[:, 0]
+ batch_size = speech.shape[0]
+
+ # Encoder
+ encoder_out, encoder_out2, encoder_out_lens = self.encode(speech, speech_lengths)
+
+ loss_ctc, cer_ctc = self._calc_ctc_loss(
+ encoder_out, encoder_out_lens, text, text_lengths
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/fsmn_kws_mt/model.py#L106" target="_blank" class="src-more">View full source on GitHub →</a></details></div>
+<div class="api-detail" id="e78"><span class="dtag badge-m">method</span><h2><code>FsmnKWSMT.encode(speech, speech_lengths, **kwargs)</code></h2><div class="dmod">funasr.models.fsmn_kws_mt.FsmnKWSMT · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/fsmn_kws_mt/model.py#L158" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Encoder. Note that this method is used by asr_inference.py</p>
+<h4>Args:</h4>
+<ul>
+<li><code>speech</code> — (Batch, Length, ...)</li>
+<li><code>speech_lengths</code> — (Batch, )</li>
+<li><code>ind</code> — int</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def encode(
+ self,
+ speech: torch.Tensor,
+ speech_lengths: torch.Tensor,
+ **kwargs,
+ ) -&gt; Tuple[torch.Tensor, torch.Tensor]:
+ &quot;&quot;&quot;Encoder. Note that this method is used by asr_inference.py
+ Args:
+ speech: (Batch, Length, ...)
+ speech_lengths: (Batch, )
+ ind: int
+ &quot;&quot;&quot;
+ with autocast(False):
+ # Data augmentation
+ if self.specaug is not None and self.training:
+ speech, speech_lengths = self.specaug(speech, speech_lengths)
+
+ # Normalization for feature: e.g. Global-CMVN, Utterance-CMVN
+ if self.normalize is not None:
+ speech, speech_lengths = self.normalize(speech, speech_lengths)
+
+ # Forward encoder
+ encoder_out, encoder_out2 = self.encoder(speech)
+ encoder_out_lens = speech_lengths
+
+ if isinstance(encoder_out, tuple):
+ encoder_out = encoder_out[0]
+
+ if isinstance(encoder_out2, tuple):
+ encoder_out2 = encoder_out2[0]
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/fsmn_kws_mt/model.py#L158" target="_blank" class="src-more">View full source on GitHub →</a></details></div>
+<div class="api-detail" id="e79"><span class="dtag badge-m">method</span><h2><code>FsmnKWSMT.inference(data_in, data_lengths, key, tokenizer, frontend, **kwargs)</code></h2><div class="dmod">funasr.models.fsmn_kws_mt.FsmnKWSMT · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/fsmn_kws_mt/model.py#L242" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Run inference on input data.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>data_in</code> — Input data (audio samples, file paths, or text).</li>
+<li><code>data_lengths</code> — Lengths of each input sample in the batch.</li>
+<li><code>key</code> — Sample identifiers.</li>
+<li><code>tokenizer</code> — Tokenizer instance for text encoding/decoding.</li>
+<li><code>frontend</code> — Audio frontend for feature extraction.</li>
+<li><code>**kwargs</code> — Additional keyword arguments.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def inference(
+ self,
+ data_in,
+ data_lengths=None,
+ key: list=None,
+ tokenizer=None,
+ frontend=None,
+ **kwargs,
+ ):
+ &quot;&quot;&quot;Run inference on input data.
+
+ Args:
+ data_in: Input data (audio samples, file paths, or text).
+ data_lengths: Lengths of each input sample in the batch.
+ key: Sample identifiers.
+ tokenizer: Tokenizer instance for text encoding/decoding.
+ frontend: Audio frontend for feature extraction.
+ **kwargs: Additional keyword arguments.
+ &quot;&quot;&quot;
+ keywords = kwargs.get(&quot;keywords&quot;)
+ from funasr.utils.kws_utils import KwsCtcPrefixDecoder
+ self.kws_decoder = KwsCtcPrefixDecoder(
+ ctc=self.ctc,
+ keywords=keywords,
+ token_list=tokenizer[0].token_list,
+ seg_dict=tokenizer[0].seg_dict,
+ )
+ self.kws_decoder2 = KwsCtcPrefixDecoder(
+ ctc=self.ctc2,
+ keywords=keywords,
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/fsmn_kws_mt/model.py#L242" target="_blank" class="src-more">View full source on GitHub →</a></details></div>
+<div class="api-detail" id="e80"><span class="dtag badge-c">class</span><h2>FsmnKWSMTConvert</h2><div class="dmod">funasr.models.fsmn_kws_mt · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/fsmn_kws_mt/model.py#L352" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><ul>
+<li><code>Author</code> — Speech Lab of DAMO Academy, Alibaba Group</li>
+<li><code>Deep</code> — FSMN for Large Vocabulary Continuous Speech Recognition</li>
+</ul>
+<p>https://arxiv.org/abs/1803.05030</p>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code>class FsmnKWSMTConvert(torch.nn.Module):
+ &quot;&quot;&quot;
+ Author: Speech Lab of DAMO Academy, Alibaba Group
+ Deep-FSMN for Large Vocabulary Continuous Speech Recognition
+ https://arxiv.org/abs/1803.05030
+ &quot;&quot;&quot;
+
+ def __init__(
+ self,
+ encoder: str = None,
+ encoder_conf: Optional[Dict] = None,
+ ctc_conf: Optional[Dict] = None,
+ ctc_weight: float = 1.0,
+ input_size: int = 360,
+ blank_id: int = 0,
+ **kwargs,
+ ):
+ &quot;&quot;&quot;Initialize FsmnKWSMTConvert.
+
+ Args:
+ encoder: TODO.
+ encoder_conf: Configuration dict for encoder.
+ ctc_conf: Configuration dict for ctc.
+ ctc_weight: TODO.
+ input_size: Size/dimension parameter.
+ blank_id: TODO.
+ **kwargs: Additional keyword arguments.
+ &quot;&quot;&quot;
+ super().__init__()
+
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/fsmn_kws_mt/model.py#L352" target="_blank" class="src-more">View full source on GitHub →</a></details><h3 class="mt">Methods</h3><div class="mblk"><code>.to_kaldi_net()</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/fsmn_kws_mt/model.py#L390" target="_blank" class="gh-link">L390</a><div class="mdoc"><p>To kaldi net.</p>
+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def to_kaldi_net(self):
+ &quot;&quot;&quot;To kaldi net.&quot;&quot;&quot;
+ return self.encoder.to_kaldi_net()
+</code></pre></details></div><div class="mblk"><code>.to_kaldi_net2()</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/fsmn_kws_mt/model.py#L394" target="_blank" class="gh-link">L394</a><div class="mdoc"><p>To kaldi net2.</p>
+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def to_kaldi_net2(self):
+ &quot;&quot;&quot;To kaldi net2.&quot;&quot;&quot;
+ return self.encoder.to_kaldi_net2()
+</code></pre></details></div><div class="mblk"><code>.to_pytorch_net(kaldi_file)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/fsmn_kws_mt/model.py#L398" target="_blank" class="gh-link">L398</a><div class="mdoc"><p>To pytorch net.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>kaldi_file</code> — TODO.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def to_pytorch_net(self, kaldi_file):
+ &quot;&quot;&quot;To pytorch net.
+
+ Args:
+ kaldi_file: TODO.
+ &quot;&quot;&quot;
+ return self.encoder.to_pytorch_net(kaldi_file)
+</code></pre></details></div></div>
+<div class="api-detail" id="e81"><span class="dtag badge-m">method</span><h2><code>FsmnKWSMTConvert.to_kaldi_net()</code></h2><div class="dmod">funasr.models.fsmn_kws_mt.FsmnKWSMTConvert · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/fsmn_kws_mt/model.py#L390" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>To kaldi net.</p>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def to_kaldi_net(self):
+ &quot;&quot;&quot;To kaldi net.&quot;&quot;&quot;
+ return self.encoder.to_kaldi_net()
+</code></pre></details></div>
+<div class="api-detail" id="e82"><span class="dtag badge-m">method</span><h2><code>FsmnKWSMTConvert.to_kaldi_net2()</code></h2><div class="dmod">funasr.models.fsmn_kws_mt.FsmnKWSMTConvert · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/fsmn_kws_mt/model.py#L394" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>To kaldi net2.</p>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def to_kaldi_net2(self):
+ &quot;&quot;&quot;To kaldi net2.&quot;&quot;&quot;
+ return self.encoder.to_kaldi_net2()
+</code></pre></details></div>
+<div class="api-detail" id="e83"><span class="dtag badge-m">method</span><h2><code>FsmnKWSMTConvert.to_pytorch_net(kaldi_file)</code></h2><div class="dmod">funasr.models.fsmn_kws_mt.FsmnKWSMTConvert · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/fsmn_kws_mt/model.py#L398" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>To pytorch net.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>kaldi_file</code> — TODO.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def to_pytorch_net(self, kaldi_file):
+ &quot;&quot;&quot;To pytorch net.
+
+ Args:
+ kaldi_file: TODO.
+ &quot;&quot;&quot;
+ return self.encoder.to_pytorch_net(kaldi_file)
+</code></pre></details></div>
+<div class="api-detail" id="e84"><span class="dtag badge-c">class</span><h2>FsmnVADStreaming</h2><div class="dmod">funasr.models.fsmn_vad_streaming · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/fsmn_vad_streaming/model.py#L345" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><ul>
+<li><code>FSMN</code> — based Voice Activity Detection (streaming/offline).</li>
+</ul>
+<p>Detects speech segments in audio, returning start/end timestamps (milliseconds).</p>
+<p>Supports both offline (full audio) and streaming (chunk-by-chunk) modes.</p>
+<p>Offline output: [{&quot;key&quot;: &quot;...&quot;, &quot;value&quot;: [[start_ms, end_ms], ...]}]</p>
+<p>Streaming output: [[beg, -1]] (start), [[-1, end]] (end), [[beg, end]] (complete), [] (no event)</p>
+<ul>
+<li><code>Author</code> — Speech Lab of DAMO Academy, Alibaba Group</li>
+<li><code>Deep</code> — FSMN for Large Vocabulary Continuous Speech Recognition</li>
+</ul>
+<p>https://arxiv.org/abs/1803.05030</p>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code>class FsmnVADStreaming(nn.Module):
+ &quot;&quot;&quot;FSMN-based Voice Activity Detection (streaming/offline).
+
+ Detects speech segments in audio, returning start/end timestamps (milliseconds).
+ Supports both offline (full audio) and streaming (chunk-by-chunk) modes.
+
+ Offline output: [{&quot;key&quot;: &quot;...&quot;, &quot;value&quot;: [[start_ms, end_ms], ...]}]
+ Streaming output: [[beg, -1]] (start), [[-1, end]] (end), [[beg, end]] (complete), [] (no event)
+
+ Author: Speech Lab of DAMO Academy, Alibaba Group
+ Deep-FSMN for Large Vocabulary Continuous Speech Recognition
+ https://arxiv.org/abs/1803.05030
+ &quot;&quot;&quot;
+
+ def __init__(
+ self,
+ encoder: str = None,
+ encoder_conf: Optional[Dict] = None,
+ vad_post_args: Dict[str, Any] = None,
+ **kwargs,
+ ):
+ &quot;&quot;&quot;Initialize FsmnVADStreaming.
+
+ Args:
+ encoder: TODO.
+ encoder_conf: Configuration dict for encoder.
+ vad_post_args: TODO.
+ **kwargs: Additional keyword arguments.
+ &quot;&quot;&quot;
+ super().__init__()
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/fsmn_vad_streaming/model.py#L345" target="_blank" class="src-more">View full source on GitHub →</a></details><h3 class="mt">Methods</h3><div class="mblk"><code>.ResetDetection(cache)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/fsmn_vad_streaming/model.py#L382" target="_blank" class="gh-link">L382</a><div class="mdoc"><p>Resetdetection.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>cache</code> — State cache dict for streaming inference.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def ResetDetection(self, cache: dict = None):
+ &quot;&quot;&quot;Resetdetection.
+
+ Args:
+ cache: State cache dict for streaming inference.
+ &quot;&quot;&quot;
+ if cache is None:
+ cache = {}
+ cache[&quot;stats&quot;].continous_silence_frame_count = 0
+ cache[&quot;stats&quot;].latest_confirmed_speech_frame = 0
+ cache[&quot;stats&quot;].lastest_confirmed_silence_frame = -1
+ cache[&quot;stats&quot;].confirmed_start_frame = -1
+ cache[&quot;stats&quot;].confirmed_end_frame = -1
+ cache[&quot;stats&quot;].vad_state_machine = VadStateMachine.kVadInStateStartPointNotDetected
+ cache[&quot;windows_detector&quot;].Reset()
+ cache[&quot;stats&quot;].sil_frame = 0
+ cache[&quot;stats&quot;].frame_probs = []
+
+ if cache[&quot;stats&quot;].output_data_buf:
+ assert cache[&quot;stats&quot;].output_data_buf[-1].contain_seg_end_point == True
+ drop_frames = int(cache[&quot;stats&quot;].output_data_buf[-1].end_ms / self.vad_opts.frame_in_ms)
+ real_drop_frames = drop_frames - cache[&quot;stats&quot;].last_drop_frames
+ cache[&quot;stats&quot;].last_drop_frames = drop_frames
+ cache[&quot;stats&quot;].data_buf_all = cache[&quot;stats&quot;].data_buf_all[
+ real_drop_frames
+ * int(self.vad_opts.frame_in_ms * self.vad_opts.sample_rate / 1000) :
+ ]
+ cache[&quot;stats&quot;].decibel = cache[&quot;stats&quot;].decibel[real_drop_frames:]
+ cache[&quot;stats&quot;].scores = cache[&quot;stats&quot;].scores[:, real_drop_frames:, :]
+</code></pre></details></div><div class="mblk"><code>.ComputeDecibel(cache)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/fsmn_vad_streaming/model.py#L412" target="_blank" class="gh-link">L412</a><div class="mdoc"><p>Computedecibel.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>cache</code> — State cache dict for streaming inference.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def ComputeDecibel(self, cache: dict = None) -&gt; None:
+ &quot;&quot;&quot;Computedecibel.
+
+ Args:
+ cache: State cache dict for streaming inference.
+ &quot;&quot;&quot;
+ if cache is None:
+ cache = {}
+ frame_sample_length = int(self.vad_opts.frame_length_ms * self.vad_opts.sample_rate / 1000)
+ frame_shift_length = int(self.vad_opts.frame_in_ms * self.vad_opts.sample_rate / 1000)
+ if cache[&quot;stats&quot;].data_buf_all is None:
+ cache[&quot;stats&quot;].data_buf_all = cache[&quot;stats&quot;].waveform[
+ 0
+ ] # cache[&quot;stats&quot;].data_buf is pointed to cache[&quot;stats&quot;].waveform[0]
+ cache[&quot;stats&quot;].data_buf = cache[&quot;stats&quot;].data_buf_all
+ else:
+ cache[&quot;stats&quot;].data_buf_all = torch.cat(
+ (cache[&quot;stats&quot;].data_buf_all, cache[&quot;stats&quot;].waveform[0])
+ )
+
+ waveform_numpy = cache[&quot;stats&quot;].waveform.numpy()
+
+ offsets = np.arange(0, waveform_numpy.shape[1] - frame_sample_length + 1, frame_shift_length)
+ frames = waveform_numpy[0, offsets[:, np.newaxis] + np.arange(frame_sample_length)]
+
+ decibel_numpy = 10 * np.log10(np.sum(np.square(frames), axis=1) + 0.000001)
+ decibel_numpy = decibel_numpy.tolist()
+
+ cache[&quot;stats&quot;].decibel.extend(decibel_numpy)
+</code></pre></details></div><div class="mblk"><code>.ComputeScores(feats, cache)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/fsmn_vad_streaming/model.py#L443" target="_blank" class="gh-link">L443</a><div class="mdoc"><p>Computescores.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>feats</code> — Feature tensor (e.g., fbank), shape (batch, frames, dim).</li>
+<li><code>cache</code> — State cache dict for streaming inference.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def ComputeScores(self, feats: torch.Tensor, cache: dict = None) -&gt; None:
+ &quot;&quot;&quot;Computescores.
+
+ Args:
+ feats: Feature tensor (e.g., fbank), shape (batch, frames, dim).
+ cache: State cache dict for streaming inference.
+ &quot;&quot;&quot;
+ if cache is None:
+ cache = {}
+ scores = self.encoder(feats, cache=cache[&quot;encoder&quot;]) # return B * T * D
+ assert (
+ scores.shape[1] == feats.shape[1]
+ ), &quot;The shape between feats and scores does not match&quot;
+ self.vad_opts.nn_eval_block_size = scores.shape[1]
+ cache[&quot;stats&quot;].frm_cnt += scores.shape[1] # count total frames
+ if cache[&quot;stats&quot;].scores is None:
+ cache[&quot;stats&quot;].scores = scores # the first calculation
+ else:
+ cache[&quot;stats&quot;].scores = torch.cat((cache[&quot;stats&quot;].scores, scores), dim=1)
+</code></pre></details></div><div class="mblk"><code>.PopDataBufTillFrame(frame_idx, cache)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/fsmn_vad_streaming/model.py#L463" target="_blank" class="gh-link">L463</a><div class="mdoc"><p>Popdatabuftillframe.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>frame_idx</code> — TODO.</li>
+<li><code>cache</code> — State cache dict for streaming inference.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def PopDataBufTillFrame(self, frame_idx: int, cache: dict = None) -&gt; None: # need check again
+ &quot;&quot;&quot;Popdatabuftillframe.
+
+ Args:
+ frame_idx: TODO.
+ cache: State cache dict for streaming inference.
+ &quot;&quot;&quot;
+ while cache[&quot;stats&quot;].data_buf_start_frame &lt; frame_idx:
+ if len(cache[&quot;stats&quot;].data_buf) &gt;= int(
+ self.vad_opts.frame_in_ms * self.vad_opts.sample_rate / 1000
+ ):
+ if cache is None:
+ cache = {}
+ cache[&quot;stats&quot;].data_buf_start_frame += 1
+ cache[&quot;stats&quot;].data_buf = cache[&quot;stats&quot;].data_buf_all[
+ (cache[&quot;stats&quot;].data_buf_start_frame - cache[&quot;stats&quot;].last_drop_frames)
+ * int(self.vad_opts.frame_in_ms * self.vad_opts.sample_rate / 1000) :
+ ]
+</code></pre></details></div><div class="mblk"><code>.PopDataToOutputBuf(start_frm, frm_cnt, first_frm_is_start_point, last_frm_is_end_point, end_point_is_sent_end, cache)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/fsmn_vad_streaming/model.py#L482" target="_blank" class="gh-link">L482</a><div class="mdoc"><p>Popdatatooutputbuf.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>start_frm</code> — TODO.</li>
+<li><code>frm_cnt</code> — TODO.</li>
+<li><code>first_frm_is_start_point</code> — TODO.</li>
+<li><code>last_frm_is_end_point</code> — TODO.</li>
+<li><code>end_point_is_sent_end</code> — TODO.</li>
+<li><code>cache</code> — State cache dict for streaming inference.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def PopDataToOutputBuf(
+ self,
+ start_frm: int,
+ frm_cnt: int,
+ first_frm_is_start_point: bool,
+ last_frm_is_end_point: bool,
+ end_point_is_sent_end: bool,
+ cache: dict = None,
+ ) -&gt; None:
+ &quot;&quot;&quot;Popdatatooutputbuf.
+
+ Args:
+ start_frm: TODO.
+ frm_cnt: TODO.
+ first_frm_is_start_point: TODO.
+ last_frm_is_end_point: TODO.
+ end_point_is_sent_end: TODO.
+ cache: State cache dict for streaming inference.
+ &quot;&quot;&quot;
+ if cache is None:
+ cache = {}
+ self.PopDataBufTillFrame(start_frm, cache=cache)
+ expected_sample_number = int(
+ frm_cnt * self.vad_opts.sample_rate * self.vad_opts.frame_in_ms / 1000
+ )
+ if last_frm_is_end_point:
+ extra_sample = max(
+ 0,
+ int(
+ self.vad_opts.frame_length_ms * self.vad_opts.sample_rate / 1000
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/fsmn_vad_streaming/model.py#L482" target="_blank" class="src-more">View full source →</a></details></div><div class="mblk"><code>.OnSilenceDetected(valid_frame, cache)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/fsmn_vad_streaming/model.py#L552" target="_blank" class="gh-link">L552</a><div class="mdoc"><p>Onsilencedetected.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>valid_frame</code> — TODO.</li>
+<li><code>cache</code> — State cache dict for streaming inference.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def OnSilenceDetected(self, valid_frame: int, cache: dict = None):
+ &quot;&quot;&quot;Onsilencedetected.
+
+ Args:
+ valid_frame: TODO.
+ cache: State cache dict for streaming inference.
+ &quot;&quot;&quot;
+ if cache is None:
+ cache = {}
+ cache[&quot;stats&quot;].lastest_confirmed_silence_frame = valid_frame
+ if cache[&quot;stats&quot;].vad_state_machine == VadStateMachine.kVadInStateStartPointNotDetected:
+ self.PopDataBufTillFrame(valid_frame, cache=cache)
+</code></pre></details></div><div class="mblk"><code>.OnVoiceDetected(valid_frame, cache)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/fsmn_vad_streaming/model.py#L568" target="_blank" class="gh-link">L568</a><div class="mdoc"><p>Onvoicedetected.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>valid_frame</code> — TODO.</li>
+<li><code>cache</code> — State cache dict for streaming inference.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def OnVoiceDetected(self, valid_frame: int, cache: dict = None) -&gt; None:
+ &quot;&quot;&quot;Onvoicedetected.
+
+ Args:
+ valid_frame: TODO.
+ cache: State cache dict for streaming inference.
+ &quot;&quot;&quot;
+ if cache is None:
+ cache = {}
+ cache[&quot;stats&quot;].latest_confirmed_speech_frame = valid_frame
+ self.PopDataToOutputBuf(valid_frame, 1, False, False, False, cache=cache)
+</code></pre></details></div><div class="mblk"><code>.OnVoiceStart(start_frame, fake_result, cache)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/fsmn_vad_streaming/model.py#L580" target="_blank" class="gh-link">L580</a><div class="mdoc"><p>Onvoicestart.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>start_frame</code> — TODO.</li>
+<li><code>fake_result</code> — TODO.</li>
+<li><code>cache</code> — State cache dict for streaming inference.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def OnVoiceStart(self, start_frame: int, fake_result: bool = False, cache: dict = None) -&gt; None:
+ &quot;&quot;&quot;Onvoicestart.
+
+ Args:
+ start_frame: TODO.
+ fake_result: TODO.
+ cache: State cache dict for streaming inference.
+ &quot;&quot;&quot;
+ if cache is None:
+ cache = {}
+ if self.vad_opts.do_start_point_detection:
+ pass
+ if cache[&quot;stats&quot;].confirmed_start_frame != -1:
+ print(&quot;not reset vad properly\n&quot;)
+ else:
+ cache[&quot;stats&quot;].confirmed_start_frame = start_frame
+
+ if (
+ not fake_result
+ and cache[&quot;stats&quot;].vad_state_machine == VadStateMachine.kVadInStateStartPointNotDetected
+ ):
+ self.PopDataToOutputBuf(
+ cache[&quot;stats&quot;].confirmed_start_frame, 1, True, False, False, cache=cache
+ )
+</code></pre></details></div><div class="mblk"><code>.OnVoiceEnd(end_frame, fake_result, is_last_frame, cache)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/fsmn_vad_streaming/model.py#L605" target="_blank" class="gh-link">L605</a><div class="mdoc"><p>Onvoiceend.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>end_frame</code> — TODO.</li>
+<li><code>fake_result</code> — TODO.</li>
+<li><code>is_last_frame</code> — Boolean flag for last frame.</li>
+<li><code>cache</code> — State cache dict for streaming inference.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def OnVoiceEnd(
+ self, end_frame: int, fake_result: bool, is_last_frame: bool, cache: dict = None
+ ) -&gt; None:
+ &quot;&quot;&quot;Onvoiceend.
+
+ Args:
+ end_frame: TODO.
+ fake_result: TODO.
+ is_last_frame: Boolean flag for last frame.
+ cache: State cache dict for streaming inference.
+ &quot;&quot;&quot;
+ if cache is None:
+ cache = {}
+ for t in range(cache[&quot;stats&quot;].latest_confirmed_speech_frame + 1, end_frame):
+ self.OnVoiceDetected(t, cache=cache)
+ if self.vad_opts.do_end_point_detection:
+ pass
+ if cache[&quot;stats&quot;].confirmed_end_frame != -1:
+ print(&quot;not reset vad properly\n&quot;)
+ else:
+ cache[&quot;stats&quot;].confirmed_end_frame = end_frame
+ if not fake_result:
+ cache[&quot;stats&quot;].sil_frame = 0
+ self.PopDataToOutputBuf(
+ cache[&quot;stats&quot;].confirmed_end_frame, 1, False, True, is_last_frame, cache=cache
+ )
+ cache[&quot;stats&quot;].number_end_time_detected += 1
+</code></pre></details></div><div class="mblk"><code>.MaybeOnVoiceEndIfLastFrame(is_final_frame, cur_frm_idx, cache)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/fsmn_vad_streaming/model.py#L633" target="_blank" class="gh-link">L633</a><div class="mdoc"><p>Maybeonvoiceendiflastframe.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>is_final_frame</code> — Boolean flag for final frame.</li>
+<li><code>cur_frm_idx</code> — TODO.</li>
+<li><code>cache</code> — State cache dict for streaming inference.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def MaybeOnVoiceEndIfLastFrame(
+ self, is_final_frame: bool, cur_frm_idx: int, cache: dict = None
+ ) -&gt; None:
+ &quot;&quot;&quot;Maybeonvoiceendiflastframe.
+
+ Args:
+ is_final_frame: Boolean flag for final frame.
+ cur_frm_idx: TODO.
+ cache: State cache dict for streaming inference.
+ &quot;&quot;&quot;
+ if cache is None:
+ cache = {}
+ if is_final_frame:
+ self.OnVoiceEnd(cur_frm_idx, False, True, cache=cache)
+ cache[&quot;stats&quot;].vad_state_machine = VadStateMachine.kVadInStateEndPointDetected
+</code></pre></details></div><div class="mblk"><code>.GetLatency(cache)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/fsmn_vad_streaming/model.py#L649" target="_blank" class="gh-link">L649</a><div class="mdoc"><p>Getlatency.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>cache</code> — State cache dict for streaming inference.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def GetLatency(self, cache: dict = None) -&gt; int:
+ &quot;&quot;&quot;Getlatency.
+
+ Args:
+ cache: State cache dict for streaming inference.
+ &quot;&quot;&quot;
+ if cache is None:
+ cache = {}
+ return int(self.LatencyFrmNumAtStartPoint(cache=cache) * self.vad_opts.frame_in_ms)
+</code></pre></details></div><div class="mblk"><code>.LatencyFrmNumAtStartPoint(cache)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/fsmn_vad_streaming/model.py#L659" target="_blank" class="gh-link">L659</a><div class="mdoc"><p>Latencyfrmnumatstartpoint.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>cache</code> — State cache dict for streaming inference.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def LatencyFrmNumAtStartPoint(self, cache: dict = None) -&gt; int:
+ &quot;&quot;&quot;Latencyfrmnumatstartpoint.
+
+ Args:
+ cache: State cache dict for streaming inference.
+ &quot;&quot;&quot;
+ if cache is None:
+ cache = {}
+ vad_latency = cache[&quot;windows_detector&quot;].GetWinSize()
+ if self.vad_opts.do_extend:
+ vad_latency += int(self.vad_opts.lookback_time_start_point / self.vad_opts.frame_in_ms)
+ return vad_latency
+</code></pre></details></div><div class="mblk"><code>.GetFrameState(t, cache)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/fsmn_vad_streaming/model.py#L672" target="_blank" class="gh-link">L672</a><div class="mdoc"><p>Getframestate.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>t</code> — TODO.</li>
+<li><code>cache</code> — State cache dict for streaming inference.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def GetFrameState(self, t: int, cache: dict = None):
+ &quot;&quot;&quot;Getframestate.
+
+ Args:
+ t: TODO.
+ cache: State cache dict for streaming inference.
+ &quot;&quot;&quot;
+ if cache is None:
+ cache = {}
+ frame_state = FrameState.kFrameStateInvalid
+ if t &gt;= len(cache[&quot;stats&quot;].decibel):
+ return FrameState.kFrameStateSil
+ cur_decibel = cache[&quot;stats&quot;].decibel[t]
+ cur_snr = cur_decibel - cache[&quot;stats&quot;].noise_average_decibel
+ # for each frame, calc log posterior probability of each state
+ if cur_decibel &lt; self.vad_opts.decibel_thres:
+ frame_state = FrameState.kFrameStateSil
+ self.DetectOneFrame(frame_state, t, False, cache=cache)
+ return frame_state
+
+ sum_score = 0.0
+ noise_prob = 0.0
+ assert len(cache[&quot;stats&quot;].sil_pdf_ids) == self.vad_opts.silence_pdf_num
+ if len(cache[&quot;stats&quot;].sil_pdf_ids) &gt; 0:
+ assert len(cache[&quot;stats&quot;].scores) == 1 # 只支持batch_size = 1的测试
+ &quot;&quot;&quot;
+ - Change type of `sum_score` to float. The reason is that `sum_score` is a tensor with single element.
+ and `torch.Tensor` is slower `float` when tensor has only one element.
+ - Put the iteration of `sil_pdf_ids` inside `sum()` to reduce the overhead of creating a new list.
+ - The default `sil_pdf_ids` is [0], the `if` statement is used to reduce the overhead of expression
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/fsmn_vad_streaming/model.py#L672" target="_blank" class="src-more">View full source →</a></details></div><div class="mblk"><code>.forward(feats, waveform, cache, is_final, **kwargs)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/fsmn_vad_streaming/model.py#L737" target="_blank" class="gh-link">L737</a><div class="mdoc"><p>Forward pass for training.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>feats</code> — Feature tensor (e.g., fbank), shape (batch, frames, dim).</li>
+<li><code>waveform</code> — TODO.</li>
+<li><code>cache</code> — State cache dict for streaming inference.</li>
+<li><code>is_final</code> — Whether this is the final chunk in streaming.</li>
+<li><code>**kwargs</code> — Additional keyword arguments.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def forward(
+ self,
+ feats: torch.Tensor,
+ waveform: torch.tensor,
+ cache: dict = None,
+ is_final: bool = False,
+ **kwargs,
+ ):
+ &quot;&quot;&quot;Forward pass for training.
+
+ Args:
+ feats: Feature tensor (e.g., fbank), shape (batch, frames, dim).
+ waveform: TODO.
+ cache: State cache dict for streaming inference.
+ is_final: Whether this is the final chunk in streaming.
+ **kwargs: Additional keyword arguments.
+ &quot;&quot;&quot;
+ if cache is None:
+ cache = {}
+ # if len(cache) == 0:
+ # self.AllResetDetection()
+ # self.waveform = waveform # compute decibel for each frame
+ cache[&quot;stats&quot;].waveform = waveform
+ is_streaming_input = kwargs.get(&quot;is_streaming_input&quot;, True)
+ self.ComputeDecibel(cache=cache)
+ self.ComputeScores(feats, cache=cache)
+ if not is_final:
+ self.DetectCommonFrames(cache=cache)
+ else:
+ self.DetectLastFrames(cache=cache)
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/fsmn_vad_streaming/model.py#L737" target="_blank" class="src-more">View full source →</a></details></div><div class="mblk"><code>.init_cache(cache, **kwargs)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/fsmn_vad_streaming/model.py#L820" target="_blank" class="gh-link">L820</a><div class="mdoc"><p>Init cache.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>cache</code> — State cache dict for streaming inference.</li>
+<li><code>**kwargs</code> — Additional keyword arguments.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def init_cache(self, cache: dict = None, **kwargs):
+ &quot;&quot;&quot;Init cache.
+
+ Args:
+ cache: State cache dict for streaming inference.
+ **kwargs: Additional keyword arguments.
+ &quot;&quot;&quot;
+ if cache is None:
+ cache = {}
+
+ cache[&quot;frontend&quot;] = {}
+ cache[&quot;prev_samples&quot;] = torch.empty(0)
+ cache[&quot;encoder&quot;] = {}
+
+ if kwargs.get(&quot;max_end_silence_time&quot;) is not None:
+ # update the max_end_silence_time
+ self.vad_opts.max_end_silence_time = kwargs.get(&quot;max_end_silence_time&quot;)
+
+ windows_detector = WindowDetector(
+ self.vad_opts.window_size_ms,
+ self.vad_opts.sil_to_speech_time_thres,
+ self.vad_opts.speech_to_sil_time_thres,
+ self.vad_opts.frame_in_ms,
+ )
+ windows_detector.Reset()
+
+ stats = Stats(
+ sil_pdf_ids=self.vad_opts.sil_pdf_ids,
+ max_end_sil_frame_cnt_thresh=self.vad_opts.max_end_silence_time
+ - self.vad_opts.speech_to_sil_time_thres,
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/fsmn_vad_streaming/model.py#L820" target="_blank" class="src-more">View full source →</a></details></div><div class="mblk"><code>.inference(data_in, data_lengths, key, tokenizer, frontend, cache, **kwargs)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/fsmn_vad_streaming/model.py#L856" target="_blank" class="gh-link">L856</a><div class="mdoc"><p>Run inference on input data.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>data_in</code> — Input data (audio samples, file paths, or text).</li>
+<li><code>data_lengths</code> — Lengths of each input sample in the batch.</li>
+<li><code>key</code> — Sample identifiers.</li>
+<li><code>tokenizer</code> — Tokenizer instance for text encoding/decoding.</li>
+<li><code>frontend</code> — Audio frontend for feature extraction.</li>
+<li><code>cache</code> — State cache dict for streaming inference.</li>
+<li><code>**kwargs</code> — Additional keyword arguments.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def inference(
+ self,
+ data_in,
+ data_lengths=None,
+ key: list = None,
+ tokenizer=None,
+ frontend=None,
+ cache: dict = None,
+ **kwargs,
+ ):
+ &quot;&quot;&quot;Run inference on input data.
+
+ Args:
+ data_in: Input data (audio samples, file paths, or text).
+ data_lengths: Lengths of each input sample in the batch.
+ key: Sample identifiers.
+ tokenizer: Tokenizer instance for text encoding/decoding.
+ frontend: Audio frontend for feature extraction.
+ cache: State cache dict for streaming inference.
+ **kwargs: Additional keyword arguments.
+ &quot;&quot;&quot;
+ if cache is None:
+ cache = {}
+ if len(cache) == 0:
+ self.init_cache(cache, **kwargs)
+
+ meta_data = {}
+ chunk_size = kwargs.get(&quot;chunk_size&quot;, 60000) # 50ms
+ chunk_stride_samples = int(chunk_size * frontend.fs / 1000)
+
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/fsmn_vad_streaming/model.py#L856" target="_blank" class="src-more">View full source →</a></details></div><div class="mblk"><code>.export(**kwargs)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/fsmn_vad_streaming/model.py#L970" target="_blank" class="gh-link">L970</a><div class="mdoc"><p>Export.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>**kwargs</code> — Additional keyword arguments.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def export(self, **kwargs):
+
+ &quot;&quot;&quot;Export.
+
+ Args:
+ **kwargs: Additional keyword arguments.
+ &quot;&quot;&quot;
+ from .export_meta import export_rebuild_model
+
+ models = export_rebuild_model(model=self, **kwargs)
+ return models
+</code></pre></details></div><div class="mblk"><code>.DetectCommonFrames(cache)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/fsmn_vad_streaming/model.py#L982" target="_blank" class="gh-link">L982</a><div class="mdoc"><p>Detectcommonframes.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>cache</code> — State cache dict for streaming inference.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def DetectCommonFrames(self, cache: dict = None) -&gt; int:
+ &quot;&quot;&quot;Detectcommonframes.
+
+ Args:
+ cache: State cache dict for streaming inference.
+ &quot;&quot;&quot;
+ if cache is None:
+ cache = {}
+ if cache[&quot;stats&quot;].vad_state_machine == VadStateMachine.kVadInStateEndPointDetected:
+ return 0
+ for i in range(self.vad_opts.nn_eval_block_size - 1, -1, -1):
+ frame_state = FrameState.kFrameStateInvalid
+ frame_state = self.GetFrameState(
+ cache[&quot;stats&quot;].frm_cnt - 1 - i - cache[&quot;stats&quot;].last_drop_frames, cache=cache
+ )
+ self.DetectOneFrame(frame_state, cache[&quot;stats&quot;].frm_cnt - 1 - i, False, cache=cache)
+
+ return 0
+</code></pre></details></div><div class="mblk"><code>.DetectLastFrames(cache)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/fsmn_vad_streaming/model.py#L1001" target="_blank" class="gh-link">L1001</a><div class="mdoc"><p>Detectlastframes.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>cache</code> — State cache dict for streaming inference.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def DetectLastFrames(self, cache: dict = None) -&gt; int:
+ &quot;&quot;&quot;Detectlastframes.
+
+ Args:
+ cache: State cache dict for streaming inference.
+ &quot;&quot;&quot;
+ if cache is None:
+ cache = {}
+ if cache[&quot;stats&quot;].vad_state_machine == VadStateMachine.kVadInStateEndPointDetected:
+ return 0
+ for i in range(self.vad_opts.nn_eval_block_size - 1, -1, -1):
+ frame_state = FrameState.kFrameStateInvalid
+ frame_state = self.GetFrameState(
+ cache[&quot;stats&quot;].frm_cnt - 1 - i - cache[&quot;stats&quot;].last_drop_frames, cache=cache
+ )
+ if i != 0:
+ self.DetectOneFrame(frame_state, cache[&quot;stats&quot;].frm_cnt - 1 - i, False, cache=cache)
+ else:
+ self.DetectOneFrame(frame_state, cache[&quot;stats&quot;].frm_cnt - 1, True, cache=cache)
+
+ return 0
+</code></pre></details></div><div class="mblk"><code>.DetectOneFrame(cur_frm_state, cur_frm_idx, is_final_frame, cache)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/fsmn_vad_streaming/model.py#L1023" target="_blank" class="gh-link">L1023</a><div class="mdoc"><p>Detectoneframe.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>cur_frm_state</code> — TODO.</li>
+<li><code>cur_frm_idx</code> — TODO.</li>
+<li><code>is_final_frame</code> — Boolean flag for final frame.</li>
+<li><code>cache</code> — State cache dict for streaming inference.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def DetectOneFrame(
+ self, cur_frm_state: FrameState, cur_frm_idx: int, is_final_frame: bool, cache: dict = None
+ ) -&gt; None:
+ &quot;&quot;&quot;Detectoneframe.
+
+ Args:
+ cur_frm_state: TODO.
+ cur_frm_idx: TODO.
+ is_final_frame: Boolean flag for final frame.
+ cache: State cache dict for streaming inference.
+ &quot;&quot;&quot;
+ if cache is None:
+ cache = {}
+ tmp_cur_frm_state = FrameState.kFrameStateInvalid
+ if cur_frm_state == FrameState.kFrameStateSpeech:
+ if math.fabs(1.0) &gt; self.vad_opts.fe_prior_thres:
+ tmp_cur_frm_state = FrameState.kFrameStateSpeech
+ else:
+ tmp_cur_frm_state = FrameState.kFrameStateSil
+ elif cur_frm_state == FrameState.kFrameStateSil:
+ tmp_cur_frm_state = FrameState.kFrameStateSil
+ state_change = cache[&quot;windows_detector&quot;].DetectOneFrame(
+ tmp_cur_frm_state, cur_frm_idx, cache=cache
+ )
+ frm_shift_in_ms = self.vad_opts.frame_in_ms
+ if AudioChangeState.kChangeStateSil2Speech == state_change:
+ silence_frame_count = cache[&quot;stats&quot;].continous_silence_frame_count
+ cache[&quot;stats&quot;].continous_silence_frame_count = 0
+ cache[&quot;stats&quot;].pre_end_silence_detected = False
+ start_frame = 0
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/fsmn_vad_streaming/model.py#L1023" target="_blank" class="src-more">View full source →</a></details></div></div>
+<div class="api-detail" id="e85"><span class="dtag badge-m">method</span><h2><code>FsmnVADStreaming.ResetDetection(cache)</code></h2><div class="dmod">funasr.models.fsmn_vad_streaming.FsmnVADStreaming · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/fsmn_vad_streaming/model.py#L382" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Resetdetection.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>cache</code> — State cache dict for streaming inference.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def ResetDetection(self, cache: dict = None):
+ &quot;&quot;&quot;Resetdetection.
+
+ Args:
+ cache: State cache dict for streaming inference.
+ &quot;&quot;&quot;
+ if cache is None:
+ cache = {}
+ cache[&quot;stats&quot;].continous_silence_frame_count = 0
+ cache[&quot;stats&quot;].latest_confirmed_speech_frame = 0
+ cache[&quot;stats&quot;].lastest_confirmed_silence_frame = -1
+ cache[&quot;stats&quot;].confirmed_start_frame = -1
+ cache[&quot;stats&quot;].confirmed_end_frame = -1
+ cache[&quot;stats&quot;].vad_state_machine = VadStateMachine.kVadInStateStartPointNotDetected
+ cache[&quot;windows_detector&quot;].Reset()
+ cache[&quot;stats&quot;].sil_frame = 0
+ cache[&quot;stats&quot;].frame_probs = []
+
+ if cache[&quot;stats&quot;].output_data_buf:
+ assert cache[&quot;stats&quot;].output_data_buf[-1].contain_seg_end_point == True
+ drop_frames = int(cache[&quot;stats&quot;].output_data_buf[-1].end_ms / self.vad_opts.frame_in_ms)
+ real_drop_frames = drop_frames - cache[&quot;stats&quot;].last_drop_frames
+ cache[&quot;stats&quot;].last_drop_frames = drop_frames
+ cache[&quot;stats&quot;].data_buf_all = cache[&quot;stats&quot;].data_buf_all[
+ real_drop_frames
+ * int(self.vad_opts.frame_in_ms * self.vad_opts.sample_rate / 1000) :
+ ]
+ cache[&quot;stats&quot;].decibel = cache[&quot;stats&quot;].decibel[real_drop_frames:]
+ cache[&quot;stats&quot;].scores = cache[&quot;stats&quot;].scores[:, real_drop_frames:, :]
+</code></pre></details></div>
+<div class="api-detail" id="e86"><span class="dtag badge-m">method</span><h2><code>FsmnVADStreaming.ComputeDecibel(cache)</code></h2><div class="dmod">funasr.models.fsmn_vad_streaming.FsmnVADStreaming · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/fsmn_vad_streaming/model.py#L412" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Computedecibel.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>cache</code> — State cache dict for streaming inference.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def ComputeDecibel(self, cache: dict = None) -&gt; None:
+ &quot;&quot;&quot;Computedecibel.
+
+ Args:
+ cache: State cache dict for streaming inference.
+ &quot;&quot;&quot;
+ if cache is None:
+ cache = {}
+ frame_sample_length = int(self.vad_opts.frame_length_ms * self.vad_opts.sample_rate / 1000)
+ frame_shift_length = int(self.vad_opts.frame_in_ms * self.vad_opts.sample_rate / 1000)
+ if cache[&quot;stats&quot;].data_buf_all is None:
+ cache[&quot;stats&quot;].data_buf_all = cache[&quot;stats&quot;].waveform[
+ 0
+ ] # cache[&quot;stats&quot;].data_buf is pointed to cache[&quot;stats&quot;].waveform[0]
+ cache[&quot;stats&quot;].data_buf = cache[&quot;stats&quot;].data_buf_all
+ else:
+ cache[&quot;stats&quot;].data_buf_all = torch.cat(
+ (cache[&quot;stats&quot;].data_buf_all, cache[&quot;stats&quot;].waveform[0])
+ )
+
+ waveform_numpy = cache[&quot;stats&quot;].waveform.numpy()
+
+ offsets = np.arange(0, waveform_numpy.shape[1] - frame_sample_length + 1, frame_shift_length)
+ frames = waveform_numpy[0, offsets[:, np.newaxis] + np.arange(frame_sample_length)]
+
+ decibel_numpy = 10 * np.log10(np.sum(np.square(frames), axis=1) + 0.000001)
+ decibel_numpy = decibel_numpy.tolist()
+
+ cache[&quot;stats&quot;].decibel.extend(decibel_numpy)
+</code></pre></details></div>
+<div class="api-detail" id="e87"><span class="dtag badge-m">method</span><h2><code>FsmnVADStreaming.ComputeScores(feats, cache)</code></h2><div class="dmod">funasr.models.fsmn_vad_streaming.FsmnVADStreaming · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/fsmn_vad_streaming/model.py#L443" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Computescores.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>feats</code> — Feature tensor (e.g., fbank), shape (batch, frames, dim).</li>
+<li><code>cache</code> — State cache dict for streaming inference.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def ComputeScores(self, feats: torch.Tensor, cache: dict = None) -&gt; None:
+ &quot;&quot;&quot;Computescores.
+
+ Args:
+ feats: Feature tensor (e.g., fbank), shape (batch, frames, dim).
+ cache: State cache dict for streaming inference.
+ &quot;&quot;&quot;
+ if cache is None:
+ cache = {}
+ scores = self.encoder(feats, cache=cache[&quot;encoder&quot;]) # return B * T * D
+ assert (
+ scores.shape[1] == feats.shape[1]
+ ), &quot;The shape between feats and scores does not match&quot;
+ self.vad_opts.nn_eval_block_size = scores.shape[1]
+ cache[&quot;stats&quot;].frm_cnt += scores.shape[1] # count total frames
+ if cache[&quot;stats&quot;].scores is None:
+ cache[&quot;stats&quot;].scores = scores # the first calculation
+ else:
+ cache[&quot;stats&quot;].scores = torch.cat((cache[&quot;stats&quot;].scores, scores), dim=1)
+</code></pre></details></div>
+<div class="api-detail" id="e88"><span class="dtag badge-m">method</span><h2><code>FsmnVADStreaming.PopDataBufTillFrame(frame_idx, cache)</code></h2><div class="dmod">funasr.models.fsmn_vad_streaming.FsmnVADStreaming · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/fsmn_vad_streaming/model.py#L463" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Popdatabuftillframe.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>frame_idx</code> — TODO.</li>
+<li><code>cache</code> — State cache dict for streaming inference.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def PopDataBufTillFrame(self, frame_idx: int, cache: dict = None) -&gt; None: # need check again
+ &quot;&quot;&quot;Popdatabuftillframe.
+
+ Args:
+ frame_idx: TODO.
+ cache: State cache dict for streaming inference.
+ &quot;&quot;&quot;
+ while cache[&quot;stats&quot;].data_buf_start_frame &lt; frame_idx:
+ if len(cache[&quot;stats&quot;].data_buf) &gt;= int(
+ self.vad_opts.frame_in_ms * self.vad_opts.sample_rate / 1000
+ ):
+ if cache is None:
+ cache = {}
+ cache[&quot;stats&quot;].data_buf_start_frame += 1
+ cache[&quot;stats&quot;].data_buf = cache[&quot;stats&quot;].data_buf_all[
+ (cache[&quot;stats&quot;].data_buf_start_frame - cache[&quot;stats&quot;].last_drop_frames)
+ * int(self.vad_opts.frame_in_ms * self.vad_opts.sample_rate / 1000) :
+ ]
+</code></pre></details></div>
+<div class="api-detail" id="e89"><span class="dtag badge-m">method</span><h2><code>FsmnVADStreaming.PopDataToOutputBuf(start_frm, frm_cnt, first_frm_is_start_point, last_frm_is_end_point, end_point_is_sent_end, cache)</code></h2><div class="dmod">funasr.models.fsmn_vad_streaming.FsmnVADStreaming · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/fsmn_vad_streaming/model.py#L482" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Popdatatooutputbuf.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>start_frm</code> — TODO.</li>
+<li><code>frm_cnt</code> — TODO.</li>
+<li><code>first_frm_is_start_point</code> — TODO.</li>
+<li><code>last_frm_is_end_point</code> — TODO.</li>
+<li><code>end_point_is_sent_end</code> — TODO.</li>
+<li><code>cache</code> — State cache dict for streaming inference.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def PopDataToOutputBuf(
+ self,
+ start_frm: int,
+ frm_cnt: int,
+ first_frm_is_start_point: bool,
+ last_frm_is_end_point: bool,
+ end_point_is_sent_end: bool,
+ cache: dict = None,
+ ) -&gt; None:
+ &quot;&quot;&quot;Popdatatooutputbuf.
+
+ Args:
+ start_frm: TODO.
+ frm_cnt: TODO.
+ first_frm_is_start_point: TODO.
+ last_frm_is_end_point: TODO.
+ end_point_is_sent_end: TODO.
+ cache: State cache dict for streaming inference.
+ &quot;&quot;&quot;
+ if cache is None:
+ cache = {}
+ self.PopDataBufTillFrame(start_frm, cache=cache)
+ expected_sample_number = int(
+ frm_cnt * self.vad_opts.sample_rate * self.vad_opts.frame_in_ms / 1000
+ )
+ if last_frm_is_end_point:
+ extra_sample = max(
+ 0,
+ int(
+ self.vad_opts.frame_length_ms * self.vad_opts.sample_rate / 1000
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/fsmn_vad_streaming/model.py#L482" target="_blank" class="src-more">View full source on GitHub →</a></details></div>
+<div class="api-detail" id="e90"><span class="dtag badge-m">method</span><h2><code>FsmnVADStreaming.OnSilenceDetected(valid_frame, cache)</code></h2><div class="dmod">funasr.models.fsmn_vad_streaming.FsmnVADStreaming · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/fsmn_vad_streaming/model.py#L552" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Onsilencedetected.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>valid_frame</code> — TODO.</li>
+<li><code>cache</code> — State cache dict for streaming inference.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def OnSilenceDetected(self, valid_frame: int, cache: dict = None):
+ &quot;&quot;&quot;Onsilencedetected.
+
+ Args:
+ valid_frame: TODO.
+ cache: State cache dict for streaming inference.
+ &quot;&quot;&quot;
+ if cache is None:
+ cache = {}
+ cache[&quot;stats&quot;].lastest_confirmed_silence_frame = valid_frame
+ if cache[&quot;stats&quot;].vad_state_machine == VadStateMachine.kVadInStateStartPointNotDetected:
+ self.PopDataBufTillFrame(valid_frame, cache=cache)
+</code></pre></details></div>
+<div class="api-detail" id="e91"><span class="dtag badge-m">method</span><h2><code>FsmnVADStreaming.OnVoiceDetected(valid_frame, cache)</code></h2><div class="dmod">funasr.models.fsmn_vad_streaming.FsmnVADStreaming · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/fsmn_vad_streaming/model.py#L568" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Onvoicedetected.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>valid_frame</code> — TODO.</li>
+<li><code>cache</code> — State cache dict for streaming inference.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def OnVoiceDetected(self, valid_frame: int, cache: dict = None) -&gt; None:
+ &quot;&quot;&quot;Onvoicedetected.
+
+ Args:
+ valid_frame: TODO.
+ cache: State cache dict for streaming inference.
+ &quot;&quot;&quot;
+ if cache is None:
+ cache = {}
+ cache[&quot;stats&quot;].latest_confirmed_speech_frame = valid_frame
+ self.PopDataToOutputBuf(valid_frame, 1, False, False, False, cache=cache)
+</code></pre></details></div>
+<div class="api-detail" id="e92"><span class="dtag badge-m">method</span><h2><code>FsmnVADStreaming.OnVoiceStart(start_frame, fake_result, cache)</code></h2><div class="dmod">funasr.models.fsmn_vad_streaming.FsmnVADStreaming · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/fsmn_vad_streaming/model.py#L580" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Onvoicestart.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>start_frame</code> — TODO.</li>
+<li><code>fake_result</code> — TODO.</li>
+<li><code>cache</code> — State cache dict for streaming inference.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def OnVoiceStart(self, start_frame: int, fake_result: bool = False, cache: dict = None) -&gt; None:
+ &quot;&quot;&quot;Onvoicestart.
+
+ Args:
+ start_frame: TODO.
+ fake_result: TODO.
+ cache: State cache dict for streaming inference.
+ &quot;&quot;&quot;
+ if cache is None:
+ cache = {}
+ if self.vad_opts.do_start_point_detection:
+ pass
+ if cache[&quot;stats&quot;].confirmed_start_frame != -1:
+ print(&quot;not reset vad properly\n&quot;)
+ else:
+ cache[&quot;stats&quot;].confirmed_start_frame = start_frame
+
+ if (
+ not fake_result
+ and cache[&quot;stats&quot;].vad_state_machine == VadStateMachine.kVadInStateStartPointNotDetected
+ ):
+ self.PopDataToOutputBuf(
+ cache[&quot;stats&quot;].confirmed_start_frame, 1, True, False, False, cache=cache
+ )
+</code></pre></details></div>
+<div class="api-detail" id="e93"><span class="dtag badge-m">method</span><h2><code>FsmnVADStreaming.OnVoiceEnd(end_frame, fake_result, is_last_frame, cache)</code></h2><div class="dmod">funasr.models.fsmn_vad_streaming.FsmnVADStreaming · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/fsmn_vad_streaming/model.py#L605" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Onvoiceend.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>end_frame</code> — TODO.</li>
+<li><code>fake_result</code> — TODO.</li>
+<li><code>is_last_frame</code> — Boolean flag for last frame.</li>
+<li><code>cache</code> — State cache dict for streaming inference.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def OnVoiceEnd(
+ self, end_frame: int, fake_result: bool, is_last_frame: bool, cache: dict = None
+ ) -&gt; None:
+ &quot;&quot;&quot;Onvoiceend.
+
+ Args:
+ end_frame: TODO.
+ fake_result: TODO.
+ is_last_frame: Boolean flag for last frame.
+ cache: State cache dict for streaming inference.
+ &quot;&quot;&quot;
+ if cache is None:
+ cache = {}
+ for t in range(cache[&quot;stats&quot;].latest_confirmed_speech_frame + 1, end_frame):
+ self.OnVoiceDetected(t, cache=cache)
+ if self.vad_opts.do_end_point_detection:
+ pass
+ if cache[&quot;stats&quot;].confirmed_end_frame != -1:
+ print(&quot;not reset vad properly\n&quot;)
+ else:
+ cache[&quot;stats&quot;].confirmed_end_frame = end_frame
+ if not fake_result:
+ cache[&quot;stats&quot;].sil_frame = 0
+ self.PopDataToOutputBuf(
+ cache[&quot;stats&quot;].confirmed_end_frame, 1, False, True, is_last_frame, cache=cache
+ )
+ cache[&quot;stats&quot;].number_end_time_detected += 1
+</code></pre></details></div>
+<div class="api-detail" id="e94"><span class="dtag badge-m">method</span><h2><code>FsmnVADStreaming.MaybeOnVoiceEndIfLastFrame(is_final_frame, cur_frm_idx, cache)</code></h2><div class="dmod">funasr.models.fsmn_vad_streaming.FsmnVADStreaming · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/fsmn_vad_streaming/model.py#L633" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Maybeonvoiceendiflastframe.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>is_final_frame</code> — Boolean flag for final frame.</li>
+<li><code>cur_frm_idx</code> — TODO.</li>
+<li><code>cache</code> — State cache dict for streaming inference.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def MaybeOnVoiceEndIfLastFrame(
+ self, is_final_frame: bool, cur_frm_idx: int, cache: dict = None
+ ) -&gt; None:
+ &quot;&quot;&quot;Maybeonvoiceendiflastframe.
+
+ Args:
+ is_final_frame: Boolean flag for final frame.
+ cur_frm_idx: TODO.
+ cache: State cache dict for streaming inference.
+ &quot;&quot;&quot;
+ if cache is None:
+ cache = {}
+ if is_final_frame:
+ self.OnVoiceEnd(cur_frm_idx, False, True, cache=cache)
+ cache[&quot;stats&quot;].vad_state_machine = VadStateMachine.kVadInStateEndPointDetected
+</code></pre></details></div>
+<div class="api-detail" id="e95"><span class="dtag badge-m">method</span><h2><code>FsmnVADStreaming.GetLatency(cache)</code></h2><div class="dmod">funasr.models.fsmn_vad_streaming.FsmnVADStreaming · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/fsmn_vad_streaming/model.py#L649" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Getlatency.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>cache</code> — State cache dict for streaming inference.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def GetLatency(self, cache: dict = None) -&gt; int:
+ &quot;&quot;&quot;Getlatency.
+
+ Args:
+ cache: State cache dict for streaming inference.
+ &quot;&quot;&quot;
+ if cache is None:
+ cache = {}
+ return int(self.LatencyFrmNumAtStartPoint(cache=cache) * self.vad_opts.frame_in_ms)
+</code></pre></details></div>
+<div class="api-detail" id="e96"><span class="dtag badge-m">method</span><h2><code>FsmnVADStreaming.LatencyFrmNumAtStartPoint(cache)</code></h2><div class="dmod">funasr.models.fsmn_vad_streaming.FsmnVADStreaming · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/fsmn_vad_streaming/model.py#L659" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Latencyfrmnumatstartpoint.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>cache</code> — State cache dict for streaming inference.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def LatencyFrmNumAtStartPoint(self, cache: dict = None) -&gt; int:
+ &quot;&quot;&quot;Latencyfrmnumatstartpoint.
+
+ Args:
+ cache: State cache dict for streaming inference.
+ &quot;&quot;&quot;
+ if cache is None:
+ cache = {}
+ vad_latency = cache[&quot;windows_detector&quot;].GetWinSize()
+ if self.vad_opts.do_extend:
+ vad_latency += int(self.vad_opts.lookback_time_start_point / self.vad_opts.frame_in_ms)
+ return vad_latency
+</code></pre></details></div>
+<div class="api-detail" id="e97"><span class="dtag badge-m">method</span><h2><code>FsmnVADStreaming.GetFrameState(t, cache)</code></h2><div class="dmod">funasr.models.fsmn_vad_streaming.FsmnVADStreaming · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/fsmn_vad_streaming/model.py#L672" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Getframestate.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>t</code> — TODO.</li>
+<li><code>cache</code> — State cache dict for streaming inference.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def GetFrameState(self, t: int, cache: dict = None):
+ &quot;&quot;&quot;Getframestate.
+
+ Args:
+ t: TODO.
+ cache: State cache dict for streaming inference.
+ &quot;&quot;&quot;
+ if cache is None:
+ cache = {}
+ frame_state = FrameState.kFrameStateInvalid
+ if t &gt;= len(cache[&quot;stats&quot;].decibel):
+ return FrameState.kFrameStateSil
+ cur_decibel = cache[&quot;stats&quot;].decibel[t]
+ cur_snr = cur_decibel - cache[&quot;stats&quot;].noise_average_decibel
+ # for each frame, calc log posterior probability of each state
+ if cur_decibel &lt; self.vad_opts.decibel_thres:
+ frame_state = FrameState.kFrameStateSil
+ self.DetectOneFrame(frame_state, t, False, cache=cache)
+ return frame_state
+
+ sum_score = 0.0
+ noise_prob = 0.0
+ assert len(cache[&quot;stats&quot;].sil_pdf_ids) == self.vad_opts.silence_pdf_num
+ if len(cache[&quot;stats&quot;].sil_pdf_ids) &gt; 0:
+ assert len(cache[&quot;stats&quot;].scores) == 1 # 只支持batch_size = 1的测试
+ &quot;&quot;&quot;
+ - Change type of `sum_score` to float. The reason is that `sum_score` is a tensor with single element.
+ and `torch.Tensor` is slower `float` when tensor has only one element.
+ - Put the iteration of `sil_pdf_ids` inside `sum()` to reduce the overhead of creating a new list.
+ - The default `sil_pdf_ids` is [0], the `if` statement is used to reduce the overhead of expression
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/fsmn_vad_streaming/model.py#L672" target="_blank" class="src-more">View full source on GitHub →</a></details></div>
+<div class="api-detail" id="e98"><span class="dtag badge-m">method</span><h2><code>FsmnVADStreaming.forward(feats, waveform, cache, is_final, **kwargs)</code></h2><div class="dmod">funasr.models.fsmn_vad_streaming.FsmnVADStreaming · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/fsmn_vad_streaming/model.py#L737" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Forward pass for training.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>feats</code> — Feature tensor (e.g., fbank), shape (batch, frames, dim).</li>
+<li><code>waveform</code> — TODO.</li>
+<li><code>cache</code> — State cache dict for streaming inference.</li>
+<li><code>is_final</code> — Whether this is the final chunk in streaming.</li>
+<li><code>**kwargs</code> — Additional keyword arguments.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def forward(
+ self,
+ feats: torch.Tensor,
+ waveform: torch.tensor,
+ cache: dict = None,
+ is_final: bool = False,
+ **kwargs,
+ ):
+ &quot;&quot;&quot;Forward pass for training.
+
+ Args:
+ feats: Feature tensor (e.g., fbank), shape (batch, frames, dim).
+ waveform: TODO.
+ cache: State cache dict for streaming inference.
+ is_final: Whether this is the final chunk in streaming.
+ **kwargs: Additional keyword arguments.
+ &quot;&quot;&quot;
+ if cache is None:
+ cache = {}
+ # if len(cache) == 0:
+ # self.AllResetDetection()
+ # self.waveform = waveform # compute decibel for each frame
+ cache[&quot;stats&quot;].waveform = waveform
+ is_streaming_input = kwargs.get(&quot;is_streaming_input&quot;, True)
+ self.ComputeDecibel(cache=cache)
+ self.ComputeScores(feats, cache=cache)
+ if not is_final:
+ self.DetectCommonFrames(cache=cache)
+ else:
+ self.DetectLastFrames(cache=cache)
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/fsmn_vad_streaming/model.py#L737" target="_blank" class="src-more">View full source on GitHub →</a></details></div>
+<div class="api-detail" id="e99"><span class="dtag badge-m">method</span><h2><code>FsmnVADStreaming.init_cache(cache, **kwargs)</code></h2><div class="dmod">funasr.models.fsmn_vad_streaming.FsmnVADStreaming · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/fsmn_vad_streaming/model.py#L820" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Init cache.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>cache</code> — State cache dict for streaming inference.</li>
+<li><code>**kwargs</code> — Additional keyword arguments.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def init_cache(self, cache: dict = None, **kwargs):
+ &quot;&quot;&quot;Init cache.
+
+ Args:
+ cache: State cache dict for streaming inference.
+ **kwargs: Additional keyword arguments.
+ &quot;&quot;&quot;
+ if cache is None:
+ cache = {}
+
+ cache[&quot;frontend&quot;] = {}
+ cache[&quot;prev_samples&quot;] = torch.empty(0)
+ cache[&quot;encoder&quot;] = {}
+
+ if kwargs.get(&quot;max_end_silence_time&quot;) is not None:
+ # update the max_end_silence_time
+ self.vad_opts.max_end_silence_time = kwargs.get(&quot;max_end_silence_time&quot;)
+
+ windows_detector = WindowDetector(
+ self.vad_opts.window_size_ms,
+ self.vad_opts.sil_to_speech_time_thres,
+ self.vad_opts.speech_to_sil_time_thres,
+ self.vad_opts.frame_in_ms,
+ )
+ windows_detector.Reset()
+
+ stats = Stats(
+ sil_pdf_ids=self.vad_opts.sil_pdf_ids,
+ max_end_sil_frame_cnt_thresh=self.vad_opts.max_end_silence_time
+ - self.vad_opts.speech_to_sil_time_thres,
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/fsmn_vad_streaming/model.py#L820" target="_blank" class="src-more">View full source on GitHub →</a></details></div>
+<div class="api-detail" id="e100"><span class="dtag badge-m">method</span><h2><code>FsmnVADStreaming.inference(data_in, data_lengths, key, tokenizer, frontend, cache, **kwargs)</code></h2><div class="dmod">funasr.models.fsmn_vad_streaming.FsmnVADStreaming · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/fsmn_vad_streaming/model.py#L856" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Run inference on input data.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>data_in</code> — Input data (audio samples, file paths, or text).</li>
+<li><code>data_lengths</code> — Lengths of each input sample in the batch.</li>
+<li><code>key</code> — Sample identifiers.</li>
+<li><code>tokenizer</code> — Tokenizer instance for text encoding/decoding.</li>
+<li><code>frontend</code> — Audio frontend for feature extraction.</li>
+<li><code>cache</code> — State cache dict for streaming inference.</li>
+<li><code>**kwargs</code> — Additional keyword arguments.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def inference(
+ self,
+ data_in,
+ data_lengths=None,
+ key: list = None,
+ tokenizer=None,
+ frontend=None,
+ cache: dict = None,
+ **kwargs,
+ ):
+ &quot;&quot;&quot;Run inference on input data.
+
+ Args:
+ data_in: Input data (audio samples, file paths, or text).
+ data_lengths: Lengths of each input sample in the batch.
+ key: Sample identifiers.
+ tokenizer: Tokenizer instance for text encoding/decoding.
+ frontend: Audio frontend for feature extraction.
+ cache: State cache dict for streaming inference.
+ **kwargs: Additional keyword arguments.
+ &quot;&quot;&quot;
+ if cache is None:
+ cache = {}
+ if len(cache) == 0:
+ self.init_cache(cache, **kwargs)
+
+ meta_data = {}
+ chunk_size = kwargs.get(&quot;chunk_size&quot;, 60000) # 50ms
+ chunk_stride_samples = int(chunk_size * frontend.fs / 1000)
+
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/fsmn_vad_streaming/model.py#L856" target="_blank" class="src-more">View full source on GitHub →</a></details></div>
+<div class="api-detail" id="e101"><span class="dtag badge-m">method</span><h2><code>FsmnVADStreaming.export(**kwargs)</code></h2><div class="dmod">funasr.models.fsmn_vad_streaming.FsmnVADStreaming · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/fsmn_vad_streaming/model.py#L970" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Export.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>**kwargs</code> — Additional keyword arguments.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def export(self, **kwargs):
+
+ &quot;&quot;&quot;Export.
+
+ Args:
+ **kwargs: Additional keyword arguments.
+ &quot;&quot;&quot;
+ from .export_meta import export_rebuild_model
+
+ models = export_rebuild_model(model=self, **kwargs)
+ return models
+</code></pre></details></div>
+<div class="api-detail" id="e102"><span class="dtag badge-m">method</span><h2><code>FsmnVADStreaming.DetectCommonFrames(cache)</code></h2><div class="dmod">funasr.models.fsmn_vad_streaming.FsmnVADStreaming · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/fsmn_vad_streaming/model.py#L982" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Detectcommonframes.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>cache</code> — State cache dict for streaming inference.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def DetectCommonFrames(self, cache: dict = None) -&gt; int:
+ &quot;&quot;&quot;Detectcommonframes.
+
+ Args:
+ cache: State cache dict for streaming inference.
+ &quot;&quot;&quot;
+ if cache is None:
+ cache = {}
+ if cache[&quot;stats&quot;].vad_state_machine == VadStateMachine.kVadInStateEndPointDetected:
+ return 0
+ for i in range(self.vad_opts.nn_eval_block_size - 1, -1, -1):
+ frame_state = FrameState.kFrameStateInvalid
+ frame_state = self.GetFrameState(
+ cache[&quot;stats&quot;].frm_cnt - 1 - i - cache[&quot;stats&quot;].last_drop_frames, cache=cache
+ )
+ self.DetectOneFrame(frame_state, cache[&quot;stats&quot;].frm_cnt - 1 - i, False, cache=cache)
+
+ return 0
+</code></pre></details></div>
+<div class="api-detail" id="e103"><span class="dtag badge-m">method</span><h2><code>FsmnVADStreaming.DetectLastFrames(cache)</code></h2><div class="dmod">funasr.models.fsmn_vad_streaming.FsmnVADStreaming · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/fsmn_vad_streaming/model.py#L1001" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Detectlastframes.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>cache</code> — State cache dict for streaming inference.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def DetectLastFrames(self, cache: dict = None) -&gt; int:
+ &quot;&quot;&quot;Detectlastframes.
+
+ Args:
+ cache: State cache dict for streaming inference.
+ &quot;&quot;&quot;
+ if cache is None:
+ cache = {}
+ if cache[&quot;stats&quot;].vad_state_machine == VadStateMachine.kVadInStateEndPointDetected:
+ return 0
+ for i in range(self.vad_opts.nn_eval_block_size - 1, -1, -1):
+ frame_state = FrameState.kFrameStateInvalid
+ frame_state = self.GetFrameState(
+ cache[&quot;stats&quot;].frm_cnt - 1 - i - cache[&quot;stats&quot;].last_drop_frames, cache=cache
+ )
+ if i != 0:
+ self.DetectOneFrame(frame_state, cache[&quot;stats&quot;].frm_cnt - 1 - i, False, cache=cache)
+ else:
+ self.DetectOneFrame(frame_state, cache[&quot;stats&quot;].frm_cnt - 1, True, cache=cache)
+
+ return 0
+</code></pre></details></div>
+<div class="api-detail" id="e104"><span class="dtag badge-m">method</span><h2><code>FsmnVADStreaming.DetectOneFrame(cur_frm_state, cur_frm_idx, is_final_frame, cache)</code></h2><div class="dmod">funasr.models.fsmn_vad_streaming.FsmnVADStreaming · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/fsmn_vad_streaming/model.py#L1023" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Detectoneframe.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>cur_frm_state</code> — TODO.</li>
+<li><code>cur_frm_idx</code> — TODO.</li>
+<li><code>is_final_frame</code> — Boolean flag for final frame.</li>
+<li><code>cache</code> — State cache dict for streaming inference.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def DetectOneFrame(
+ self, cur_frm_state: FrameState, cur_frm_idx: int, is_final_frame: bool, cache: dict = None
+ ) -&gt; None:
+ &quot;&quot;&quot;Detectoneframe.
+
+ Args:
+ cur_frm_state: TODO.
+ cur_frm_idx: TODO.
+ is_final_frame: Boolean flag for final frame.
+ cache: State cache dict for streaming inference.
+ &quot;&quot;&quot;
+ if cache is None:
+ cache = {}
+ tmp_cur_frm_state = FrameState.kFrameStateInvalid
+ if cur_frm_state == FrameState.kFrameStateSpeech:
+ if math.fabs(1.0) &gt; self.vad_opts.fe_prior_thres:
+ tmp_cur_frm_state = FrameState.kFrameStateSpeech
+ else:
+ tmp_cur_frm_state = FrameState.kFrameStateSil
+ elif cur_frm_state == FrameState.kFrameStateSil:
+ tmp_cur_frm_state = FrameState.kFrameStateSil
+ state_change = cache[&quot;windows_detector&quot;].DetectOneFrame(
+ tmp_cur_frm_state, cur_frm_idx, cache=cache
+ )
+ frm_shift_in_ms = self.vad_opts.frame_in_ms
+ if AudioChangeState.kChangeStateSil2Speech == state_change:
+ silence_frame_count = cache[&quot;stats&quot;].continous_silence_frame_count
+ cache[&quot;stats&quot;].continous_silence_frame_count = 0
+ cache[&quot;stats&quot;].pre_end_silence_detected = False
+ start_frame = 0
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/fsmn_vad_streaming/model.py#L1023" target="_blank" class="src-more">View full source on GitHub →</a></details></div>
+<div class="api-detail" id="e105"><span class="dtag badge-c">class</span><h2>FunASRNano</h2><div class="dmod">funasr.models.fun_asr_nano · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/fun_asr_nano/model.py#L27" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><ul>
+<li><code>Fun</code> — ASR-Nano: End-to-End ASR Large Model.</li>
+</ul>
+<p>Trained on tens of millions of hours of real speech data.</p>
+<p>Supports 31 languages including Chinese dialects and regional accents.</p>
+<h4>Features:</h4>
+<ul>
+<li>Character-level timestamps (via CTC forced alignment)</li>
+<li>Hotword customization</li>
+<li>Speaker diarization (when combined with spk_model)</li>
+<li>Lyrics and rap recognition</li>
+<li>Streaming chunk-by-chunk inference (demo2.py)</li>
+<li><code>Output</code> — {&quot;key&quot;: ..., &quot;text&quot;: ..., &quot;timestamps&quot;: [{&quot;token&quot;, &quot;start_time&quot;, &quot;end_time&quot;}, ...],</li>
+<li class="sub">&quot;ctc_timestamps&quot;: [...]}</li>
+<li><code>Note</code> — Outputs punctuation natively — punc_model is NOT needed.</li>
+<li><code>Requirements</code> — pip install tiktoken huggingface_hub</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code>class FunASRNano(nn.Module):
+ &quot;&quot;&quot;Fun-ASR-Nano: End-to-End ASR Large Model.
+
+ Trained on tens of millions of hours of real speech data.
+ Supports 31 languages including Chinese dialects and regional accents.
+
+ Features:
+ - Character-level timestamps (via CTC forced alignment)
+ - Hotword customization
+ - Speaker diarization (when combined with spk_model)
+ - Lyrics and rap recognition
+ - Streaming chunk-by-chunk inference (demo2.py)
+
+ Output: {&quot;key&quot;: ..., &quot;text&quot;: ..., &quot;timestamps&quot;: [{&quot;token&quot;, &quot;start_time&quot;, &quot;end_time&quot;}, ...],
+ &quot;ctc_timestamps&quot;: [...]}
+
+ Note: Outputs punctuation natively — punc_model is NOT needed.
+
+ Requirements: pip install tiktoken huggingface_hub
+ &quot;&quot;&quot;
+
+ def __init__(
+ self,
+ audio_encoder: str = None,
+ audio_encoder_conf: dict = None,
+ audio_adaptor: str = None,
+ audio_adaptor_conf: dict = None,
+ llm: str = None,
+ llm_conf: dict = None,
+ input_size: int = 80,
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/fun_asr_nano/model.py#L27" target="_blank" class="src-more">View full source on GitHub →</a></details><h3 class="mt">Methods</h3><div class="mblk"><code>.forward(speech, speech_lengths, input_ids, attention_mask, labels_ids, fbank_beg, fbank_mask, **kwargs)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/fun_asr_nano/model.py#L194" target="_blank" class="gh-link">L194</a><div class="mdoc"><p>Forward pass for training.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>speech</code> — Speech audio tensor, shape (batch, time).</li>
+<li><code>speech_lengths</code> — Length of each speech sample.</li>
+<li><code>input_ids</code> — TODO.</li>
+<li><code>attention_mask</code> — TODO.</li>
+<li><code>labels_ids</code> — TODO.</li>
+<li><code>fbank_beg</code> — TODO.</li>
+<li><code>fbank_mask</code> — TODO.</li>
+<li><code>**kwargs</code> — Additional keyword arguments.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def forward(
+ self,
+ speech: torch.Tensor = None,
+ speech_lengths: torch.Tensor = None,
+ input_ids: torch.Tensor = None,
+ attention_mask: torch.Tensor = None,
+ labels_ids: torch.Tensor = None,
+ fbank_beg: torch.Tensor = None,
+ fbank_mask: torch.Tensor = None,
+ **kwargs,
+ ):
+ &quot;&quot;&quot;Forward pass for training.
+
+ Args:
+ speech: Speech audio tensor, shape (batch, time).
+ speech_lengths: Length of each speech sample.
+ input_ids: TODO.
+ attention_mask: TODO.
+ labels_ids: TODO.
+ fbank_beg: TODO.
+ fbank_mask: TODO.
+ **kwargs: Additional keyword arguments.
+ &quot;&quot;&quot;
+ batch_size, token_num = input_ids.shape
+ stats = {}
+ input_ids[input_ids &lt; 0] = 0
+ inputs_embeds = self.llm.model.get_input_embeddings()(input_ids)
+ if speech is not None:
+ if len(speech_lengths.size()) &gt; 1:
+ speech_lengths = speech_lengths[:, 0]
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/fun_asr_nano/model.py#L194" target="_blank" class="src-more">View full source →</a></details></div><div class="mblk"><code>.forward_export(speech, speech_lengths, **kwargs)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/fun_asr_nano/model.py#L317" target="_blank" class="gh-link">L317</a><div class="mdoc"><p>Forward export.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>speech</code> — Speech audio tensor, shape (batch, time).</li>
+<li><code>speech_lengths</code> — Length of each speech sample.</li>
+<li><code>**kwargs</code> — Additional keyword arguments.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def forward_export(self, speech, speech_lengths, **kwargs):
+ &quot;&quot;&quot;Forward export.
+
+ Args:
+ speech: Speech audio tensor, shape (batch, time).
+ speech_lengths: Length of each speech sample.
+ **kwargs: Additional keyword arguments.
+ &quot;&quot;&quot;
+ x, olens = self.audio_encoder(speech, speech_lengths)
+ encoder_out, encoder_out_lens = self.audio_adaptor(x, olens)
+ return encoder_out, encoder_out_lens
+</code></pre></details></div><div class="mblk"><code>.encode(speech, speech_lengths)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/fun_asr_nano/model.py#L329" target="_blank" class="gh-link">L329</a><div class="mdoc"><p>Encode.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>speech</code> — Speech audio tensor, shape (batch, time).</li>
+<li><code>speech_lengths</code> — Length of each speech sample.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def encode(self, speech, speech_lengths):
+ # audio encoder
+ &quot;&quot;&quot;Encode.
+
+ Args:
+ speech: Speech audio tensor, shape (batch, time).
+ speech_lengths: Length of each speech sample.
+ &quot;&quot;&quot;
+ encoder_out, encoder_out_lens = self.audio_encoder(speech, speech_lengths)
+
+ return encoder_out, encoder_out_lens
+</code></pre></details></div><div class="mblk"><code>.data_template(data)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/fun_asr_nano/model.py#L341" target="_blank" class="gh-link">L341</a><div class="mdoc"><p>Data template.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>data</code> — TODO.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def data_template(self, data):
+ &quot;&quot;&quot;Data template.
+
+ Args:
+ data: TODO.
+ &quot;&quot;&quot;
+ system, user, assistant = [], [], []
+ for i, item in enumerate(data):
+ role = item[&quot;role&quot;]
+ content = item[&quot;content&quot;]
+ if role == &quot;system&quot;:
+ system.append(content)
+ elif role == &quot;user&quot;:
+ if &quot;audio&quot; in item:
+ audio = item[&quot;audio&quot;]
+ content = [content, audio]
+ user.append(content)
+ elif role == &quot;assistant&quot;:
+ assistant.append(content)
+
+ system = system * len(user)
+
+ contents = {
+ &quot;system&quot;: system,
+ &quot;user&quot;: user,
+ &quot;assistant&quot;: assistant,
+ }
+
+ return contents
+</code></pre></details></div><div class="mblk"><code>.data_load_speech(contents, tokenizer, frontend, meta_data, **kwargs)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/fun_asr_nano/model.py#L371" target="_blank" class="gh-link">L371</a><div class="mdoc"><p>Data load speech.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>contents</code> — TODO.</li>
+<li><code>tokenizer</code> — Tokenizer instance for text encoding/decoding.</li>
+<li><code>frontend</code> — Audio frontend for feature extraction.</li>
+<li><code>meta_data</code> — TODO.</li>
+<li><code>**kwargs</code> — Additional keyword arguments.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def data_load_speech(self, contents: dict, tokenizer, frontend, meta_data={}, **kwargs):
+ &quot;&quot;&quot;Data load speech.
+
+ Args:
+ contents: TODO.
+ tokenizer: Tokenizer instance for text encoding/decoding.
+ frontend: Audio frontend for feature extraction.
+ meta_data: TODO.
+ **kwargs: Additional keyword arguments.
+ &quot;&quot;&quot;
+ system = contents[&quot;system&quot;]
+ user = contents[&quot;user&quot;]
+ assistant = contents[&quot;assistant&quot;]
+ pattern = re.compile(r&quot;(&lt;\|startofspeech\|&gt;.*?&lt;\|endofspeech\|&gt;)&quot;)
+ do_think = True
+ sys_prompt = True
+ if &quot;dataset_conf&quot; in kwargs:
+ do_think = kwargs[&quot;dataset_conf&quot;].get(&quot;do_think&quot;, True)
+ sys_prompt = kwargs[&quot;dataset_conf&quot;].get(&quot;sys_prompt&quot;, True)
+
+ input_ids, labels, fbank, fbank_lens, fbank_mask, fbank_beg, fake_token_len = (
+ [],
+ [],
+ [],
+ [],
+ [],
+ [],
+ [],
+ )
+ input_source_ids = []
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/fun_asr_nano/model.py#L371" target="_blank" class="src-more">View full source →</a></details></div><div class="mblk"><code>.inference_prepare(data_in, data_lengths, key, tokenizer, frontend, **kwargs)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/fun_asr_nano/model.py#L533" target="_blank" class="gh-link">L533</a><div class="mdoc"><p>Inference prepare.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>data_in</code> — Input data (audio samples, file paths, or text).</li>
+<li><code>data_lengths</code> — Lengths of each input sample in the batch.</li>
+<li><code>key</code> — Sample identifiers.</li>
+<li><code>tokenizer</code> — Tokenizer instance for text encoding/decoding.</li>
+<li><code>frontend</code> — Audio frontend for feature extraction.</li>
+<li><code>**kwargs</code> — Additional keyword arguments.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def inference_prepare(
+ self,
+ data_in,
+ data_lengths=None,
+ key: list = None,
+ tokenizer=None,
+ frontend=None,
+ **kwargs,
+ ):
+ &quot;&quot;&quot;Inference prepare.
+
+ Args:
+ data_in: Input data (audio samples, file paths, or text).
+ data_lengths: Lengths of each input sample in the batch.
+ key: Sample identifiers.
+ tokenizer: Tokenizer instance for text encoding/decoding.
+ frontend: Audio frontend for feature extraction.
+ **kwargs: Additional keyword arguments.
+ &quot;&quot;&quot;
+ meta_data = {}
+
+ if len(data_in) &gt; 1:
+ raise NotImplementedError(&quot;batch decoding is not implemented&quot;)
+
+ contents = self.data_template(data_in[0])
+ output = self.data_load_speech(contents, tokenizer, frontend, meta_data=meta_data, **kwargs)
+ batch = to_device(output, kwargs[&quot;device&quot;])
+
+ # audio encoder
+ speech = batch[&quot;speech&quot;]
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/fun_asr_nano/model.py#L533" target="_blank" class="src-more">View full source →</a></details></div><div class="mblk"><code>.get_prompt(hotwords, language, itn)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/fun_asr_nano/model.py#L632" target="_blank" class="gh-link">L632</a><div class="mdoc"><p>Get prompt.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>hotwords</code> — TODO.</li>
+<li><code>language</code> — Language identifier.</li>
+<li><code>itn</code> — TODO.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def get_prompt(self, hotwords: list[str], language: str = None, itn: bool = True):
+ &quot;&quot;&quot;Get prompt.
+
+ Args:
+ hotwords: TODO.
+ language: Language identifier.
+ itn: TODO.
+ &quot;&quot;&quot;
+ if len(hotwords) &gt; 0:
+ hotwords = &quot;, &quot;.join(hotwords)
+ prompt = f&quot;请结合上下文信息,更加准确地完成语音转写任务。如果没有相关信息,我们会留空。\n\n\n**上下文信息:**\n\n\n&quot;
+ prompt += f&quot;热词列表:[{hotwords}]\n&quot;
+ else:
+ prompt = &quot;&quot;
+ if language is None:
+ prompt += &quot;语音转写&quot;
+ else:
+ prompt += f&quot;语音转写成{language}&quot;
+ if not itn:
+ prompt += &quot;,不进行文本规整&quot;
+ return prompt + &quot;:&quot;
+</code></pre></details></div><div class="mblk"><code>.generate_chatml(prompt, data)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/fun_asr_nano/model.py#L654" target="_blank" class="gh-link">L654</a><div class="mdoc"><p>Generate chatml.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>prompt</code> — TODO.</li>
+<li><code>data</code> — TODO.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def generate_chatml(self, prompt: str, data: Union[str, torch.Tensor]):
+ &quot;&quot;&quot;Generate chatml.
+
+ Args:
+ prompt: TODO.
+ data: TODO.
+ &quot;&quot;&quot;
+ if isinstance(data, str):
+ return [
+ {&quot;role&quot;: &quot;system&quot;, &quot;content&quot;: &quot;You are a helpful assistant.&quot;},
+ {&quot;role&quot;: &quot;user&quot;, &quot;content&quot;: f&quot;{prompt}&lt;|startofspeech|&gt;!{data}&lt;|endofspeech|&gt;&quot;},
+ {&quot;role&quot;: &quot;assistant&quot;, &quot;content&quot;: &quot;null&quot;},
+ ]
+ elif isinstance(data, torch.Tensor):
+ return [
+ {&quot;role&quot;: &quot;system&quot;, &quot;content&quot;: &quot;You are a helpful assistant.&quot;},
+ {
+ &quot;role&quot;: &quot;user&quot;,
+ &quot;content&quot;: f&quot;{prompt}&lt;|startofspeech|&gt;!!&lt;|endofspeech|&gt;&quot;,
+ &quot;audio&quot;: data,
+ },
+ {&quot;role&quot;: &quot;assistant&quot;, &quot;content&quot;: &quot;null&quot;},
+ ]
+</code></pre></details></div><div class="mblk"><code>.inference(data_in, data_lengths, key, tokenizer, frontend, **kwargs)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/fun_asr_nano/model.py#L678" target="_blank" class="gh-link">L678</a><div class="mdoc"><p>Run inference on input data.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>data_in</code> — Input data (audio samples, file paths, or text).</li>
+<li><code>data_lengths</code> — Lengths of each input sample in the batch.</li>
+<li><code>key</code> — Sample identifiers.</li>
+<li><code>tokenizer</code> — Tokenizer instance for text encoding/decoding.</li>
+<li><code>frontend</code> — Audio frontend for feature extraction.</li>
+<li><code>**kwargs</code> — Additional keyword arguments.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def inference(
+ self,
+ data_in,
+ data_lengths=None,
+ key: list = None,
+ tokenizer=None,
+ frontend=None,
+ **kwargs,
+ ):
+ &quot;&quot;&quot;Run inference on input data.
+
+ Args:
+ data_in: Input data (audio samples, file paths, or text).
+ data_lengths: Lengths of each input sample in the batch.
+ key: Sample identifiers.
+ tokenizer: Tokenizer instance for text encoding/decoding.
+ frontend: Audio frontend for feature extraction.
+ **kwargs: Additional keyword arguments.
+ &quot;&quot;&quot;
+ prompt = self.get_prompt(
+ kwargs.get(&quot;hotwords&quot;, []), kwargs.get(&quot;language&quot;, None), kwargs.get(&quot;itn&quot;, True)
+ )
+ data_in = [self.generate_chatml(prompt, data) for data in data_in]
+
+ if key is None:
+ key = []
+ for _ in data_in:
+ chars = string.ascii_letters + string.digits
+ key.append(&quot;rand_key_&quot; + &quot;&quot;.join(random.choice(chars) for _ in range(13)))
+
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/fun_asr_nano/model.py#L678" target="_blank" class="src-more">View full source →</a></details></div><div class="mblk"><code>.inference_llm(data_in, data_lengths, key, tokenizer, frontend, **kwargs)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/fun_asr_nano/model.py#L717" target="_blank" class="gh-link">L717</a><div class="mdoc"><p>Inference llm.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>data_in</code> — Input data (audio samples, file paths, or text).</li>
+<li><code>data_lengths</code> — Lengths of each input sample in the batch.</li>
+<li><code>key</code> — Sample identifiers.</li>
+<li><code>tokenizer</code> — Tokenizer instance for text encoding/decoding.</li>
+<li><code>frontend</code> — Audio frontend for feature extraction.</li>
+<li><code>**kwargs</code> — Additional keyword arguments.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def inference_llm(
+ self,
+ data_in,
+ data_lengths=None,
+ key: list = None,
+ tokenizer=None,
+ frontend=None,
+ **kwargs,
+ ):
+ &quot;&quot;&quot;Inference llm.
+
+ Args:
+ data_in: Input data (audio samples, file paths, or text).
+ data_lengths: Lengths of each input sample in the batch.
+ key: Sample identifiers.
+ tokenizer: Tokenizer instance for text encoding/decoding.
+ frontend: Audio frontend for feature extraction.
+ **kwargs: Additional keyword arguments.
+ &quot;&quot;&quot;
+ inputs_embeds, contents, batch, source_ids, meta_data = self.inference_prepare(
+ data_in, data_lengths, key, tokenizer, frontend, **kwargs
+ )
+
+ ctc_results = []
+ if self.ctc_decoder is not None:
+ encoder_out = meta_data[&quot;encoder_out&quot;]
+ encoder_out_lens = meta_data[&quot;encoder_out_lens&quot;]
+ decoder_out, decoder_out_lens = self.ctc_decoder(encoder_out, encoder_out_lens)
+ ctc_logits = self.ctc.log_softmax(decoder_out)
+
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/fun_asr_nano/model.py#L717" target="_blank" class="src-more">View full source →</a></details></div><div class="mblk"><code>.from_pretrained(model, **kwargs)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/fun_asr_nano/model.py#L856" target="_blank" class="gh-link">L856</a><div class="mdoc"><p>From pretrained.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>model</code> — Model instance or model name.</li>
+<li><code>**kwargs</code> — Additional keyword arguments.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def from_pretrained(model: str = None, **kwargs):
+ &quot;&quot;&quot;From pretrained.
+
+ Args:
+ model: Model instance or model name.
+ **kwargs: Additional keyword arguments.
+ &quot;&quot;&quot;
+ from funasr import AutoModel
+
+ model, kwargs = AutoModel.build_model(model=model, trust_remote_code=True, **kwargs)
+
+ return model, kwargs
+</code></pre></details></div></div>
+<div class="api-detail" id="e106"><span class="dtag badge-m">method</span><h2><code>FunASRNano.forward(speech, speech_lengths, input_ids, attention_mask, labels_ids, fbank_beg, fbank_mask, **kwargs)</code></h2><div class="dmod">funasr.models.fun_asr_nano.FunASRNano · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/fun_asr_nano/model.py#L194" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Forward pass for training.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>speech</code> — Speech audio tensor, shape (batch, time).</li>
+<li><code>speech_lengths</code> — Length of each speech sample.</li>
+<li><code>input_ids</code> — TODO.</li>
+<li><code>attention_mask</code> — TODO.</li>
+<li><code>labels_ids</code> — TODO.</li>
+<li><code>fbank_beg</code> — TODO.</li>
+<li><code>fbank_mask</code> — TODO.</li>
+<li><code>**kwargs</code> — Additional keyword arguments.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def forward(
+ self,
+ speech: torch.Tensor = None,
+ speech_lengths: torch.Tensor = None,
+ input_ids: torch.Tensor = None,
+ attention_mask: torch.Tensor = None,
+ labels_ids: torch.Tensor = None,
+ fbank_beg: torch.Tensor = None,
+ fbank_mask: torch.Tensor = None,
+ **kwargs,
+ ):
+ &quot;&quot;&quot;Forward pass for training.
+
+ Args:
+ speech: Speech audio tensor, shape (batch, time).
+ speech_lengths: Length of each speech sample.
+ input_ids: TODO.
+ attention_mask: TODO.
+ labels_ids: TODO.
+ fbank_beg: TODO.
+ fbank_mask: TODO.
+ **kwargs: Additional keyword arguments.
+ &quot;&quot;&quot;
+ batch_size, token_num = input_ids.shape
+ stats = {}
+ input_ids[input_ids &lt; 0] = 0
+ inputs_embeds = self.llm.model.get_input_embeddings()(input_ids)
+ if speech is not None:
+ if len(speech_lengths.size()) &gt; 1:
+ speech_lengths = speech_lengths[:, 0]
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/fun_asr_nano/model.py#L194" target="_blank" class="src-more">View full source on GitHub →</a></details></div>
+<div class="api-detail" id="e107"><span class="dtag badge-m">method</span><h2><code>FunASRNano.forward_export(speech, speech_lengths, **kwargs)</code></h2><div class="dmod">funasr.models.fun_asr_nano.FunASRNano · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/fun_asr_nano/model.py#L317" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Forward export.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>speech</code> — Speech audio tensor, shape (batch, time).</li>
+<li><code>speech_lengths</code> — Length of each speech sample.</li>
+<li><code>**kwargs</code> — Additional keyword arguments.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def forward_export(self, speech, speech_lengths, **kwargs):
+ &quot;&quot;&quot;Forward export.
+
+ Args:
+ speech: Speech audio tensor, shape (batch, time).
+ speech_lengths: Length of each speech sample.
+ **kwargs: Additional keyword arguments.
+ &quot;&quot;&quot;
+ x, olens = self.audio_encoder(speech, speech_lengths)
+ encoder_out, encoder_out_lens = self.audio_adaptor(x, olens)
+ return encoder_out, encoder_out_lens
+</code></pre></details></div>
+<div class="api-detail" id="e108"><span class="dtag badge-m">method</span><h2><code>FunASRNano.encode(speech, speech_lengths)</code></h2><div class="dmod">funasr.models.fun_asr_nano.FunASRNano · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/fun_asr_nano/model.py#L329" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Encode.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>speech</code> — Speech audio tensor, shape (batch, time).</li>
+<li><code>speech_lengths</code> — Length of each speech sample.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def encode(self, speech, speech_lengths):
+ # audio encoder
+ &quot;&quot;&quot;Encode.
+
+ Args:
+ speech: Speech audio tensor, shape (batch, time).
+ speech_lengths: Length of each speech sample.
+ &quot;&quot;&quot;
+ encoder_out, encoder_out_lens = self.audio_encoder(speech, speech_lengths)
+
+ return encoder_out, encoder_out_lens
+</code></pre></details></div>
+<div class="api-detail" id="e109"><span class="dtag badge-m">method</span><h2><code>FunASRNano.data_template(data)</code></h2><div class="dmod">funasr.models.fun_asr_nano.FunASRNano · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/fun_asr_nano/model.py#L341" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Data template.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>data</code> — TODO.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def data_template(self, data):
+ &quot;&quot;&quot;Data template.
+
+ Args:
+ data: TODO.
+ &quot;&quot;&quot;
+ system, user, assistant = [], [], []
+ for i, item in enumerate(data):
+ role = item[&quot;role&quot;]
+ content = item[&quot;content&quot;]
+ if role == &quot;system&quot;:
+ system.append(content)
+ elif role == &quot;user&quot;:
+ if &quot;audio&quot; in item:
+ audio = item[&quot;audio&quot;]
+ content = [content, audio]
+ user.append(content)
+ elif role == &quot;assistant&quot;:
+ assistant.append(content)
+
+ system = system * len(user)
+
+ contents = {
+ &quot;system&quot;: system,
+ &quot;user&quot;: user,
+ &quot;assistant&quot;: assistant,
+ }
+
+ return contents
+</code></pre></details></div>
+<div class="api-detail" id="e110"><span class="dtag badge-m">method</span><h2><code>FunASRNano.data_load_speech(contents, tokenizer, frontend, meta_data, **kwargs)</code></h2><div class="dmod">funasr.models.fun_asr_nano.FunASRNano · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/fun_asr_nano/model.py#L371" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Data load speech.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>contents</code> — TODO.</li>
+<li><code>tokenizer</code> — Tokenizer instance for text encoding/decoding.</li>
+<li><code>frontend</code> — Audio frontend for feature extraction.</li>
+<li><code>meta_data</code> — TODO.</li>
+<li><code>**kwargs</code> — Additional keyword arguments.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def data_load_speech(self, contents: dict, tokenizer, frontend, meta_data={}, **kwargs):
+ &quot;&quot;&quot;Data load speech.
+
+ Args:
+ contents: TODO.
+ tokenizer: Tokenizer instance for text encoding/decoding.
+ frontend: Audio frontend for feature extraction.
+ meta_data: TODO.
+ **kwargs: Additional keyword arguments.
+ &quot;&quot;&quot;
+ system = contents[&quot;system&quot;]
+ user = contents[&quot;user&quot;]
+ assistant = contents[&quot;assistant&quot;]
+ pattern = re.compile(r&quot;(&lt;\|startofspeech\|&gt;.*?&lt;\|endofspeech\|&gt;)&quot;)
+ do_think = True
+ sys_prompt = True
+ if &quot;dataset_conf&quot; in kwargs:
+ do_think = kwargs[&quot;dataset_conf&quot;].get(&quot;do_think&quot;, True)
+ sys_prompt = kwargs[&quot;dataset_conf&quot;].get(&quot;sys_prompt&quot;, True)
+
+ input_ids, labels, fbank, fbank_lens, fbank_mask, fbank_beg, fake_token_len = (
+ [],
+ [],
+ [],
+ [],
+ [],
+ [],
+ [],
+ )
+ input_source_ids = []
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/fun_asr_nano/model.py#L371" target="_blank" class="src-more">View full source on GitHub →</a></details></div>
+<div class="api-detail" id="e111"><span class="dtag badge-m">method</span><h2><code>FunASRNano.inference_prepare(data_in, data_lengths, key, tokenizer, frontend, **kwargs)</code></h2><div class="dmod">funasr.models.fun_asr_nano.FunASRNano · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/fun_asr_nano/model.py#L533" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Inference prepare.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>data_in</code> — Input data (audio samples, file paths, or text).</li>
+<li><code>data_lengths</code> — Lengths of each input sample in the batch.</li>
+<li><code>key</code> — Sample identifiers.</li>
+<li><code>tokenizer</code> — Tokenizer instance for text encoding/decoding.</li>
+<li><code>frontend</code> — Audio frontend for feature extraction.</li>
+<li><code>**kwargs</code> — Additional keyword arguments.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def inference_prepare(
+ self,
+ data_in,
+ data_lengths=None,
+ key: list = None,
+ tokenizer=None,
+ frontend=None,
+ **kwargs,
+ ):
+ &quot;&quot;&quot;Inference prepare.
+
+ Args:
+ data_in: Input data (audio samples, file paths, or text).
+ data_lengths: Lengths of each input sample in the batch.
+ key: Sample identifiers.
+ tokenizer: Tokenizer instance for text encoding/decoding.
+ frontend: Audio frontend for feature extraction.
+ **kwargs: Additional keyword arguments.
+ &quot;&quot;&quot;
+ meta_data = {}
+
+ if len(data_in) &gt; 1:
+ raise NotImplementedError(&quot;batch decoding is not implemented&quot;)
+
+ contents = self.data_template(data_in[0])
+ output = self.data_load_speech(contents, tokenizer, frontend, meta_data=meta_data, **kwargs)
+ batch = to_device(output, kwargs[&quot;device&quot;])
+
+ # audio encoder
+ speech = batch[&quot;speech&quot;]
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/fun_asr_nano/model.py#L533" target="_blank" class="src-more">View full source on GitHub →</a></details></div>
+<div class="api-detail" id="e112"><span class="dtag badge-m">method</span><h2><code>FunASRNano.get_prompt(hotwords, language, itn)</code></h2><div class="dmod">funasr.models.fun_asr_nano.FunASRNano · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/fun_asr_nano/model.py#L632" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Get prompt.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>hotwords</code> — TODO.</li>
+<li><code>language</code> — Language identifier.</li>
+<li><code>itn</code> — TODO.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def get_prompt(self, hotwords: list[str], language: str = None, itn: bool = True):
+ &quot;&quot;&quot;Get prompt.
+
+ Args:
+ hotwords: TODO.
+ language: Language identifier.
+ itn: TODO.
+ &quot;&quot;&quot;
+ if len(hotwords) &gt; 0:
+ hotwords = &quot;, &quot;.join(hotwords)
+ prompt = f&quot;请结合上下文信息,更加准确地完成语音转写任务。如果没有相关信息,我们会留空。\n\n\n**上下文信息:**\n\n\n&quot;
+ prompt += f&quot;热词列表:[{hotwords}]\n&quot;
+ else:
+ prompt = &quot;&quot;
+ if language is None:
+ prompt += &quot;语音转写&quot;
+ else:
+ prompt += f&quot;语音转写成{language}&quot;
+ if not itn:
+ prompt += &quot;,不进行文本规整&quot;
+ return prompt + &quot;:&quot;
+</code></pre></details></div>
+<div class="api-detail" id="e113"><span class="dtag badge-m">method</span><h2><code>FunASRNano.generate_chatml(prompt, data)</code></h2><div class="dmod">funasr.models.fun_asr_nano.FunASRNano · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/fun_asr_nano/model.py#L654" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Generate chatml.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>prompt</code> — TODO.</li>
+<li><code>data</code> — TODO.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def generate_chatml(self, prompt: str, data: Union[str, torch.Tensor]):
+ &quot;&quot;&quot;Generate chatml.
+
+ Args:
+ prompt: TODO.
+ data: TODO.
+ &quot;&quot;&quot;
+ if isinstance(data, str):
+ return [
+ {&quot;role&quot;: &quot;system&quot;, &quot;content&quot;: &quot;You are a helpful assistant.&quot;},
+ {&quot;role&quot;: &quot;user&quot;, &quot;content&quot;: f&quot;{prompt}&lt;|startofspeech|&gt;!{data}&lt;|endofspeech|&gt;&quot;},
+ {&quot;role&quot;: &quot;assistant&quot;, &quot;content&quot;: &quot;null&quot;},
+ ]
+ elif isinstance(data, torch.Tensor):
+ return [
+ {&quot;role&quot;: &quot;system&quot;, &quot;content&quot;: &quot;You are a helpful assistant.&quot;},
+ {
+ &quot;role&quot;: &quot;user&quot;,
+ &quot;content&quot;: f&quot;{prompt}&lt;|startofspeech|&gt;!!&lt;|endofspeech|&gt;&quot;,
+ &quot;audio&quot;: data,
+ },
+ {&quot;role&quot;: &quot;assistant&quot;, &quot;content&quot;: &quot;null&quot;},
+ ]
+</code></pre></details></div>
+<div class="api-detail" id="e114"><span class="dtag badge-m">method</span><h2><code>FunASRNano.inference(data_in, data_lengths, key, tokenizer, frontend, **kwargs)</code></h2><div class="dmod">funasr.models.fun_asr_nano.FunASRNano · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/fun_asr_nano/model.py#L678" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Run inference on input data.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>data_in</code> — Input data (audio samples, file paths, or text).</li>
+<li><code>data_lengths</code> — Lengths of each input sample in the batch.</li>
+<li><code>key</code> — Sample identifiers.</li>
+<li><code>tokenizer</code> — Tokenizer instance for text encoding/decoding.</li>
+<li><code>frontend</code> — Audio frontend for feature extraction.</li>
+<li><code>**kwargs</code> — Additional keyword arguments.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def inference(
+ self,
+ data_in,
+ data_lengths=None,
+ key: list = None,
+ tokenizer=None,
+ frontend=None,
+ **kwargs,
+ ):
+ &quot;&quot;&quot;Run inference on input data.
+
+ Args:
+ data_in: Input data (audio samples, file paths, or text).
+ data_lengths: Lengths of each input sample in the batch.
+ key: Sample identifiers.
+ tokenizer: Tokenizer instance for text encoding/decoding.
+ frontend: Audio frontend for feature extraction.
+ **kwargs: Additional keyword arguments.
+ &quot;&quot;&quot;
+ prompt = self.get_prompt(
+ kwargs.get(&quot;hotwords&quot;, []), kwargs.get(&quot;language&quot;, None), kwargs.get(&quot;itn&quot;, True)
+ )
+ data_in = [self.generate_chatml(prompt, data) for data in data_in]
+
+ if key is None:
+ key = []
+ for _ in data_in:
+ chars = string.ascii_letters + string.digits
+ key.append(&quot;rand_key_&quot; + &quot;&quot;.join(random.choice(chars) for _ in range(13)))
+
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/fun_asr_nano/model.py#L678" target="_blank" class="src-more">View full source on GitHub →</a></details></div>
+<div class="api-detail" id="e115"><span class="dtag badge-m">method</span><h2><code>FunASRNano.inference_llm(data_in, data_lengths, key, tokenizer, frontend, **kwargs)</code></h2><div class="dmod">funasr.models.fun_asr_nano.FunASRNano · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/fun_asr_nano/model.py#L717" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Inference llm.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>data_in</code> — Input data (audio samples, file paths, or text).</li>
+<li><code>data_lengths</code> — Lengths of each input sample in the batch.</li>
+<li><code>key</code> — Sample identifiers.</li>
+<li><code>tokenizer</code> — Tokenizer instance for text encoding/decoding.</li>
+<li><code>frontend</code> — Audio frontend for feature extraction.</li>
+<li><code>**kwargs</code> — Additional keyword arguments.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def inference_llm(
+ self,
+ data_in,
+ data_lengths=None,
+ key: list = None,
+ tokenizer=None,
+ frontend=None,
+ **kwargs,
+ ):
+ &quot;&quot;&quot;Inference llm.
+
+ Args:
+ data_in: Input data (audio samples, file paths, or text).
+ data_lengths: Lengths of each input sample in the batch.
+ key: Sample identifiers.
+ tokenizer: Tokenizer instance for text encoding/decoding.
+ frontend: Audio frontend for feature extraction.
+ **kwargs: Additional keyword arguments.
+ &quot;&quot;&quot;
+ inputs_embeds, contents, batch, source_ids, meta_data = self.inference_prepare(
+ data_in, data_lengths, key, tokenizer, frontend, **kwargs
+ )
+
+ ctc_results = []
+ if self.ctc_decoder is not None:
+ encoder_out = meta_data[&quot;encoder_out&quot;]
+ encoder_out_lens = meta_data[&quot;encoder_out_lens&quot;]
+ decoder_out, decoder_out_lens = self.ctc_decoder(encoder_out, encoder_out_lens)
+ ctc_logits = self.ctc.log_softmax(decoder_out)
+
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/fun_asr_nano/model.py#L717" target="_blank" class="src-more">View full source on GitHub →</a></details></div>
+<div class="api-detail" id="e116"><span class="dtag badge-m">method</span><h2><code>FunASRNano.from_pretrained(model, **kwargs)</code></h2><div class="dmod">funasr.models.fun_asr_nano.FunASRNano · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/fun_asr_nano/model.py#L856" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>From pretrained.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>model</code> — Model instance or model name.</li>
+<li><code>**kwargs</code> — Additional keyword arguments.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def from_pretrained(model: str = None, **kwargs):
+ &quot;&quot;&quot;From pretrained.
+
+ Args:
+ model: Model instance or model name.
+ **kwargs: Additional keyword arguments.
+ &quot;&quot;&quot;
+ from funasr import AutoModel
+
+ model, kwargs = AutoModel.build_model(model=model, trust_remote_code=True, **kwargs)
+
+ return model, kwargs
+</code></pre></details></div>
+<div class="api-detail" id="e117"><span class="dtag badge-c">class</span><h2>GLMASR</h2><div class="dmod">funasr.models.glm_asr · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/glm_asr/model.py#L14" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p class="muted">No documentation yet.</p></div><details class="src-block"><summary>📄 Source code</summary><pre><code>class GLMASR(nn.Module):
+
+ def __init__(self, **kwargs):
+ &quot;&quot;&quot;Initialize GLMASR.
+
+ Args:
+ **kwargs: Additional keyword arguments.
+ &quot;&quot;&quot;
+ super().__init__()
+ model_path = kwargs.get(&quot;model_path&quot;, kwargs.get(&quot;model&quot;, &quot;zai-org/GLM-ASR-Nano-2512&quot;))
+ device = kwargs.get(&quot;device&quot;, &quot;cuda:0&quot;)
+ dtype = kwargs.get(&quot;dtype&quot;, &quot;bf16&quot;)
+ hub = kwargs.get(&quot;hub&quot;, &quot;ms&quot;)
+ self._max_new_tokens = kwargs.get(&quot;max_new_tokens&quot;, 512)
+
+ self._dtype_map = {&quot;bf16&quot;: torch.bfloat16, &quot;fp16&quot;: torch.float16, &quot;fp32&quot;: torch.float32}
+ self._device = device
+ self._torch_dtype = self._dtype_map.get(dtype, torch.bfloat16)
+ self._placeholder = nn.Parameter(torch.empty(0))
+
+ model_path = self._resolve_model_path(model_path, hub, kwargs)
+ self.model_path = model_path
+
+ from transformers import AutoModel as HFAutoModel
+ from transformers import AutoProcessor
+
+ self.processor = AutoProcessor.from_pretrained(model_path, trust_remote_code=True)
+ self.glm_model = HFAutoModel.from_pretrained(
+ model_path,
+ dtype=self._torch_dtype,
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/glm_asr/model.py#L14" target="_blank" class="src-more">View full source on GitHub →</a></details><h3 class="mt">Methods</h3><div class="mblk"><code>.forward(**kwargs)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/glm_asr/model.py#L73" target="_blank" class="gh-link">L73</a><div class="mdoc"><p>Forward pass for training.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>**kwargs</code> — Additional keyword arguments.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def forward(self, **kwargs):
+ &quot;&quot;&quot;Forward pass for training.
+
+ Args:
+ **kwargs: Additional keyword arguments.
+ &quot;&quot;&quot;
+ raise NotImplementedError(&quot;GLMASR only supports inference mode&quot;)
+</code></pre></details></div><div class="mblk"><code>.inference(data_in, data_lengths, key, tokenizer, frontend, **kwargs)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/glm_asr/model.py#L81" target="_blank" class="gh-link">L81</a><div class="mdoc"><p>Run inference on input data.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>data_in</code> — Input data (audio samples, file paths, or text).</li>
+<li><code>data_lengths</code> — Lengths of each input sample in the batch.</li>
+<li><code>key</code> — Sample identifiers.</li>
+<li><code>tokenizer</code> — Tokenizer instance for text encoding/decoding.</li>
+<li><code>frontend</code> — Audio frontend for feature extraction.</li>
+<li><code>**kwargs</code> — Additional keyword arguments.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def inference(
+ self,
+ data_in,
+ data_lengths=None,
+ key: list = None,
+ tokenizer=None,
+ frontend=None,
+ **kwargs,
+ ):
+ &quot;&quot;&quot;Run inference on input data.
+
+ Args:
+ data_in: Input data (audio samples, file paths, or text).
+ data_lengths: Lengths of each input sample in the batch.
+ key: Sample identifiers.
+ tokenizer: Tokenizer instance for text encoding/decoding.
+ frontend: Audio frontend for feature extraction.
+ **kwargs: Additional keyword arguments.
+ &quot;&quot;&quot;
+ meta_data = {}
+ time1 = time.perf_counter()
+
+ prompt = kwargs.get(&quot;prompt&quot;, &quot;Please transcribe this audio into text&quot;)
+
+ if isinstance(data_in, (list, tuple)):
+ audio_list = list(data_in)
+ elif isinstance(data_in, str):
+ audio_list = [data_in]
+ else:
+ audio_list = [data_in]
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/glm_asr/model.py#L81" target="_blank" class="src-more">View full source →</a></details></div></div>
+<div class="api-detail" id="e118"><span class="dtag badge-m">method</span><h2><code>GLMASR.forward(**kwargs)</code></h2><div class="dmod">funasr.models.glm_asr.GLMASR · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/glm_asr/model.py#L73" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Forward pass for training.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>**kwargs</code> — Additional keyword arguments.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def forward(self, **kwargs):
+ &quot;&quot;&quot;Forward pass for training.
+
+ Args:
+ **kwargs: Additional keyword arguments.
+ &quot;&quot;&quot;
+ raise NotImplementedError(&quot;GLMASR only supports inference mode&quot;)
+</code></pre></details></div>
+<div class="api-detail" id="e119"><span class="dtag badge-m">method</span><h2><code>GLMASR.inference(data_in, data_lengths, key, tokenizer, frontend, **kwargs)</code></h2><div class="dmod">funasr.models.glm_asr.GLMASR · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/glm_asr/model.py#L81" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Run inference on input data.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>data_in</code> — Input data (audio samples, file paths, or text).</li>
+<li><code>data_lengths</code> — Lengths of each input sample in the batch.</li>
+<li><code>key</code> — Sample identifiers.</li>
+<li><code>tokenizer</code> — Tokenizer instance for text encoding/decoding.</li>
+<li><code>frontend</code> — Audio frontend for feature extraction.</li>
+<li><code>**kwargs</code> — Additional keyword arguments.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def inference(
+ self,
+ data_in,
+ data_lengths=None,
+ key: list = None,
+ tokenizer=None,
+ frontend=None,
+ **kwargs,
+ ):
+ &quot;&quot;&quot;Run inference on input data.
+
+ Args:
+ data_in: Input data (audio samples, file paths, or text).
+ data_lengths: Lengths of each input sample in the batch.
+ key: Sample identifiers.
+ tokenizer: Tokenizer instance for text encoding/decoding.
+ frontend: Audio frontend for feature extraction.
+ **kwargs: Additional keyword arguments.
+ &quot;&quot;&quot;
+ meta_data = {}
+ time1 = time.perf_counter()
+
+ prompt = kwargs.get(&quot;prompt&quot;, &quot;Please transcribe this audio into text&quot;)
+
+ if isinstance(data_in, (list, tuple)):
+ audio_list = list(data_in)
+ elif isinstance(data_in, str):
+ audio_list = [data_in]
+ else:
+ audio_list = [data_in]
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/glm_asr/model.py#L81" target="_blank" class="src-more">View full source on GitHub →</a></details></div>
+<div class="api-detail" id="e120"><span class="dtag badge-c">class</span><h2>LCBNet</h2><div class="dmod">funasr.models.lcbnet · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/lcbnet/model.py#L28" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><ul>
+<li><code>LCBNet</code> — Lightweight Convolutional Block Network for ASR.</li>
+</ul>
+<p>Efficient model design using depthwise separable convolutions</p>
+<p>for low-resource deployment scenarios.</p>
+<p>Inherits Paraformer pipeline.</p>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code>class LCBNet(nn.Module):
+ &quot;&quot;&quot;LCBNet: Lightweight Convolutional Block Network for ASR.
+
+ Efficient model design using depthwise separable convolutions
+ for low-resource deployment scenarios.
+
+ Inherits Paraformer pipeline.
+ &quot;&quot;&quot;
+
+ def __init__(
+ self,
+ specaug: str = None,
+ specaug_conf: dict = None,
+ normalize: str = None,
+ normalize_conf: dict = None,
+ encoder: str = None,
+ encoder_conf: dict = None,
+ decoder: str = None,
+ decoder_conf: dict = None,
+ text_encoder: str = None,
+ text_encoder_conf: dict = None,
+ bias_predictor: str = None,
+ bias_predictor_conf: dict = None,
+ fusion_encoder: str = None,
+ fusion_encoder_conf: dict = None,
+ ctc: str = None,
+ ctc_conf: dict = None,
+ ctc_weight: float = 0.5,
+ interctc_weight: float = 0.0,
+ select_num: int = 2,
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/lcbnet/model.py#L28" target="_blank" class="src-more">View full source on GitHub →</a></details><h3 class="mt">Methods</h3><div class="mblk"><code>.forward(speech, speech_lengths, text, text_lengths, **kwargs)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/lcbnet/model.py#L208" target="_blank" class="gh-link">L208</a><div class="mdoc"><p>Encoder + Decoder + Calc loss</p>
+<h4>Args:</h4>
+<ul>
+<li><code>speech</code> — (Batch, Length, ...)</li>
+<li><code>speech_lengths</code> — (Batch, )</li>
+<li><code>text</code> — (Batch, Length)</li>
+<li><code>text_lengths</code> — (Batch,)</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def forward(
+ self,
+ speech: torch.Tensor,
+ speech_lengths: torch.Tensor,
+ text: torch.Tensor,
+ text_lengths: torch.Tensor,
+ **kwargs,
+ ) -&gt; Tuple[torch.Tensor, Dict[str, torch.Tensor], torch.Tensor]:
+ &quot;&quot;&quot;Encoder + Decoder + Calc loss
+ Args:
+ speech: (Batch, Length, ...)
+ speech_lengths: (Batch, )
+ text: (Batch, Length)
+ text_lengths: (Batch,)
+ &quot;&quot;&quot;
+
+ if len(text_lengths.size()) &gt; 1:
+ text_lengths = text_lengths[:, 0]
+ if len(speech_lengths.size()) &gt; 1:
+ speech_lengths = speech_lengths[:, 0]
+
+ batch_size = speech.shape[0]
+
+ # 1. Encoder
+ encoder_out, encoder_out_lens = self.encode(speech, speech_lengths)
+ intermediate_outs = None
+ if isinstance(encoder_out, tuple):
+ intermediate_outs = encoder_out[1]
+ encoder_out = encoder_out[0]
+
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/lcbnet/model.py#L208" target="_blank" class="src-more">View full source →</a></details></div><div class="mblk"><code>.encode(speech, speech_lengths, **kwargs)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/lcbnet/model.py#L302" target="_blank" class="gh-link">L302</a><div class="mdoc"><p>Frontend + Encoder. Note that this method is used by asr_inference.py</p>
+<h4>Args:</h4>
+<ul>
+<li><code>speech</code> — (Batch, Length, ...)</li>
+<li><code>speech_lengths</code> — (Batch, )</li>
+<li><code>ind</code> — int</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def encode(
+ self,
+ speech: torch.Tensor,
+ speech_lengths: torch.Tensor,
+ **kwargs,
+ ) -&gt; Tuple[torch.Tensor, torch.Tensor]:
+ &quot;&quot;&quot;Frontend + Encoder. Note that this method is used by asr_inference.py
+ Args:
+ speech: (Batch, Length, ...)
+ speech_lengths: (Batch, )
+ ind: int
+ &quot;&quot;&quot;
+ with autocast(False):
+ # Data augmentation
+ if self.specaug is not None and self.training:
+ speech, speech_lengths = self.specaug(speech, speech_lengths)
+ # Normalization for feature: e.g. Global-CMVN, Utterance-CMVN
+ if self.normalize is not None:
+ speech, speech_lengths = self.normalize(speech, speech_lengths)
+ # Forward encoder
+ # feats: (Batch, Length, Dim)
+ # -&gt; encoder_out: (Batch, Length2, Dim2)
+ if self.encoder.interctc_use_conditioning:
+ encoder_out, encoder_out_lens, _ = self.encoder(speech, speech_lengths, ctc=self.ctc)
+ else:
+ encoder_out, encoder_out_lens, _ = self.encoder(speech, speech_lengths)
+ intermediate_outs = None
+ if isinstance(encoder_out, tuple):
+ intermediate_outs = encoder_out[1]
+ encoder_out = encoder_out[0]
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/lcbnet/model.py#L302" target="_blank" class="src-more">View full source →</a></details></div><div class="mblk"><code>.init_beam_search(**kwargs)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/lcbnet/model.py#L400" target="_blank" class="gh-link">L400</a><div class="mdoc"><p>Init beam search.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>**kwargs</code> — Additional keyword arguments.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def init_beam_search(
+ self,
+ **kwargs,
+ ):
+ &quot;&quot;&quot;Init beam search.
+
+ Args:
+ **kwargs: Additional keyword arguments.
+ &quot;&quot;&quot;
+ from funasr.models.transformer.search import BeamSearch
+ from funasr.models.transformer.scorers.ctc import CTCPrefixScorer
+ from funasr.models.transformer.scorers.length_bonus import LengthBonus
+
+ # 1. Build ASR model
+ scorers = {}
+
+ if self.ctc != None:
+ ctc = CTCPrefixScorer(ctc=self.ctc, eos=self.eos)
+ scorers.update(ctc=ctc)
+ token_list = kwargs.get(&quot;token_list&quot;)
+ scorers.update(
+ decoder=self.decoder,
+ length_bonus=LengthBonus(len(token_list)),
+ )
+
+ # 3. Build ngram model
+ # ngram is not supported now
+ ngram = None
+ scorers[&quot;ngram&quot;] = ngram
+
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/lcbnet/model.py#L400" target="_blank" class="src-more">View full source →</a></details></div><div class="mblk"><code>.inference(data_in, data_lengths, key, tokenizer, frontend, **kwargs)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/lcbnet/model.py#L450" target="_blank" class="gh-link">L450</a><div class="mdoc"><p>Run inference on input data.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>data_in</code> — Input data (audio samples, file paths, or text).</li>
+<li><code>data_lengths</code> — Lengths of each input sample in the batch.</li>
+<li><code>key</code> — Sample identifiers.</li>
+<li><code>tokenizer</code> — Tokenizer instance for text encoding/decoding.</li>
+<li><code>frontend</code> — Audio frontend for feature extraction.</li>
+<li><code>**kwargs</code> — Additional keyword arguments.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def inference(
+ self,
+ data_in,
+ data_lengths=None,
+ key: list = None,
+ tokenizer=None,
+ frontend=None,
+ **kwargs,
+ ):
+
+ &quot;&quot;&quot;Run inference on input data.
+
+ Args:
+ data_in: Input data (audio samples, file paths, or text).
+ data_lengths: Lengths of each input sample in the batch.
+ key: Sample identifiers.
+ tokenizer: Tokenizer instance for text encoding/decoding.
+ frontend: Audio frontend for feature extraction.
+ **kwargs: Additional keyword arguments.
+ &quot;&quot;&quot;
+ if kwargs.get(&quot;batch_size&quot;, 1) &gt; 1:
+ raise NotImplementedError(&quot;batch decoding is not implemented&quot;)
+
+ # init beamsearch
+ if self.beam_search is None:
+ logging.info(&quot;enable beam_search&quot;)
+ self.init_beam_search(**kwargs)
+ self.nbest = kwargs.get(&quot;nbest&quot;, 1)
+
+ meta_data = {}
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/lcbnet/model.py#L450" target="_blank" class="src-more">View full source →</a></details></div></div>
+<div class="api-detail" id="e121"><span class="dtag badge-m">method</span><h2><code>LCBNet.forward(speech, speech_lengths, text, text_lengths, **kwargs)</code></h2><div class="dmod">funasr.models.lcbnet.LCBNet · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/lcbnet/model.py#L208" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Encoder + Decoder + Calc loss</p>
+<h4>Args:</h4>
+<ul>
+<li><code>speech</code> — (Batch, Length, ...)</li>
+<li><code>speech_lengths</code> — (Batch, )</li>
+<li><code>text</code> — (Batch, Length)</li>
+<li><code>text_lengths</code> — (Batch,)</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def forward(
+ self,
+ speech: torch.Tensor,
+ speech_lengths: torch.Tensor,
+ text: torch.Tensor,
+ text_lengths: torch.Tensor,
+ **kwargs,
+ ) -&gt; Tuple[torch.Tensor, Dict[str, torch.Tensor], torch.Tensor]:
+ &quot;&quot;&quot;Encoder + Decoder + Calc loss
+ Args:
+ speech: (Batch, Length, ...)
+ speech_lengths: (Batch, )
+ text: (Batch, Length)
+ text_lengths: (Batch,)
+ &quot;&quot;&quot;
+
+ if len(text_lengths.size()) &gt; 1:
+ text_lengths = text_lengths[:, 0]
+ if len(speech_lengths.size()) &gt; 1:
+ speech_lengths = speech_lengths[:, 0]
+
+ batch_size = speech.shape[0]
+
+ # 1. Encoder
+ encoder_out, encoder_out_lens = self.encode(speech, speech_lengths)
+ intermediate_outs = None
+ if isinstance(encoder_out, tuple):
+ intermediate_outs = encoder_out[1]
+ encoder_out = encoder_out[0]
+
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/lcbnet/model.py#L208" target="_blank" class="src-more">View full source on GitHub →</a></details></div>
+<div class="api-detail" id="e122"><span class="dtag badge-m">method</span><h2><code>LCBNet.encode(speech, speech_lengths, **kwargs)</code></h2><div class="dmod">funasr.models.lcbnet.LCBNet · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/lcbnet/model.py#L302" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Frontend + Encoder. Note that this method is used by asr_inference.py</p>
+<h4>Args:</h4>
+<ul>
+<li><code>speech</code> — (Batch, Length, ...)</li>
+<li><code>speech_lengths</code> — (Batch, )</li>
+<li><code>ind</code> — int</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def encode(
+ self,
+ speech: torch.Tensor,
+ speech_lengths: torch.Tensor,
+ **kwargs,
+ ) -&gt; Tuple[torch.Tensor, torch.Tensor]:
+ &quot;&quot;&quot;Frontend + Encoder. Note that this method is used by asr_inference.py
+ Args:
+ speech: (Batch, Length, ...)
+ speech_lengths: (Batch, )
+ ind: int
+ &quot;&quot;&quot;
+ with autocast(False):
+ # Data augmentation
+ if self.specaug is not None and self.training:
+ speech, speech_lengths = self.specaug(speech, speech_lengths)
+ # Normalization for feature: e.g. Global-CMVN, Utterance-CMVN
+ if self.normalize is not None:
+ speech, speech_lengths = self.normalize(speech, speech_lengths)
+ # Forward encoder
+ # feats: (Batch, Length, Dim)
+ # -&gt; encoder_out: (Batch, Length2, Dim2)
+ if self.encoder.interctc_use_conditioning:
+ encoder_out, encoder_out_lens, _ = self.encoder(speech, speech_lengths, ctc=self.ctc)
+ else:
+ encoder_out, encoder_out_lens, _ = self.encoder(speech, speech_lengths)
+ intermediate_outs = None
+ if isinstance(encoder_out, tuple):
+ intermediate_outs = encoder_out[1]
+ encoder_out = encoder_out[0]
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/lcbnet/model.py#L302" target="_blank" class="src-more">View full source on GitHub →</a></details></div>
+<div class="api-detail" id="e123"><span class="dtag badge-m">method</span><h2><code>LCBNet.init_beam_search(**kwargs)</code></h2><div class="dmod">funasr.models.lcbnet.LCBNet · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/lcbnet/model.py#L400" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Init beam search.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>**kwargs</code> — Additional keyword arguments.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def init_beam_search(
+ self,
+ **kwargs,
+ ):
+ &quot;&quot;&quot;Init beam search.
+
+ Args:
+ **kwargs: Additional keyword arguments.
+ &quot;&quot;&quot;
+ from funasr.models.transformer.search import BeamSearch
+ from funasr.models.transformer.scorers.ctc import CTCPrefixScorer
+ from funasr.models.transformer.scorers.length_bonus import LengthBonus
+
+ # 1. Build ASR model
+ scorers = {}
+
+ if self.ctc != None:
+ ctc = CTCPrefixScorer(ctc=self.ctc, eos=self.eos)
+ scorers.update(ctc=ctc)
+ token_list = kwargs.get(&quot;token_list&quot;)
+ scorers.update(
+ decoder=self.decoder,
+ length_bonus=LengthBonus(len(token_list)),
+ )
+
+ # 3. Build ngram model
+ # ngram is not supported now
+ ngram = None
+ scorers[&quot;ngram&quot;] = ngram
+
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/lcbnet/model.py#L400" target="_blank" class="src-more">View full source on GitHub →</a></details></div>
+<div class="api-detail" id="e124"><span class="dtag badge-m">method</span><h2><code>LCBNet.inference(data_in, data_lengths, key, tokenizer, frontend, **kwargs)</code></h2><div class="dmod">funasr.models.lcbnet.LCBNet · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/lcbnet/model.py#L450" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Run inference on input data.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>data_in</code> — Input data (audio samples, file paths, or text).</li>
+<li><code>data_lengths</code> — Lengths of each input sample in the batch.</li>
+<li><code>key</code> — Sample identifiers.</li>
+<li><code>tokenizer</code> — Tokenizer instance for text encoding/decoding.</li>
+<li><code>frontend</code> — Audio frontend for feature extraction.</li>
+<li><code>**kwargs</code> — Additional keyword arguments.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def inference(
+ self,
+ data_in,
+ data_lengths=None,
+ key: list = None,
+ tokenizer=None,
+ frontend=None,
+ **kwargs,
+ ):
+
+ &quot;&quot;&quot;Run inference on input data.
+
+ Args:
+ data_in: Input data (audio samples, file paths, or text).
+ data_lengths: Lengths of each input sample in the batch.
+ key: Sample identifiers.
+ tokenizer: Tokenizer instance for text encoding/decoding.
+ frontend: Audio frontend for feature extraction.
+ **kwargs: Additional keyword arguments.
+ &quot;&quot;&quot;
+ if kwargs.get(&quot;batch_size&quot;, 1) &gt; 1:
+ raise NotImplementedError(&quot;batch decoding is not implemented&quot;)
+
+ # init beamsearch
+ if self.beam_search is None:
+ logging.info(&quot;enable beam_search&quot;)
+ self.init_beam_search(**kwargs)
+ self.nbest = kwargs.get(&quot;nbest&quot;, 1)
+
+ meta_data = {}
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/lcbnet/model.py#L450" target="_blank" class="src-more">View full source on GitHub →</a></details></div>
+<div class="api-detail" id="e125"><span class="dtag badge-c">class</span><h2>LLMASR</h2><div class="dmod">funasr.models.llm_asr · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/llm_asr/model.py#L28" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><ul>
+<li><code>LLM</code> — ASR: Large Language Model based Speech Recognition.</li>
+</ul>
+<p>Combines audio encoder with LLM decoder for speech-to-text.</p>
+<ul>
+<li><code>Output</code> — {&quot;key&quot;: str, &quot;text&quot;: str}</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code>class LLMASR(nn.Module):
+ &quot;&quot;&quot;LLM-ASR: Large Language Model based Speech Recognition.
+
+ Combines audio encoder with LLM decoder for speech-to-text.
+ Output: {&quot;key&quot;: str, &quot;text&quot;: str}
+ &quot;&quot;&quot;
+
+ def __init__(
+ self,
+ specaug: str = None,
+ specaug_conf: dict = None,
+ normalize: str = None,
+ normalize_conf: dict = None,
+ audio_encoder: str = None,
+ audio_encoder_conf: dict = None,
+ audio_adaptor: str = None,
+ audio_adaptor_conf: dict = None,
+ decoder: str = None,
+ decoder_conf: dict = None,
+ ctc: str = None,
+ ctc_conf: dict = None,
+ ctc_weight: float = 0.5,
+ llm: str = None,
+ llm_conf: dict = None,
+ input_size: int = 80,
+ vocab_size: int = -1,
+ ignore_id: int = -1,
+ blank_id: int = 0,
+ sos: int = 1,
+ eos: int = 2,
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/llm_asr/model.py#L28" target="_blank" class="src-more">View full source on GitHub →</a></details><h3 class="mt">Methods</h3><div class="mblk"><code>.forward(speech, speech_lengths, text, text_lengths, input_ids, attention_mask, labels_ids, label_mask, audio_mask, **kwargs)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/llm_asr/model.py#L188" target="_blank" class="gh-link">L188</a><div class="mdoc"><p>Encoder + Decoder + Calc loss</p>
+<h4>Args:</h4>
+<ul>
+<li><code>speech</code> — (Batch, Length, ...)</li>
+<li><code>speech_lengths</code> — (Batch, )</li>
+<li><code>text</code> — (Batch, Length)</li>
+<li><code>text_lengths</code> — (Batch,)</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def forward(
+ self,
+ speech: torch.Tensor,
+ speech_lengths: torch.Tensor,
+ text: torch.Tensor,
+ text_lengths: torch.Tensor,
+ input_ids: torch.Tensor,
+ attention_mask: torch.Tensor,
+ labels_ids: torch.Tensor,
+ label_mask: torch.Tensor,
+ audio_mask: torch.Tensor,
+ **kwargs,
+ ) -&gt; Tuple[torch.Tensor, Dict[str, torch.Tensor], torch.Tensor]:
+ &quot;&quot;&quot;Encoder + Decoder + Calc loss
+ Args:
+ speech: (Batch, Length, ...)
+ speech_lengths: (Batch, )
+ text: (Batch, Length)
+ text_lengths: (Batch,)
+ &quot;&quot;&quot;
+ if len(text_lengths.size()) &gt; 1:
+ text_lengths = text_lengths[:, 0]
+ if len(speech_lengths.size()) &gt; 1:
+ speech_lengths = speech_lengths[:, 0]
+
+ batch_size = speech.shape[0]
+
+ # audio encoder
+ encoder_out, encoder_out_lens = self.encode(speech, speech_lengths)
+
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/llm_asr/model.py#L188" target="_blank" class="src-more">View full source →</a></details></div><div class="mblk"><code>.encode(speech, speech_lengths, **kwargs)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/llm_asr/model.py#L258" target="_blank" class="gh-link">L258</a><div class="mdoc"><p>Encode.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>speech</code> — Speech audio tensor, shape (batch, time).</li>
+<li><code>speech_lengths</code> — Length of each speech sample.</li>
+<li><code>**kwargs</code> — Additional keyword arguments.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def encode(
+ self,
+ speech: torch.Tensor,
+ speech_lengths: torch.Tensor,
+ **kwargs,
+ ):
+ &quot;&quot;&quot;Encode.
+
+ Args:
+ speech: Speech audio tensor, shape (batch, time).
+ speech_lengths: Length of each speech sample.
+ **kwargs: Additional keyword arguments.
+ &quot;&quot;&quot;
+ speech = speech.permute(0, 2, 1)
+ res = self.audio_encoder(speech)
+ if isinstance(res, (list, tuple)):
+ encoder_out, encoder_out_lens = res[0], res[1]
+ else:
+ encoder_out, encoder_out_lens = res, speech_lengths
+ return encoder_out, encoder_out_lens
+</code></pre></details></div><div class="mblk"><code>.inference(data_in, data_lengths, key, tokenizer, frontend, **kwargs)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/llm_asr/model.py#L279" target="_blank" class="gh-link">L279</a><div class="mdoc"><p>Run inference on input data.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>data_in</code> — Input data (audio samples, file paths, or text).</li>
+<li><code>data_lengths</code> — Lengths of each input sample in the batch.</li>
+<li><code>key</code> — Sample identifiers.</li>
+<li><code>tokenizer</code> — Tokenizer instance for text encoding/decoding.</li>
+<li><code>frontend</code> — Audio frontend for feature extraction.</li>
+<li><code>**kwargs</code> — Additional keyword arguments.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def inference(
+ self,
+ data_in,
+ data_lengths=None,
+ key: list = None,
+ tokenizer=None,
+ frontend=None,
+ **kwargs,
+ ):
+
+ &quot;&quot;&quot;Run inference on input data.
+
+ Args:
+ data_in: Input data (audio samples, file paths, or text).
+ data_lengths: Lengths of each input sample in the batch.
+ key: Sample identifiers.
+ tokenizer: Tokenizer instance for text encoding/decoding.
+ frontend: Audio frontend for feature extraction.
+ **kwargs: Additional keyword arguments.
+ &quot;&quot;&quot;
+ prompt = kwargs.get(&quot;prompt&quot;, &quot;Transcribe speech to text.&quot;)
+
+ if kwargs.get(&quot;batch_size&quot;, 1) &gt; 1:
+ raise NotImplementedError(&quot;batch decoding is not implemented&quot;)
+
+ meta_data = {}
+ if (
+ isinstance(data_in, torch.Tensor) and kwargs.get(&quot;data_type&quot;, &quot;sound&quot;) == &quot;fbank&quot;
+ ): # fbank
+ speech, speech_lengths = data_in, data_lengths
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/llm_asr/model.py#L279" target="_blank" class="src-more">View full source →</a></details></div></div>
+<div class="api-detail" id="e126"><span class="dtag badge-m">method</span><h2><code>LLMASR.forward(speech, speech_lengths, text, text_lengths, input_ids, attention_mask, labels_ids, label_mask, audio_mask, **kwargs)</code></h2><div class="dmod">funasr.models.llm_asr.LLMASR · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/llm_asr/model.py#L188" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Encoder + Decoder + Calc loss</p>
+<h4>Args:</h4>
+<ul>
+<li><code>speech</code> — (Batch, Length, ...)</li>
+<li><code>speech_lengths</code> — (Batch, )</li>
+<li><code>text</code> — (Batch, Length)</li>
+<li><code>text_lengths</code> — (Batch,)</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def forward(
+ self,
+ speech: torch.Tensor,
+ speech_lengths: torch.Tensor,
+ text: torch.Tensor,
+ text_lengths: torch.Tensor,
+ input_ids: torch.Tensor,
+ attention_mask: torch.Tensor,
+ labels_ids: torch.Tensor,
+ label_mask: torch.Tensor,
+ audio_mask: torch.Tensor,
+ **kwargs,
+ ) -&gt; Tuple[torch.Tensor, Dict[str, torch.Tensor], torch.Tensor]:
+ &quot;&quot;&quot;Encoder + Decoder + Calc loss
+ Args:
+ speech: (Batch, Length, ...)
+ speech_lengths: (Batch, )
+ text: (Batch, Length)
+ text_lengths: (Batch,)
+ &quot;&quot;&quot;
+ if len(text_lengths.size()) &gt; 1:
+ text_lengths = text_lengths[:, 0]
+ if len(speech_lengths.size()) &gt; 1:
+ speech_lengths = speech_lengths[:, 0]
+
+ batch_size = speech.shape[0]
+
+ # audio encoder
+ encoder_out, encoder_out_lens = self.encode(speech, speech_lengths)
+
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/llm_asr/model.py#L188" target="_blank" class="src-more">View full source on GitHub →</a></details></div>
+<div class="api-detail" id="e127"><span class="dtag badge-m">method</span><h2><code>LLMASR.encode(speech, speech_lengths, **kwargs)</code></h2><div class="dmod">funasr.models.llm_asr.LLMASR · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/llm_asr/model.py#L258" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Encode.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>speech</code> — Speech audio tensor, shape (batch, time).</li>
+<li><code>speech_lengths</code> — Length of each speech sample.</li>
+<li><code>**kwargs</code> — Additional keyword arguments.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def encode(
+ self,
+ speech: torch.Tensor,
+ speech_lengths: torch.Tensor,
+ **kwargs,
+ ):
+ &quot;&quot;&quot;Encode.
+
+ Args:
+ speech: Speech audio tensor, shape (batch, time).
+ speech_lengths: Length of each speech sample.
+ **kwargs: Additional keyword arguments.
+ &quot;&quot;&quot;
+ speech = speech.permute(0, 2, 1)
+ res = self.audio_encoder(speech)
+ if isinstance(res, (list, tuple)):
+ encoder_out, encoder_out_lens = res[0], res[1]
+ else:
+ encoder_out, encoder_out_lens = res, speech_lengths
+ return encoder_out, encoder_out_lens
+</code></pre></details></div>
+<div class="api-detail" id="e128"><span class="dtag badge-m">method</span><h2><code>LLMASR.inference(data_in, data_lengths, key, tokenizer, frontend, **kwargs)</code></h2><div class="dmod">funasr.models.llm_asr.LLMASR · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/llm_asr/model.py#L279" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Run inference on input data.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>data_in</code> — Input data (audio samples, file paths, or text).</li>
+<li><code>data_lengths</code> — Lengths of each input sample in the batch.</li>
+<li><code>key</code> — Sample identifiers.</li>
+<li><code>tokenizer</code> — Tokenizer instance for text encoding/decoding.</li>
+<li><code>frontend</code> — Audio frontend for feature extraction.</li>
+<li><code>**kwargs</code> — Additional keyword arguments.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def inference(
+ self,
+ data_in,
+ data_lengths=None,
+ key: list = None,
+ tokenizer=None,
+ frontend=None,
+ **kwargs,
+ ):
+
+ &quot;&quot;&quot;Run inference on input data.
+
+ Args:
+ data_in: Input data (audio samples, file paths, or text).
+ data_lengths: Lengths of each input sample in the batch.
+ key: Sample identifiers.
+ tokenizer: Tokenizer instance for text encoding/decoding.
+ frontend: Audio frontend for feature extraction.
+ **kwargs: Additional keyword arguments.
+ &quot;&quot;&quot;
+ prompt = kwargs.get(&quot;prompt&quot;, &quot;Transcribe speech to text.&quot;)
+
+ if kwargs.get(&quot;batch_size&quot;, 1) &gt; 1:
+ raise NotImplementedError(&quot;batch decoding is not implemented&quot;)
+
+ meta_data = {}
+ if (
+ isinstance(data_in, torch.Tensor) and kwargs.get(&quot;data_type&quot;, &quot;sound&quot;) == &quot;fbank&quot;
+ ): # fbank
+ speech, speech_lengths = data_in, data_lengths
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/llm_asr/model.py#L279" target="_blank" class="src-more">View full source on GitHub →</a></details></div>
+<div class="api-detail" id="e129"><span class="dtag badge-c">class</span><h2>LLMASR2</h2><div class="dmod">funasr.models.llm_asr · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/llm_asr/model.py#L403" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p class="muted">No documentation yet.</p></div><details class="src-block"><summary>📄 Source code</summary><pre><code>class LLMASR2(nn.Module):
+ &quot;&quot;&quot; &quot;&quot;&quot;
+
+ def __init__(
+ self,
+ specaug: str = None,
+ specaug_conf: dict = None,
+ normalize: str = None,
+ normalize_conf: dict = None,
+ audio_encoder: str = None,
+ audio_encoder_conf: dict = None,
+ audio_adaptor: str = None,
+ audio_adaptor_conf: dict = None,
+ decoder: str = None,
+ decoder_conf: dict = None,
+ ctc: str = None,
+ ctc_conf: dict = None,
+ ctc_weight: float = 0.5,
+ llm: str = None,
+ llm_conf: dict = None,
+ input_size: int = 80,
+ vocab_size: int = -1,
+ ignore_id: int = -1,
+ blank_id: int = 0,
+ sos: int = 1,
+ eos: int = 2,
+ lsm_weight: float = 0.0,
+ length_normalized_loss: bool = False,
+ report_cer: bool = True,
+ report_wer: bool = True,
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/llm_asr/model.py#L403" target="_blank" class="src-more">View full source on GitHub →</a></details><h3 class="mt">Methods</h3><div class="mblk"><code>.forward(speech, speech_lengths, input_ids, attention_mask, labels_ids, fbank_beg, fbank_mask, **kwargs)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/llm_asr/model.py#L561" target="_blank" class="gh-link">L561</a><div class="mdoc"><p>Encoder + Decoder + Calc loss</p>
+<h4>Args:</h4>
+<ul>
+<li><code>speech</code> — (Batch, Length, ...)</li>
+<li><code>speech_lengths</code> — (Batch, )</li>
+<li><code>text</code> — (Batch, Length)</li>
+<li><code>text_lengths</code> — (Batch,)</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def forward(
+ self,
+ speech: torch.Tensor,
+ speech_lengths: torch.Tensor,
+ input_ids: torch.Tensor,
+ attention_mask: torch.Tensor,
+ labels_ids: torch.Tensor,
+ fbank_beg: torch.Tensor,
+ fbank_mask: torch.Tensor,
+ **kwargs,
+ ) -&gt; Tuple[torch.Tensor, Dict[str, torch.Tensor], torch.Tensor]:
+ &quot;&quot;&quot;Encoder + Decoder + Calc loss
+ Args:
+ speech: (Batch, Length, ...)
+ speech_lengths: (Batch, )
+ text: (Batch, Length)
+ text_lengths: (Batch,)
+ &quot;&quot;&quot;
+ # import pdb;
+ # pdb.set_trace()
+ if len(speech_lengths.size()) &gt; 1:
+ speech_lengths = speech_lengths[:, 0]
+
+ batch_size, frames, _ = speech.shape
+
+ with torch.cuda.amp.autocast(enabled=False):
+ # audio encoder
+ encoder_out, encoder_out_lens = self.encode(speech, speech_lengths)
+
+ # audio_adaptor
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/llm_asr/model.py#L561" target="_blank" class="src-more">View full source →</a></details></div><div class="mblk"><code>.encode(speech, speech_lengths)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/llm_asr/model.py#L654" target="_blank" class="gh-link">L654</a><div class="mdoc"><p>Encode.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>speech</code> — Speech audio tensor, shape (batch, time).</li>
+<li><code>speech_lengths</code> — Length of each speech sample.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def encode(self, speech, speech_lengths):
+ # audio encoder
+ &quot;&quot;&quot;Encode.
+
+ Args:
+ speech: Speech audio tensor, shape (batch, time).
+ speech_lengths: Length of each speech sample.
+ &quot;&quot;&quot;
+ encoder_out, encoder_out_lens = self.audio_encoder(speech.permute(0, 2, 1), speech_lengths)
+
+ return encoder_out, encoder_out_lens
+</code></pre></details></div><div class="mblk"><code>.data_template(data)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/llm_asr/model.py#L666" target="_blank" class="gh-link">L666</a><div class="mdoc"><p>Data template.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>data</code> — TODO.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def data_template(self, data):
+ &quot;&quot;&quot;Data template.
+
+ Args:
+ data: TODO.
+ &quot;&quot;&quot;
+ system, user, assistant = [], [], []
+ for i, item in enumerate(data):
+ role = item[&quot;role&quot;]
+ content = item[&quot;content&quot;]
+ if role == &quot;system&quot;:
+ system.append(content)
+ elif role == &quot;user&quot;:
+ user.append(content)
+ elif role == &quot;assistant&quot;:
+ assistant.append(content)
+
+ system = system * len(user)
+
+ contents = {
+ &quot;system&quot;: system,
+ &quot;user&quot;: user,
+ &quot;assistant&quot;: assistant,
+ }
+
+ return contents
+</code></pre></details></div><div class="mblk"><code>.data_load_speech(contents, tokenizer, frontend, meta_data, **kwargs)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/llm_asr/model.py#L693" target="_blank" class="gh-link">L693</a><div class="mdoc"><p>Data load speech.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>contents</code> — TODO.</li>
+<li><code>tokenizer</code> — Tokenizer instance for text encoding/decoding.</li>
+<li><code>frontend</code> — Audio frontend for feature extraction.</li>
+<li><code>meta_data</code> — TODO.</li>
+<li><code>**kwargs</code> — Additional keyword arguments.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def data_load_speech(self, contents: dict, tokenizer, frontend, meta_data={}, **kwargs):
+
+ &quot;&quot;&quot;Data load speech.
+
+ Args:
+ contents: TODO.
+ tokenizer: Tokenizer instance for text encoding/decoding.
+ frontend: Audio frontend for feature extraction.
+ meta_data: TODO.
+ **kwargs: Additional keyword arguments.
+ &quot;&quot;&quot;
+ system = contents[&quot;system&quot;]
+ user = contents[&quot;user&quot;]
+ assistant = contents[&quot;assistant&quot;]
+ pattern = re.compile(r&quot;(&lt;\|startofspeech\|&gt;.*?&lt;\|endofspeech\|&gt;)&quot;)
+ input_ids, labels, source_ids, target_ids, fbank, fbank_lens, fbank_mask, fbank_beg = (
+ [],
+ [],
+ [],
+ [],
+ [],
+ [],
+ [],
+ [],
+ )
+
+ for i, (system_prompt, user_prompt, target_out) in enumerate(zip(system, user, assistant)):
+
+ source_input = f&quot;&lt;|im_start|&gt;system\n{system_prompt}&lt;|im_end|&gt;\n&lt;|im_start|&gt;user\n{user_prompt}&lt;|im_end|&gt;\n&lt;|im_start|&gt;assistant\n&quot;
+
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/llm_asr/model.py#L693" target="_blank" class="src-more">View full source →</a></details></div><div class="mblk"><code>.inference(data_in, data_lengths, key, tokenizer, frontend, **kwargs)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/llm_asr/model.py#L820" target="_blank" class="gh-link">L820</a><div class="mdoc"><p>Run inference on input data.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>data_in</code> — Input data (audio samples, file paths, or text).</li>
+<li><code>data_lengths</code> — Lengths of each input sample in the batch.</li>
+<li><code>key</code> — Sample identifiers.</li>
+<li><code>tokenizer</code> — Tokenizer instance for text encoding/decoding.</li>
+<li><code>frontend</code> — Audio frontend for feature extraction.</li>
+<li><code>**kwargs</code> — Additional keyword arguments.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def inference(
+ self,
+ data_in,
+ data_lengths=None,
+ key: list = None,
+ tokenizer=None,
+ frontend=None,
+ **kwargs,
+ ):
+
+ &quot;&quot;&quot;Run inference on input data.
+
+ Args:
+ data_in: Input data (audio samples, file paths, or text).
+ data_lengths: Lengths of each input sample in the batch.
+ key: Sample identifiers.
+ tokenizer: Tokenizer instance for text encoding/decoding.
+ frontend: Audio frontend for feature extraction.
+ **kwargs: Additional keyword arguments.
+ &quot;&quot;&quot;
+ meta_data = {}
+ prompt = kwargs.get(&quot;prompt&quot;, None)
+
+ if kwargs.get(&quot;batch_size&quot;, 1) &gt; 1:
+ raise NotImplementedError(&quot;batch decoding is not implemented&quot;)
+
+ contents = self.data_template(data_in[0])
+ output = self.data_load_speech(contents, tokenizer, frontend, meta_data=meta_data, **kwargs)
+ batch = to_device(output, kwargs[&quot;device&quot;])
+
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/llm_asr/model.py#L820" target="_blank" class="src-more">View full source →</a></details></div></div>
+<div class="api-detail" id="e130"><span class="dtag badge-m">method</span><h2><code>LLMASR2.forward(speech, speech_lengths, input_ids, attention_mask, labels_ids, fbank_beg, fbank_mask, **kwargs)</code></h2><div class="dmod">funasr.models.llm_asr.LLMASR2 · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/llm_asr/model.py#L561" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Encoder + Decoder + Calc loss</p>
+<h4>Args:</h4>
+<ul>
+<li><code>speech</code> — (Batch, Length, ...)</li>
+<li><code>speech_lengths</code> — (Batch, )</li>
+<li><code>text</code> — (Batch, Length)</li>
+<li><code>text_lengths</code> — (Batch,)</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def forward(
+ self,
+ speech: torch.Tensor,
+ speech_lengths: torch.Tensor,
+ input_ids: torch.Tensor,
+ attention_mask: torch.Tensor,
+ labels_ids: torch.Tensor,
+ fbank_beg: torch.Tensor,
+ fbank_mask: torch.Tensor,
+ **kwargs,
+ ) -&gt; Tuple[torch.Tensor, Dict[str, torch.Tensor], torch.Tensor]:
+ &quot;&quot;&quot;Encoder + Decoder + Calc loss
+ Args:
+ speech: (Batch, Length, ...)
+ speech_lengths: (Batch, )
+ text: (Batch, Length)
+ text_lengths: (Batch,)
+ &quot;&quot;&quot;
+ # import pdb;
+ # pdb.set_trace()
+ if len(speech_lengths.size()) &gt; 1:
+ speech_lengths = speech_lengths[:, 0]
+
+ batch_size, frames, _ = speech.shape
+
+ with torch.cuda.amp.autocast(enabled=False):
+ # audio encoder
+ encoder_out, encoder_out_lens = self.encode(speech, speech_lengths)
+
+ # audio_adaptor
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/llm_asr/model.py#L561" target="_blank" class="src-more">View full source on GitHub →</a></details></div>
+<div class="api-detail" id="e131"><span class="dtag badge-m">method</span><h2><code>LLMASR2.encode(speech, speech_lengths)</code></h2><div class="dmod">funasr.models.llm_asr.LLMASR2 · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/llm_asr/model.py#L654" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Encode.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>speech</code> — Speech audio tensor, shape (batch, time).</li>
+<li><code>speech_lengths</code> — Length of each speech sample.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def encode(self, speech, speech_lengths):
+ # audio encoder
+ &quot;&quot;&quot;Encode.
+
+ Args:
+ speech: Speech audio tensor, shape (batch, time).
+ speech_lengths: Length of each speech sample.
+ &quot;&quot;&quot;
+ encoder_out, encoder_out_lens = self.audio_encoder(speech.permute(0, 2, 1), speech_lengths)
+
+ return encoder_out, encoder_out_lens
+</code></pre></details></div>
+<div class="api-detail" id="e132"><span class="dtag badge-m">method</span><h2><code>LLMASR2.data_template(data)</code></h2><div class="dmod">funasr.models.llm_asr.LLMASR2 · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/llm_asr/model.py#L666" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Data template.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>data</code> — TODO.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def data_template(self, data):
+ &quot;&quot;&quot;Data template.
+
+ Args:
+ data: TODO.
+ &quot;&quot;&quot;
+ system, user, assistant = [], [], []
+ for i, item in enumerate(data):
+ role = item[&quot;role&quot;]
+ content = item[&quot;content&quot;]
+ if role == &quot;system&quot;:
+ system.append(content)
+ elif role == &quot;user&quot;:
+ user.append(content)
+ elif role == &quot;assistant&quot;:
+ assistant.append(content)
+
+ system = system * len(user)
+
+ contents = {
+ &quot;system&quot;: system,
+ &quot;user&quot;: user,
+ &quot;assistant&quot;: assistant,
+ }
+
+ return contents
+</code></pre></details></div>
+<div class="api-detail" id="e133"><span class="dtag badge-m">method</span><h2><code>LLMASR2.data_load_speech(contents, tokenizer, frontend, meta_data, **kwargs)</code></h2><div class="dmod">funasr.models.llm_asr.LLMASR2 · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/llm_asr/model.py#L693" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Data load speech.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>contents</code> — TODO.</li>
+<li><code>tokenizer</code> — Tokenizer instance for text encoding/decoding.</li>
+<li><code>frontend</code> — Audio frontend for feature extraction.</li>
+<li><code>meta_data</code> — TODO.</li>
+<li><code>**kwargs</code> — Additional keyword arguments.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def data_load_speech(self, contents: dict, tokenizer, frontend, meta_data={}, **kwargs):
+
+ &quot;&quot;&quot;Data load speech.
+
+ Args:
+ contents: TODO.
+ tokenizer: Tokenizer instance for text encoding/decoding.
+ frontend: Audio frontend for feature extraction.
+ meta_data: TODO.
+ **kwargs: Additional keyword arguments.
+ &quot;&quot;&quot;
+ system = contents[&quot;system&quot;]
+ user = contents[&quot;user&quot;]
+ assistant = contents[&quot;assistant&quot;]
+ pattern = re.compile(r&quot;(&lt;\|startofspeech\|&gt;.*?&lt;\|endofspeech\|&gt;)&quot;)
+ input_ids, labels, source_ids, target_ids, fbank, fbank_lens, fbank_mask, fbank_beg = (
+ [],
+ [],
+ [],
+ [],
+ [],
+ [],
+ [],
+ [],
+ )
+
+ for i, (system_prompt, user_prompt, target_out) in enumerate(zip(system, user, assistant)):
+
+ source_input = f&quot;&lt;|im_start|&gt;system\n{system_prompt}&lt;|im_end|&gt;\n&lt;|im_start|&gt;user\n{user_prompt}&lt;|im_end|&gt;\n&lt;|im_start|&gt;assistant\n&quot;
+
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/llm_asr/model.py#L693" target="_blank" class="src-more">View full source on GitHub →</a></details></div>
+<div class="api-detail" id="e134"><span class="dtag badge-m">method</span><h2><code>LLMASR2.inference(data_in, data_lengths, key, tokenizer, frontend, **kwargs)</code></h2><div class="dmod">funasr.models.llm_asr.LLMASR2 · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/llm_asr/model.py#L820" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Run inference on input data.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>data_in</code> — Input data (audio samples, file paths, or text).</li>
+<li><code>data_lengths</code> — Lengths of each input sample in the batch.</li>
+<li><code>key</code> — Sample identifiers.</li>
+<li><code>tokenizer</code> — Tokenizer instance for text encoding/decoding.</li>
+<li><code>frontend</code> — Audio frontend for feature extraction.</li>
+<li><code>**kwargs</code> — Additional keyword arguments.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def inference(
+ self,
+ data_in,
+ data_lengths=None,
+ key: list = None,
+ tokenizer=None,
+ frontend=None,
+ **kwargs,
+ ):
+
+ &quot;&quot;&quot;Run inference on input data.
+
+ Args:
+ data_in: Input data (audio samples, file paths, or text).
+ data_lengths: Lengths of each input sample in the batch.
+ key: Sample identifiers.
+ tokenizer: Tokenizer instance for text encoding/decoding.
+ frontend: Audio frontend for feature extraction.
+ **kwargs: Additional keyword arguments.
+ &quot;&quot;&quot;
+ meta_data = {}
+ prompt = kwargs.get(&quot;prompt&quot;, None)
+
+ if kwargs.get(&quot;batch_size&quot;, 1) &gt; 1:
+ raise NotImplementedError(&quot;batch decoding is not implemented&quot;)
+
+ contents = self.data_template(data_in[0])
+ output = self.data_load_speech(contents, tokenizer, frontend, meta_data=meta_data, **kwargs)
+ batch = to_device(output, kwargs[&quot;device&quot;])
+
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/llm_asr/model.py#L820" target="_blank" class="src-more">View full source on GitHub →</a></details></div>
+<div class="api-detail" id="e135"><span class="dtag badge-c">class</span><h2>LLMASR3</h2><div class="dmod">funasr.models.llm_asr · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/llm_asr/model.py#L947" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p class="muted">No documentation yet.</p></div><details class="src-block"><summary>📄 Source code</summary><pre><code>class LLMASR3(LLMASR2):
+ &quot;&quot;&quot; &quot;&quot;&quot;
+
+ def __init__(
+ self,
+ *args,
+ **kwargs,
+ ):
+
+ &quot;&quot;&quot;Initialize LLMASR3.
+
+ Args:
+ *args: Variable positional arguments.
+ **kwargs: Additional keyword arguments.
+ &quot;&quot;&quot;
+ super().__init__(*args, **kwargs)
+
+ def encode(self, speech, speech_lengths):
+ # audio encoder
+ &quot;&quot;&quot;Encode.
+
+ Args:
+ speech: Speech audio tensor, shape (batch, time).
+ speech_lengths: Length of each speech sample.
+ &quot;&quot;&quot;
+ encoder_out, encoder_out_lens = self.audio_encoder(speech, speech_lengths)
+ return encoder_out, encoder_out_lens
+</code></pre></details><h3 class="mt">Methods</h3><div class="mblk"><code>.encode(speech, speech_lengths)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/llm_asr/model.py#L964" target="_blank" class="gh-link">L964</a><div class="mdoc"><p>Encode.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>speech</code> — Speech audio tensor, shape (batch, time).</li>
+<li><code>speech_lengths</code> — Length of each speech sample.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def encode(self, speech, speech_lengths):
+ # audio encoder
+ &quot;&quot;&quot;Encode.
+
+ Args:
+ speech: Speech audio tensor, shape (batch, time).
+ speech_lengths: Length of each speech sample.
+ &quot;&quot;&quot;
+ encoder_out, encoder_out_lens = self.audio_encoder(speech, speech_lengths)
+ return encoder_out, encoder_out_lens
+</code></pre></details></div></div>
+<div class="api-detail" id="e136"><span class="dtag badge-m">method</span><h2><code>LLMASR3.encode(speech, speech_lengths)</code></h2><div class="dmod">funasr.models.llm_asr.LLMASR3 · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/llm_asr/model.py#L964" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Encode.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>speech</code> — Speech audio tensor, shape (batch, time).</li>
+<li><code>speech_lengths</code> — Length of each speech sample.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def encode(self, speech, speech_lengths):
+ # audio encoder
+ &quot;&quot;&quot;Encode.
+
+ Args:
+ speech: Speech audio tensor, shape (batch, time).
+ speech_lengths: Length of each speech sample.
+ &quot;&quot;&quot;
+ encoder_out, encoder_out_lens = self.audio_encoder(speech, speech_lengths)
+ return encoder_out, encoder_out_lens
+</code></pre></details></div>
+<div class="api-detail" id="e137"><span class="dtag badge-c">class</span><h2>LLMASR4</h2><div class="dmod">funasr.models.llm_asr · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/llm_asr/model.py#L977" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p class="muted">No documentation yet.</p></div><details class="src-block"><summary>📄 Source code</summary><pre><code>class LLMASR4(nn.Module):
+ &quot;&quot;&quot; &quot;&quot;&quot;
+
+ def __init__(
+ self,
+ specaug: str = None,
+ specaug_conf: dict = None,
+ normalize: str = None,
+ normalize_conf: dict = None,
+ audio_encoder: str = None,
+ audio_encoder_conf: dict = None,
+ audio_adaptor: str = None,
+ audio_adaptor_conf: dict = None,
+ decoder: str = None,
+ decoder_conf: dict = None,
+ ctc: str = None,
+ ctc_conf: dict = None,
+ ctc_weight: float = 0.5,
+ llm: str = None,
+ llm_conf: dict = None,
+ input_size: int = 80,
+ vocab_size: int = -1,
+ ignore_id: int = -1,
+ blank_id: int = 0,
+ sos: int = 1,
+ eos: int = 2,
+ lsm_weight: float = 0.0,
+ length_normalized_loss: bool = False,
+ report_cer: bool = True,
+ report_wer: bool = True,
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/llm_asr/model.py#L977" target="_blank" class="src-more">View full source on GitHub →</a></details><h3 class="mt">Methods</h3><div class="mblk"><code>.forward(speech, speech_lengths, input_ids, attention_mask, labels_ids, fbank_beg, fbank_mask, **kwargs)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/llm_asr/model.py#L1135" target="_blank" class="gh-link">L1135</a><div class="mdoc"><p>Encoder + Decoder + Calc loss</p>
+<h4>Args:</h4>
+<ul>
+<li><code>speech</code> — (Batch, Length, ...)</li>
+<li><code>speech_lengths</code> — (Batch, )</li>
+<li><code>text</code> — (Batch, Length)</li>
+<li><code>text_lengths</code> — (Batch,)</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def forward(
+ self,
+ speech: torch.Tensor,
+ speech_lengths: torch.Tensor,
+ input_ids: torch.Tensor,
+ attention_mask: torch.Tensor,
+ labels_ids: torch.Tensor,
+ fbank_beg: torch.Tensor,
+ fbank_mask: torch.Tensor,
+ **kwargs,
+ ) -&gt; Tuple[torch.Tensor, Dict[str, torch.Tensor], torch.Tensor]:
+ &quot;&quot;&quot;Encoder + Decoder + Calc loss
+ Args:
+ speech: (Batch, Length, ...)
+ speech_lengths: (Batch, )
+ text: (Batch, Length)
+ text_lengths: (Batch,)
+ &quot;&quot;&quot;
+ # import pdb
+ #
+ # pdb.set_trace()
+ if len(speech_lengths.size()) &gt; 1:
+ speech_lengths = speech_lengths[:, 0]
+
+ batch_size_speech, frames, _ = speech.shape
+ batch_size, token_num = input_ids.shape
+
+ with torch.cuda.amp.autocast(enabled=False):
+ # audio encoder
+ encoder_out, encoder_out_lens = self.encode(speech, speech_lengths)
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/llm_asr/model.py#L1135" target="_blank" class="src-more">View full source →</a></details></div><div class="mblk"><code>.encode(speech, speech_lengths)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/llm_asr/model.py#L1246" target="_blank" class="gh-link">L1246</a><div class="mdoc"><p>Encode.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>speech</code> — Speech audio tensor, shape (batch, time).</li>
+<li><code>speech_lengths</code> — Length of each speech sample.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def encode(self, speech, speech_lengths):
+ # audio encoder
+ &quot;&quot;&quot;Encode.
+
+ Args:
+ speech: Speech audio tensor, shape (batch, time).
+ speech_lengths: Length of each speech sample.
+ &quot;&quot;&quot;
+ encoder_out, encoder_out_lens = self.audio_encoder(speech.permute(0, 2, 1), speech_lengths)
+
+ return encoder_out, encoder_out_lens
+</code></pre></details></div><div class="mblk"><code>.data_template(data)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/llm_asr/model.py#L1258" target="_blank" class="gh-link">L1258</a><div class="mdoc"><p>Data template.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>data</code> — TODO.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def data_template(self, data):
+ &quot;&quot;&quot;Data template.
+
+ Args:
+ data: TODO.
+ &quot;&quot;&quot;
+ system, user, assistant = [], [], []
+ for i, item in enumerate(data):
+ role = item[&quot;role&quot;]
+ content = item[&quot;content&quot;]
+ if role == &quot;system&quot;:
+ system.append(content)
+ elif role == &quot;user&quot;:
+ user.append(content)
+ elif role == &quot;assistant&quot;:
+ assistant.append(content)
+
+ system = system * len(user)
+
+ contents = {
+ &quot;system&quot;: system,
+ &quot;user&quot;: user,
+ &quot;assistant&quot;: assistant,
+ }
+
+ return contents
+</code></pre></details></div><div class="mblk"><code>.data_load_speech(contents, tokenizer, frontend, meta_data, **kwargs)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/llm_asr/model.py#L1285" target="_blank" class="gh-link">L1285</a><div class="mdoc"><p>Data load speech.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>contents</code> — TODO.</li>
+<li><code>tokenizer</code> — Tokenizer instance for text encoding/decoding.</li>
+<li><code>frontend</code> — Audio frontend for feature extraction.</li>
+<li><code>meta_data</code> — TODO.</li>
+<li><code>**kwargs</code> — Additional keyword arguments.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def data_load_speech(self, contents: dict, tokenizer, frontend, meta_data={}, **kwargs):
+
+ &quot;&quot;&quot;Data load speech.
+
+ Args:
+ contents: TODO.
+ tokenizer: Tokenizer instance for text encoding/decoding.
+ frontend: Audio frontend for feature extraction.
+ meta_data: TODO.
+ **kwargs: Additional keyword arguments.
+ &quot;&quot;&quot;
+ system = contents[&quot;system&quot;]
+ user = contents[&quot;user&quot;]
+ assistant = contents[&quot;assistant&quot;]
+ pattern = re.compile(r&quot;(&lt;\|startofspeech\|&gt;.*?&lt;\|endofspeech\|&gt;)&quot;)
+
+ input_ids, labels, fbank, fbank_lens, fbank_mask, fbank_beg, fake_token_len = (
+ [],
+ [],
+ [],
+ [],
+ [],
+ [],
+ [],
+ )
+ input_source_ids = []
+ for i, (system_prompt, user_prompt, target_out) in enumerate(zip(system, user, assistant)):
+ if i &gt;= kwargs.get(&quot;multiturn_num_max&quot;, 5):
+ break
+ if len(input_ids) &gt; kwargs.get(&quot;max_token_length&quot;, 1500):
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/llm_asr/model.py#L1285" target="_blank" class="src-more">View full source →</a></details></div><div class="mblk"><code>.inference_prepare(data_in, data_lengths, key, tokenizer, frontend, **kwargs)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/llm_asr/model.py#L1433" target="_blank" class="gh-link">L1433</a><div class="mdoc"><p>Inference prepare.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>data_in</code> — Input data (audio samples, file paths, or text).</li>
+<li><code>data_lengths</code> — Lengths of each input sample in the batch.</li>
+<li><code>key</code> — Sample identifiers.</li>
+<li><code>tokenizer</code> — Tokenizer instance for text encoding/decoding.</li>
+<li><code>frontend</code> — Audio frontend for feature extraction.</li>
+<li><code>**kwargs</code> — Additional keyword arguments.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def inference_prepare(
+ self,
+ data_in,
+ data_lengths=None,
+ key: list = None,
+ tokenizer=None,
+ frontend=None,
+ **kwargs,
+ ):
+
+ &quot;&quot;&quot;Inference prepare.
+
+ Args:
+ data_in: Input data (audio samples, file paths, or text).
+ data_lengths: Lengths of each input sample in the batch.
+ key: Sample identifiers.
+ tokenizer: Tokenizer instance for text encoding/decoding.
+ frontend: Audio frontend for feature extraction.
+ **kwargs: Additional keyword arguments.
+ &quot;&quot;&quot;
+ meta_data = {}
+ prompt = kwargs.get(&quot;prompt&quot;, None)
+
+ if kwargs.get(&quot;batch_size&quot;, 1) &gt; 1:
+ raise NotImplementedError(&quot;batch decoding is not implemented&quot;)
+
+ contents = self.data_template(data_in[0])
+ output = self.data_load_speech(contents, tokenizer, frontend, meta_data=meta_data, **kwargs)
+ batch = to_device(output, kwargs[&quot;device&quot;])
+
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/llm_asr/model.py#L1433" target="_blank" class="src-more">View full source →</a></details></div><div class="mblk"><code>.inference(data_in, data_lengths, key, tokenizer, frontend, **kwargs)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/llm_asr/model.py#L1525" target="_blank" class="gh-link">L1525</a><div class="mdoc"><p>Run inference on input data.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>data_in</code> — Input data (audio samples, file paths, or text).</li>
+<li><code>data_lengths</code> — Lengths of each input sample in the batch.</li>
+<li><code>key</code> — Sample identifiers.</li>
+<li><code>tokenizer</code> — Tokenizer instance for text encoding/decoding.</li>
+<li><code>frontend</code> — Audio frontend for feature extraction.</li>
+<li><code>**kwargs</code> — Additional keyword arguments.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def inference(
+ self,
+ data_in,
+ data_lengths=None,
+ key: list = None,
+ tokenizer=None,
+ frontend=None,
+ **kwargs,
+ ):
+
+ &quot;&quot;&quot;Run inference on input data.
+
+ Args:
+ data_in: Input data (audio samples, file paths, or text).
+ data_lengths: Lengths of each input sample in the batch.
+ key: Sample identifiers.
+ tokenizer: Tokenizer instance for text encoding/decoding.
+ frontend: Audio frontend for feature extraction.
+ **kwargs: Additional keyword arguments.
+ &quot;&quot;&quot;
+ inputs_embeds, contents, batch, source_ids, meta_data = self.inference_prepare(
+ data_in, data_lengths, key, tokenizer, frontend, **kwargs
+ )
+
+ llm_dtype = kwargs.get(&quot;llm_dtype&quot;, &quot;fp32&quot;)
+ if llm_dtype == &quot;fp32&quot;:
+ llm_dtype = &quot;fp16&quot; if kwargs.get(&quot;fp16&quot;, False) else llm_dtype
+ llm_dtype = &quot;bf16&quot; if kwargs.get(&quot;bf16&quot;, False) else llm_dtype
+
+ with torch.cuda.amp.autocast(
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/llm_asr/model.py#L1525" target="_blank" class="src-more">View full source →</a></details></div></div>
+<div class="api-detail" id="e138"><span class="dtag badge-m">method</span><h2><code>LLMASR4.forward(speech, speech_lengths, input_ids, attention_mask, labels_ids, fbank_beg, fbank_mask, **kwargs)</code></h2><div class="dmod">funasr.models.llm_asr.LLMASR4 · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/llm_asr/model.py#L1135" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Encoder + Decoder + Calc loss</p>
+<h4>Args:</h4>
+<ul>
+<li><code>speech</code> — (Batch, Length, ...)</li>
+<li><code>speech_lengths</code> — (Batch, )</li>
+<li><code>text</code> — (Batch, Length)</li>
+<li><code>text_lengths</code> — (Batch,)</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def forward(
+ self,
+ speech: torch.Tensor,
+ speech_lengths: torch.Tensor,
+ input_ids: torch.Tensor,
+ attention_mask: torch.Tensor,
+ labels_ids: torch.Tensor,
+ fbank_beg: torch.Tensor,
+ fbank_mask: torch.Tensor,
+ **kwargs,
+ ) -&gt; Tuple[torch.Tensor, Dict[str, torch.Tensor], torch.Tensor]:
+ &quot;&quot;&quot;Encoder + Decoder + Calc loss
+ Args:
+ speech: (Batch, Length, ...)
+ speech_lengths: (Batch, )
+ text: (Batch, Length)
+ text_lengths: (Batch,)
+ &quot;&quot;&quot;
+ # import pdb
+ #
+ # pdb.set_trace()
+ if len(speech_lengths.size()) &gt; 1:
+ speech_lengths = speech_lengths[:, 0]
+
+ batch_size_speech, frames, _ = speech.shape
+ batch_size, token_num = input_ids.shape
+
+ with torch.cuda.amp.autocast(enabled=False):
+ # audio encoder
+ encoder_out, encoder_out_lens = self.encode(speech, speech_lengths)
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/llm_asr/model.py#L1135" target="_blank" class="src-more">View full source on GitHub →</a></details></div>
+<div class="api-detail" id="e139"><span class="dtag badge-m">method</span><h2><code>LLMASR4.encode(speech, speech_lengths)</code></h2><div class="dmod">funasr.models.llm_asr.LLMASR4 · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/llm_asr/model.py#L1246" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Encode.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>speech</code> — Speech audio tensor, shape (batch, time).</li>
+<li><code>speech_lengths</code> — Length of each speech sample.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def encode(self, speech, speech_lengths):
+ # audio encoder
+ &quot;&quot;&quot;Encode.
+
+ Args:
+ speech: Speech audio tensor, shape (batch, time).
+ speech_lengths: Length of each speech sample.
+ &quot;&quot;&quot;
+ encoder_out, encoder_out_lens = self.audio_encoder(speech.permute(0, 2, 1), speech_lengths)
+
+ return encoder_out, encoder_out_lens
+</code></pre></details></div>
+<div class="api-detail" id="e140"><span class="dtag badge-m">method</span><h2><code>LLMASR4.data_template(data)</code></h2><div class="dmod">funasr.models.llm_asr.LLMASR4 · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/llm_asr/model.py#L1258" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Data template.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>data</code> — TODO.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def data_template(self, data):
+ &quot;&quot;&quot;Data template.
+
+ Args:
+ data: TODO.
+ &quot;&quot;&quot;
+ system, user, assistant = [], [], []
+ for i, item in enumerate(data):
+ role = item[&quot;role&quot;]
+ content = item[&quot;content&quot;]
+ if role == &quot;system&quot;:
+ system.append(content)
+ elif role == &quot;user&quot;:
+ user.append(content)
+ elif role == &quot;assistant&quot;:
+ assistant.append(content)
+
+ system = system * len(user)
+
+ contents = {
+ &quot;system&quot;: system,
+ &quot;user&quot;: user,
+ &quot;assistant&quot;: assistant,
+ }
+
+ return contents
+</code></pre></details></div>
+<div class="api-detail" id="e141"><span class="dtag badge-m">method</span><h2><code>LLMASR4.data_load_speech(contents, tokenizer, frontend, meta_data, **kwargs)</code></h2><div class="dmod">funasr.models.llm_asr.LLMASR4 · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/llm_asr/model.py#L1285" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Data load speech.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>contents</code> — TODO.</li>
+<li><code>tokenizer</code> — Tokenizer instance for text encoding/decoding.</li>
+<li><code>frontend</code> — Audio frontend for feature extraction.</li>
+<li><code>meta_data</code> — TODO.</li>
+<li><code>**kwargs</code> — Additional keyword arguments.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def data_load_speech(self, contents: dict, tokenizer, frontend, meta_data={}, **kwargs):
+
+ &quot;&quot;&quot;Data load speech.
+
+ Args:
+ contents: TODO.
+ tokenizer: Tokenizer instance for text encoding/decoding.
+ frontend: Audio frontend for feature extraction.
+ meta_data: TODO.
+ **kwargs: Additional keyword arguments.
+ &quot;&quot;&quot;
+ system = contents[&quot;system&quot;]
+ user = contents[&quot;user&quot;]
+ assistant = contents[&quot;assistant&quot;]
+ pattern = re.compile(r&quot;(&lt;\|startofspeech\|&gt;.*?&lt;\|endofspeech\|&gt;)&quot;)
+
+ input_ids, labels, fbank, fbank_lens, fbank_mask, fbank_beg, fake_token_len = (
+ [],
+ [],
+ [],
+ [],
+ [],
+ [],
+ [],
+ )
+ input_source_ids = []
+ for i, (system_prompt, user_prompt, target_out) in enumerate(zip(system, user, assistant)):
+ if i &gt;= kwargs.get(&quot;multiturn_num_max&quot;, 5):
+ break
+ if len(input_ids) &gt; kwargs.get(&quot;max_token_length&quot;, 1500):
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/llm_asr/model.py#L1285" target="_blank" class="src-more">View full source on GitHub →</a></details></div>
+<div class="api-detail" id="e142"><span class="dtag badge-m">method</span><h2><code>LLMASR4.inference_prepare(data_in, data_lengths, key, tokenizer, frontend, **kwargs)</code></h2><div class="dmod">funasr.models.llm_asr.LLMASR4 · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/llm_asr/model.py#L1433" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Inference prepare.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>data_in</code> — Input data (audio samples, file paths, or text).</li>
+<li><code>data_lengths</code> — Lengths of each input sample in the batch.</li>
+<li><code>key</code> — Sample identifiers.</li>
+<li><code>tokenizer</code> — Tokenizer instance for text encoding/decoding.</li>
+<li><code>frontend</code> — Audio frontend for feature extraction.</li>
+<li><code>**kwargs</code> — Additional keyword arguments.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def inference_prepare(
+ self,
+ data_in,
+ data_lengths=None,
+ key: list = None,
+ tokenizer=None,
+ frontend=None,
+ **kwargs,
+ ):
+
+ &quot;&quot;&quot;Inference prepare.
+
+ Args:
+ data_in: Input data (audio samples, file paths, or text).
+ data_lengths: Lengths of each input sample in the batch.
+ key: Sample identifiers.
+ tokenizer: Tokenizer instance for text encoding/decoding.
+ frontend: Audio frontend for feature extraction.
+ **kwargs: Additional keyword arguments.
+ &quot;&quot;&quot;
+ meta_data = {}
+ prompt = kwargs.get(&quot;prompt&quot;, None)
+
+ if kwargs.get(&quot;batch_size&quot;, 1) &gt; 1:
+ raise NotImplementedError(&quot;batch decoding is not implemented&quot;)
+
+ contents = self.data_template(data_in[0])
+ output = self.data_load_speech(contents, tokenizer, frontend, meta_data=meta_data, **kwargs)
+ batch = to_device(output, kwargs[&quot;device&quot;])
+
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/llm_asr/model.py#L1433" target="_blank" class="src-more">View full source on GitHub →</a></details></div>
+<div class="api-detail" id="e143"><span class="dtag badge-m">method</span><h2><code>LLMASR4.inference(data_in, data_lengths, key, tokenizer, frontend, **kwargs)</code></h2><div class="dmod">funasr.models.llm_asr.LLMASR4 · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/llm_asr/model.py#L1525" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Run inference on input data.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>data_in</code> — Input data (audio samples, file paths, or text).</li>
+<li><code>data_lengths</code> — Lengths of each input sample in the batch.</li>
+<li><code>key</code> — Sample identifiers.</li>
+<li><code>tokenizer</code> — Tokenizer instance for text encoding/decoding.</li>
+<li><code>frontend</code> — Audio frontend for feature extraction.</li>
+<li><code>**kwargs</code> — Additional keyword arguments.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def inference(
+ self,
+ data_in,
+ data_lengths=None,
+ key: list = None,
+ tokenizer=None,
+ frontend=None,
+ **kwargs,
+ ):
+
+ &quot;&quot;&quot;Run inference on input data.
+
+ Args:
+ data_in: Input data (audio samples, file paths, or text).
+ data_lengths: Lengths of each input sample in the batch.
+ key: Sample identifiers.
+ tokenizer: Tokenizer instance for text encoding/decoding.
+ frontend: Audio frontend for feature extraction.
+ **kwargs: Additional keyword arguments.
+ &quot;&quot;&quot;
+ inputs_embeds, contents, batch, source_ids, meta_data = self.inference_prepare(
+ data_in, data_lengths, key, tokenizer, frontend, **kwargs
+ )
+
+ llm_dtype = kwargs.get(&quot;llm_dtype&quot;, &quot;fp32&quot;)
+ if llm_dtype == &quot;fp32&quot;:
+ llm_dtype = &quot;fp16&quot; if kwargs.get(&quot;fp16&quot;, False) else llm_dtype
+ llm_dtype = &quot;bf16&quot; if kwargs.get(&quot;bf16&quot;, False) else llm_dtype
+
+ with torch.cuda.amp.autocast(
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/llm_asr/model.py#L1525" target="_blank" class="src-more">View full source on GitHub →</a></details></div>
+<div class="api-detail" id="e144"><span class="dtag badge-c">class</span><h2>LLMASRNAR</h2><div class="dmod">funasr.models.llm_asr_nar · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/llm_asr_nar/model.py#L26" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p class="muted">No documentation yet.</p></div><details class="src-block"><summary>📄 Source code</summary><pre><code>class LLMASRNAR(nn.Module):
+ &quot;&quot;&quot; &quot;&quot;&quot;
+
+ def __init__(
+ self,
+ specaug: str = None,
+ specaug_conf: dict = None,
+ normalize: str = None,
+ normalize_conf: dict = None,
+ encoder: str = None,
+ encoder_conf: dict = None,
+ decoder: str = None,
+ decoder_conf: dict = None,
+ ctc: str = None,
+ ctc_conf: dict = None,
+ ctc_weight: float = 0.5,
+ llm: str = None,
+ llm_conf: dict = None,
+ adaptor: str = None,
+ adaptor_conf: dict = None,
+ input_size: int = 80,
+ vocab_size: int = -1,
+ ignore_id: int = -1,
+ blank_id: int = 0,
+ sos: int = 1,
+ eos: int = 2,
+ lsm_weight: float = 0.0,
+ length_normalized_loss: bool = False,
+ report_cer: bool = True,
+ report_wer: bool = True,
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/llm_asr_nar/model.py#L26" target="_blank" class="src-more">View full source on GitHub →</a></details><h3 class="mt">Methods</h3><div class="mblk"><code>.forward(speech, speech_lengths, text, text_lengths, input_ids, attention_mask, labels_ids, label_mask, audio_mask, **kwargs)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/llm_asr_nar/model.py#L182" target="_blank" class="gh-link">L182</a><div class="mdoc"><p>Encoder + Decoder + Calc loss</p>
+<h4>Args:</h4>
+<ul>
+<li><code>speech</code> — (Batch, Length, ...)</li>
+<li><code>speech_lengths</code> — (Batch, )</li>
+<li><code>text</code> — (Batch, Length)</li>
+<li><code>text_lengths</code> — (Batch,)</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def forward(
+ self,
+ speech: torch.Tensor,
+ speech_lengths: torch.Tensor,
+ text: torch.Tensor,
+ text_lengths: torch.Tensor,
+ input_ids: torch.Tensor,
+ attention_mask: torch.Tensor,
+ labels_ids: torch.Tensor,
+ label_mask: torch.Tensor,
+ audio_mask: torch.Tensor,
+ **kwargs,
+ ) -&gt; Tuple[torch.Tensor, Dict[str, torch.Tensor], torch.Tensor]:
+ &quot;&quot;&quot;Encoder + Decoder + Calc loss
+ Args:
+ speech: (Batch, Length, ...)
+ speech_lengths: (Batch, )
+ text: (Batch, Length)
+ text_lengths: (Batch,)
+ &quot;&quot;&quot;
+ if len(text_lengths.size()) &gt; 1:
+ text_lengths = text_lengths[:, 0]
+ if len(speech_lengths.size()) &gt; 1:
+ speech_lengths = speech_lengths[:, 0]
+
+ batch_size = speech.shape[0]
+
+ # audio encoder
+ encoder_out, encoder_out_lens = self.encode(speech, speech_lengths, audio_mask=audio_mask)
+
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/llm_asr_nar/model.py#L182" target="_blank" class="src-more">View full source →</a></details></div><div class="mblk"><code>.encode(speech, speech_lengths, **kwargs)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/llm_asr_nar/model.py#L253" target="_blank" class="gh-link">L253</a><div class="mdoc"><p>Encode.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>speech</code> — Speech audio tensor, shape (batch, time).</li>
+<li><code>speech_lengths</code> — Length of each speech sample.</li>
+<li><code>**kwargs</code> — Additional keyword arguments.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def encode(
+ self,
+ speech: torch.Tensor,
+ speech_lengths: torch.Tensor,
+ **kwargs,
+ ):
+
+ &quot;&quot;&quot;Encode.
+
+ Args:
+ speech: Speech audio tensor, shape (batch, time).
+ speech_lengths: Length of each speech sample.
+ **kwargs: Additional keyword arguments.
+ &quot;&quot;&quot;
+ audio_mask = kwargs.get(&quot;audio_mask&quot;, None)
+ audio_token_lengths = audio_mask.sum(-1) if audio_mask is not None else None
+ text_token_int = kwargs.get(&quot;text_token_int&quot;, None)
+ if audio_token_lengths is None:
+ audio_token_lengths = torch.tensor([len(text_token_int)], dtype=torch.int64)
+
+ batch = {&quot;speech&quot;: speech, &quot;speech_lengths&quot;: speech_lengths}
+ enc, enc_lens = self.audio_encoder.encode(**batch)
+ with autocast(False):
+ enc_mask = sequence_mask(enc_lens, enc.size(1), device=enc.device)[:, None, :]
+ pre_acoustic_embeds, pre_token_length, _, _ = self.audio_encoder.predictor(
+ enc,
+ mask=enc_mask,
+ target_label_length=audio_token_lengths,
+ )
+
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/llm_asr_nar/model.py#L253" target="_blank" class="src-more">View full source →</a></details></div><div class="mblk"><code>.inference(data_in, data_lengths, key, tokenizer, frontend, **kwargs)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/llm_asr_nar/model.py#L285" target="_blank" class="gh-link">L285</a><div class="mdoc"><p>Run inference on input data.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>data_in</code> — Input data (audio samples, file paths, or text).</li>
+<li><code>data_lengths</code> — Lengths of each input sample in the batch.</li>
+<li><code>key</code> — Sample identifiers.</li>
+<li><code>tokenizer</code> — Tokenizer instance for text encoding/decoding.</li>
+<li><code>frontend</code> — Audio frontend for feature extraction.</li>
+<li><code>**kwargs</code> — Additional keyword arguments.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def inference(
+ self,
+ data_in,
+ data_lengths=None,
+ key: list = None,
+ tokenizer=None,
+ frontend=None,
+ **kwargs,
+ ):
+
+ &quot;&quot;&quot;Run inference on input data.
+
+ Args:
+ data_in: Input data (audio samples, file paths, or text).
+ data_lengths: Lengths of each input sample in the batch.
+ key: Sample identifiers.
+ tokenizer: Tokenizer instance for text encoding/decoding.
+ frontend: Audio frontend for feature extraction.
+ **kwargs: Additional keyword arguments.
+ &quot;&quot;&quot;
+ prompt = kwargs.get(&quot;prompt&quot;, &quot;Transcribe speech to text.&quot;)
+
+ if kwargs.get(&quot;batch_size&quot;, 1) &gt; 1:
+ raise NotImplementedError(&quot;batch decoding is not implemented&quot;)
+
+ meta_data = {}
+ if (
+ isinstance(data_in, torch.Tensor) and kwargs.get(&quot;data_type&quot;, &quot;sound&quot;) == &quot;fbank&quot;
+ ): # fbank
+ speech, speech_lengths = data_in, data_lengths
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/llm_asr_nar/model.py#L285" target="_blank" class="src-more">View full source →</a></details></div></div>
+<div class="api-detail" id="e145"><span class="dtag badge-m">method</span><h2><code>LLMASRNAR.forward(speech, speech_lengths, text, text_lengths, input_ids, attention_mask, labels_ids, label_mask, audio_mask, **kwargs)</code></h2><div class="dmod">funasr.models.llm_asr_nar.LLMASRNAR · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/llm_asr_nar/model.py#L182" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Encoder + Decoder + Calc loss</p>
+<h4>Args:</h4>
+<ul>
+<li><code>speech</code> — (Batch, Length, ...)</li>
+<li><code>speech_lengths</code> — (Batch, )</li>
+<li><code>text</code> — (Batch, Length)</li>
+<li><code>text_lengths</code> — (Batch,)</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def forward(
+ self,
+ speech: torch.Tensor,
+ speech_lengths: torch.Tensor,
+ text: torch.Tensor,
+ text_lengths: torch.Tensor,
+ input_ids: torch.Tensor,
+ attention_mask: torch.Tensor,
+ labels_ids: torch.Tensor,
+ label_mask: torch.Tensor,
+ audio_mask: torch.Tensor,
+ **kwargs,
+ ) -&gt; Tuple[torch.Tensor, Dict[str, torch.Tensor], torch.Tensor]:
+ &quot;&quot;&quot;Encoder + Decoder + Calc loss
+ Args:
+ speech: (Batch, Length, ...)
+ speech_lengths: (Batch, )
+ text: (Batch, Length)
+ text_lengths: (Batch,)
+ &quot;&quot;&quot;
+ if len(text_lengths.size()) &gt; 1:
+ text_lengths = text_lengths[:, 0]
+ if len(speech_lengths.size()) &gt; 1:
+ speech_lengths = speech_lengths[:, 0]
+
+ batch_size = speech.shape[0]
+
+ # audio encoder
+ encoder_out, encoder_out_lens = self.encode(speech, speech_lengths, audio_mask=audio_mask)
+
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/llm_asr_nar/model.py#L182" target="_blank" class="src-more">View full source on GitHub →</a></details></div>
+<div class="api-detail" id="e146"><span class="dtag badge-m">method</span><h2><code>LLMASRNAR.encode(speech, speech_lengths, **kwargs)</code></h2><div class="dmod">funasr.models.llm_asr_nar.LLMASRNAR · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/llm_asr_nar/model.py#L253" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Encode.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>speech</code> — Speech audio tensor, shape (batch, time).</li>
+<li><code>speech_lengths</code> — Length of each speech sample.</li>
+<li><code>**kwargs</code> — Additional keyword arguments.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def encode(
+ self,
+ speech: torch.Tensor,
+ speech_lengths: torch.Tensor,
+ **kwargs,
+ ):
+
+ &quot;&quot;&quot;Encode.
+
+ Args:
+ speech: Speech audio tensor, shape (batch, time).
+ speech_lengths: Length of each speech sample.
+ **kwargs: Additional keyword arguments.
+ &quot;&quot;&quot;
+ audio_mask = kwargs.get(&quot;audio_mask&quot;, None)
+ audio_token_lengths = audio_mask.sum(-1) if audio_mask is not None else None
+ text_token_int = kwargs.get(&quot;text_token_int&quot;, None)
+ if audio_token_lengths is None:
+ audio_token_lengths = torch.tensor([len(text_token_int)], dtype=torch.int64)
+
+ batch = {&quot;speech&quot;: speech, &quot;speech_lengths&quot;: speech_lengths}
+ enc, enc_lens = self.audio_encoder.encode(**batch)
+ with autocast(False):
+ enc_mask = sequence_mask(enc_lens, enc.size(1), device=enc.device)[:, None, :]
+ pre_acoustic_embeds, pre_token_length, _, _ = self.audio_encoder.predictor(
+ enc,
+ mask=enc_mask,
+ target_label_length=audio_token_lengths,
+ )
+
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/llm_asr_nar/model.py#L253" target="_blank" class="src-more">View full source on GitHub →</a></details></div>
+<div class="api-detail" id="e147"><span class="dtag badge-m">method</span><h2><code>LLMASRNAR.inference(data_in, data_lengths, key, tokenizer, frontend, **kwargs)</code></h2><div class="dmod">funasr.models.llm_asr_nar.LLMASRNAR · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/llm_asr_nar/model.py#L285" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Run inference on input data.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>data_in</code> — Input data (audio samples, file paths, or text).</li>
+<li><code>data_lengths</code> — Lengths of each input sample in the batch.</li>
+<li><code>key</code> — Sample identifiers.</li>
+<li><code>tokenizer</code> — Tokenizer instance for text encoding/decoding.</li>
+<li><code>frontend</code> — Audio frontend for feature extraction.</li>
+<li><code>**kwargs</code> — Additional keyword arguments.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def inference(
+ self,
+ data_in,
+ data_lengths=None,
+ key: list = None,
+ tokenizer=None,
+ frontend=None,
+ **kwargs,
+ ):
+
+ &quot;&quot;&quot;Run inference on input data.
+
+ Args:
+ data_in: Input data (audio samples, file paths, or text).
+ data_lengths: Lengths of each input sample in the batch.
+ key: Sample identifiers.
+ tokenizer: Tokenizer instance for text encoding/decoding.
+ frontend: Audio frontend for feature extraction.
+ **kwargs: Additional keyword arguments.
+ &quot;&quot;&quot;
+ prompt = kwargs.get(&quot;prompt&quot;, &quot;Transcribe speech to text.&quot;)
+
+ if kwargs.get(&quot;batch_size&quot;, 1) &gt; 1:
+ raise NotImplementedError(&quot;batch decoding is not implemented&quot;)
+
+ meta_data = {}
+ if (
+ isinstance(data_in, torch.Tensor) and kwargs.get(&quot;data_type&quot;, &quot;sound&quot;) == &quot;fbank&quot;
+ ): # fbank
+ speech, speech_lengths = data_in, data_lengths
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/llm_asr_nar/model.py#L285" target="_blank" class="src-more">View full source on GitHub →</a></details></div>
+<div class="api-detail" id="e148"><span class="dtag badge-c">class</span><h2>LLMASRNARPrompt</h2><div class="dmod">funasr.models.llm_asr_nar · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/llm_asr_nar/model.py#L421" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p class="muted">No documentation yet.</p></div><details class="src-block"><summary>📄 Source code</summary><pre><code>class LLMASRNARPrompt(nn.Module):
+ &quot;&quot;&quot; &quot;&quot;&quot;
+
+ def __init__(
+ self,
+ specaug: str = None,
+ specaug_conf: dict = None,
+ normalize: str = None,
+ normalize_conf: dict = None,
+ encoder: str = None,
+ encoder_conf: dict = None,
+ decoder: str = None,
+ decoder_conf: dict = None,
+ ctc: str = None,
+ ctc_conf: dict = None,
+ ctc_weight: float = 0.0,
+ llm: str = None,
+ llm_conf: dict = None,
+ adaptor: str = None,
+ adaptor_conf: dict = None,
+ input_size: int = 80,
+ vocab_size: int = -1,
+ ignore_id: int = -1,
+ blank_id: int = 0,
+ sos: int = 1,
+ eos: int = 2,
+ lsm_weight: float = 0.0,
+ length_normalized_loss: bool = False,
+ predictor_weight: int = 1.0,
+ report_cer: bool = True,
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/llm_asr_nar/model.py#L421" target="_blank" class="src-more">View full source on GitHub →</a></details><h3 class="mt">Methods</h3><div class="mblk"><code>.forward(speech, speech_lengths, text, text_lengths, input_ids, attention_mask, labels_ids, label_mask, audio_mask, **kwargs)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/llm_asr_nar/model.py#L588" target="_blank" class="gh-link">L588</a><div class="mdoc"><p>Encoder + Decoder + Calc loss</p>
+<h4>Args:</h4>
+<ul>
+<li><code>speech</code> — (Batch, Length, ...)</li>
+<li><code>speech_lengths</code> — (Batch, )</li>
+<li><code>text</code> — (Batch, Length)</li>
+<li><code>text_lengths</code> — (Batch,)</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def forward(
+ self,
+ speech: torch.Tensor,
+ speech_lengths: torch.Tensor,
+ text: torch.Tensor,
+ text_lengths: torch.Tensor,
+ input_ids: torch.Tensor,
+ attention_mask: torch.Tensor,
+ labels_ids: torch.Tensor,
+ label_mask: torch.Tensor,
+ audio_mask: torch.Tensor,
+ **kwargs,
+ ) -&gt; Tuple[torch.Tensor, Dict[str, torch.Tensor], torch.Tensor]:
+ &quot;&quot;&quot;Encoder + Decoder + Calc loss
+ Args:
+ speech: (Batch, Length, ...)
+ speech_lengths: (Batch, )
+ text: (Batch, Length)
+ text_lengths: (Batch,)
+ &quot;&quot;&quot;
+ # import pdb;
+ # pdb.set_trace()
+ if len(text_lengths.size()) &gt; 1:
+ text_lengths = text_lengths[:, 0]
+ if len(speech_lengths.size()) &gt; 1:
+ speech_lengths = speech_lengths[:, 0]
+
+ batch_size = speech.shape[0]
+
+ stats = {}
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/llm_asr_nar/model.py#L588" target="_blank" class="src-more">View full source →</a></details></div><div class="mblk"><code>.encode(speech, speech_lengths, **kwargs)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/llm_asr_nar/model.py#L691" target="_blank" class="gh-link">L691</a><div class="mdoc"><p>Encode.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>speech</code> — Speech audio tensor, shape (batch, time).</li>
+<li><code>speech_lengths</code> — Length of each speech sample.</li>
+<li><code>**kwargs</code> — Additional keyword arguments.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def encode(
+ self,
+ speech: torch.Tensor,
+ speech_lengths: torch.Tensor,
+ **kwargs,
+ ):
+
+ &quot;&quot;&quot;Encode.
+
+ Args:
+ speech: Speech audio tensor, shape (batch, time).
+ speech_lengths: Length of each speech sample.
+ **kwargs: Additional keyword arguments.
+ &quot;&quot;&quot;
+ audio_mask = kwargs.get(&quot;audio_mask&quot;, None)
+ audio_token_lengths = audio_mask.sum(-1) if audio_mask is not None else None
+ text_token_int = kwargs.get(&quot;text_token_int&quot;, None)
+ if audio_token_lengths is None and text_token_int is not None:
+ audio_token_lengths = torch.tensor([len(text_token_int)], dtype=torch.int64)
+
+ batch = {&quot;speech&quot;: speech, &quot;speech_lengths&quot;: speech_lengths}
+ enc, enc_lens = self.audio_encoder.encode(**batch)
+ with autocast(False):
+ enc_mask = sequence_mask(enc_lens, enc.size(1), device=enc.device)[:, None, :]
+ pre_acoustic_embeds, pre_token_length, _, _ = self.audio_encoder.predictor(
+ enc,
+ mask=enc_mask,
+ target_label_length=audio_token_lengths,
+ )
+ loss_pre = 0.0
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/llm_asr_nar/model.py#L691" target="_blank" class="src-more">View full source →</a></details></div><div class="mblk"><code>.inference(data_in, data_lengths, key, tokenizer, frontend, **kwargs)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/llm_asr_nar/model.py#L753" target="_blank" class="gh-link">L753</a><div class="mdoc"><p>Run inference on input data.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>data_in</code> — Input data (audio samples, file paths, or text).</li>
+<li><code>data_lengths</code> — Lengths of each input sample in the batch.</li>
+<li><code>key</code> — Sample identifiers.</li>
+<li><code>tokenizer</code> — Tokenizer instance for text encoding/decoding.</li>
+<li><code>frontend</code> — Audio frontend for feature extraction.</li>
+<li><code>**kwargs</code> — Additional keyword arguments.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def inference(
+ self,
+ data_in,
+ data_lengths=None,
+ key: list = None,
+ tokenizer=None,
+ frontend=None,
+ **kwargs,
+ ):
+
+ &quot;&quot;&quot;Run inference on input data.
+
+ Args:
+ data_in: Input data (audio samples, file paths, or text).
+ data_lengths: Lengths of each input sample in the batch.
+ key: Sample identifiers.
+ tokenizer: Tokenizer instance for text encoding/decoding.
+ frontend: Audio frontend for feature extraction.
+ **kwargs: Additional keyword arguments.
+ &quot;&quot;&quot;
+ prompt = kwargs.get(&quot;prompt&quot;, &quot;Transcribe speech to text.&quot;)
+
+ if kwargs.get(&quot;batch_size&quot;, 1) &gt; 1:
+ raise NotImplementedError(&quot;batch decoding is not implemented&quot;)
+
+ meta_data = {}
+ if (
+ isinstance(data_in, torch.Tensor) and kwargs.get(&quot;data_type&quot;, &quot;sound&quot;) == &quot;fbank&quot;
+ ): # fbank
+ speech, speech_lengths = data_in, data_lengths
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/llm_asr_nar/model.py#L753" target="_blank" class="src-more">View full source →</a></details></div></div>
+<div class="api-detail" id="e149"><span class="dtag badge-m">method</span><h2><code>LLMASRNARPrompt.forward(speech, speech_lengths, text, text_lengths, input_ids, attention_mask, labels_ids, label_mask, audio_mask, **kwargs)</code></h2><div class="dmod">funasr.models.llm_asr_nar.LLMASRNARPrompt · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/llm_asr_nar/model.py#L588" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Encoder + Decoder + Calc loss</p>
+<h4>Args:</h4>
+<ul>
+<li><code>speech</code> — (Batch, Length, ...)</li>
+<li><code>speech_lengths</code> — (Batch, )</li>
+<li><code>text</code> — (Batch, Length)</li>
+<li><code>text_lengths</code> — (Batch,)</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def forward(
+ self,
+ speech: torch.Tensor,
+ speech_lengths: torch.Tensor,
+ text: torch.Tensor,
+ text_lengths: torch.Tensor,
+ input_ids: torch.Tensor,
+ attention_mask: torch.Tensor,
+ labels_ids: torch.Tensor,
+ label_mask: torch.Tensor,
+ audio_mask: torch.Tensor,
+ **kwargs,
+ ) -&gt; Tuple[torch.Tensor, Dict[str, torch.Tensor], torch.Tensor]:
+ &quot;&quot;&quot;Encoder + Decoder + Calc loss
+ Args:
+ speech: (Batch, Length, ...)
+ speech_lengths: (Batch, )
+ text: (Batch, Length)
+ text_lengths: (Batch,)
+ &quot;&quot;&quot;
+ # import pdb;
+ # pdb.set_trace()
+ if len(text_lengths.size()) &gt; 1:
+ text_lengths = text_lengths[:, 0]
+ if len(speech_lengths.size()) &gt; 1:
+ speech_lengths = speech_lengths[:, 0]
+
+ batch_size = speech.shape[0]
+
+ stats = {}
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/llm_asr_nar/model.py#L588" target="_blank" class="src-more">View full source on GitHub →</a></details></div>
+<div class="api-detail" id="e150"><span class="dtag badge-m">method</span><h2><code>LLMASRNARPrompt.encode(speech, speech_lengths, **kwargs)</code></h2><div class="dmod">funasr.models.llm_asr_nar.LLMASRNARPrompt · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/llm_asr_nar/model.py#L691" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Encode.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>speech</code> — Speech audio tensor, shape (batch, time).</li>
+<li><code>speech_lengths</code> — Length of each speech sample.</li>
+<li><code>**kwargs</code> — Additional keyword arguments.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def encode(
+ self,
+ speech: torch.Tensor,
+ speech_lengths: torch.Tensor,
+ **kwargs,
+ ):
+
+ &quot;&quot;&quot;Encode.
+
+ Args:
+ speech: Speech audio tensor, shape (batch, time).
+ speech_lengths: Length of each speech sample.
+ **kwargs: Additional keyword arguments.
+ &quot;&quot;&quot;
+ audio_mask = kwargs.get(&quot;audio_mask&quot;, None)
+ audio_token_lengths = audio_mask.sum(-1) if audio_mask is not None else None
+ text_token_int = kwargs.get(&quot;text_token_int&quot;, None)
+ if audio_token_lengths is None and text_token_int is not None:
+ audio_token_lengths = torch.tensor([len(text_token_int)], dtype=torch.int64)
+
+ batch = {&quot;speech&quot;: speech, &quot;speech_lengths&quot;: speech_lengths}
+ enc, enc_lens = self.audio_encoder.encode(**batch)
+ with autocast(False):
+ enc_mask = sequence_mask(enc_lens, enc.size(1), device=enc.device)[:, None, :]
+ pre_acoustic_embeds, pre_token_length, _, _ = self.audio_encoder.predictor(
+ enc,
+ mask=enc_mask,
+ target_label_length=audio_token_lengths,
+ )
+ loss_pre = 0.0
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/llm_asr_nar/model.py#L691" target="_blank" class="src-more">View full source on GitHub →</a></details></div>
+<div class="api-detail" id="e151"><span class="dtag badge-m">method</span><h2><code>LLMASRNARPrompt.inference(data_in, data_lengths, key, tokenizer, frontend, **kwargs)</code></h2><div class="dmod">funasr.models.llm_asr_nar.LLMASRNARPrompt · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/llm_asr_nar/model.py#L753" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Run inference on input data.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>data_in</code> — Input data (audio samples, file paths, or text).</li>
+<li><code>data_lengths</code> — Lengths of each input sample in the batch.</li>
+<li><code>key</code> — Sample identifiers.</li>
+<li><code>tokenizer</code> — Tokenizer instance for text encoding/decoding.</li>
+<li><code>frontend</code> — Audio frontend for feature extraction.</li>
+<li><code>**kwargs</code> — Additional keyword arguments.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def inference(
+ self,
+ data_in,
+ data_lengths=None,
+ key: list = None,
+ tokenizer=None,
+ frontend=None,
+ **kwargs,
+ ):
+
+ &quot;&quot;&quot;Run inference on input data.
+
+ Args:
+ data_in: Input data (audio samples, file paths, or text).
+ data_lengths: Lengths of each input sample in the batch.
+ key: Sample identifiers.
+ tokenizer: Tokenizer instance for text encoding/decoding.
+ frontend: Audio frontend for feature extraction.
+ **kwargs: Additional keyword arguments.
+ &quot;&quot;&quot;
+ prompt = kwargs.get(&quot;prompt&quot;, &quot;Transcribe speech to text.&quot;)
+
+ if kwargs.get(&quot;batch_size&quot;, 1) &gt; 1:
+ raise NotImplementedError(&quot;batch decoding is not implemented&quot;)
+
+ meta_data = {}
+ if (
+ isinstance(data_in, torch.Tensor) and kwargs.get(&quot;data_type&quot;, &quot;sound&quot;) == &quot;fbank&quot;
+ ): # fbank
+ speech, speech_lengths = data_in, data_lengths
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/llm_asr_nar/model.py#L753" target="_blank" class="src-more">View full source on GitHub →</a></details></div>
+<div class="api-detail" id="e152"><span class="dtag badge-c">class</span><h2>MonotonicAligner</h2><div class="dmod">funasr.models.monotonic_aligner · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/monotonic_aligner/model.py#L25" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><ul>
+<li><code>MonotonicAligner</code> — Forced alignment model for timestamp prediction.</li>
+</ul>
+<p>Given audio and text, computes character-level time alignments</p>
+<p>using monotonic attention constraint.</p>
+<ul>
+<li><code>Usage</code> — model.generate(input=(audio_path, text_path), data_type=(&quot;sound&quot;, &quot;text&quot;))</li>
+<li><code>Output</code> — {&quot;key&quot;: str, &quot;timestamp&quot;: [[start_ms, end_ms], ...], &quot;text&quot;: str}</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code>class MonotonicAligner(torch.nn.Module):
+ &quot;&quot;&quot;MonotonicAligner: Forced alignment model for timestamp prediction.
+
+ Given audio and text, computes character-level time alignments
+ using monotonic attention constraint.
+
+ Usage: model.generate(input=(audio_path, text_path), data_type=(&quot;sound&quot;, &quot;text&quot;))
+ Output: {&quot;key&quot;: str, &quot;timestamp&quot;: [[start_ms, end_ms], ...], &quot;text&quot;: str}
+ &quot;&quot;&quot;
+
+ def __init__(
+ self,
+ input_size: int = 80,
+ specaug: Optional[str] = None,
+ specaug_conf: Optional[Dict] = None,
+ normalize: str = None,
+ normalize_conf: Optional[Dict] = None,
+ encoder: str = None,
+ encoder_conf: Optional[Dict] = None,
+ predictor: str = None,
+ predictor_conf: Optional[Dict] = None,
+ predictor_bias: int = 0,
+ length_normalized_loss: bool = False,
+ **kwargs,
+ ):
+ &quot;&quot;&quot;Initialize MonotonicAligner.
+
+ Args:
+ input_size: Size/dimension parameter.
+ specaug: TODO.
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/monotonic_aligner/model.py#L25" target="_blank" class="src-more">View full source on GitHub →</a></details><h3 class="mt">Methods</h3><div class="mblk"><code>.forward(speech, speech_lengths, text, text_lengths)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/monotonic_aligner/model.py#L86" target="_blank" class="gh-link">L86</a><div class="mdoc"><p>Frontend + Encoder + Decoder + Calc loss</p>
+<h4>Args:</h4>
+<ul>
+<li><code>speech</code> — (Batch, Length, ...)</li>
+<li><code>speech_lengths</code> — (Batch, )</li>
+<li><code>text</code> — (Batch, Length)</li>
+<li><code>text_lengths</code> — (Batch,)</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def forward(
+ self,
+ speech: torch.Tensor,
+ speech_lengths: torch.Tensor,
+ text: torch.Tensor,
+ text_lengths: torch.Tensor,
+ ) -&gt; Tuple[torch.Tensor, Dict[str, torch.Tensor], torch.Tensor]:
+ &quot;&quot;&quot;Frontend + Encoder + Decoder + Calc loss
+ Args:
+ speech: (Batch, Length, ...)
+ speech_lengths: (Batch, )
+ text: (Batch, Length)
+ text_lengths: (Batch,)
+ &quot;&quot;&quot;
+ assert text_lengths.dim() == 1, text_lengths.shape
+ # Check that batch_size is unified
+ assert (
+ speech.shape[0] == speech_lengths.shape[0] == text.shape[0] == text_lengths.shape[0]
+ ), (speech.shape, speech_lengths.shape, text.shape, text_lengths.shape)
+ batch_size = speech.shape[0]
+ # for data-parallel
+ text = text[:, : text_lengths.max()]
+ speech = speech[:, : speech_lengths.max()]
+
+ # 1. Encoder
+ encoder_out, encoder_out_lens = self.encode(speech, speech_lengths)
+
+ encoder_out_mask = (
+ ~make_pad_mask(encoder_out_lens, maxlen=encoder_out.size(1))[:, None, :]
+ ).to(encoder_out.device)
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/monotonic_aligner/model.py#L86" target="_blank" class="src-more">View full source →</a></details></div><div class="mblk"><code>.calc_predictor_timestamp(encoder_out, encoder_out_lens, token_num)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/monotonic_aligner/model.py#L137" target="_blank" class="gh-link">L137</a><div class="mdoc"><p>Calc predictor timestamp.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>encoder_out</code> — Encoder output tensor.</li>
+<li><code>encoder_out_lens</code> — Encoder output lengths.</li>
+<li><code>token_num</code> — TODO.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def calc_predictor_timestamp(self, encoder_out, encoder_out_lens, token_num):
+ &quot;&quot;&quot;Calc predictor timestamp.
+
+ Args:
+ encoder_out: Encoder output tensor.
+ encoder_out_lens: Encoder output lengths.
+ token_num: TODO.
+ &quot;&quot;&quot;
+ encoder_out_mask = (
+ ~make_pad_mask(encoder_out_lens, maxlen=encoder_out.size(1))[:, None, :]
+ ).to(encoder_out.device)
+ ds_alphas, ds_cif_peak, us_alphas, us_peaks = self.predictor.get_upsample_timestamp(
+ encoder_out, encoder_out_mask, token_num
+ )
+ return ds_alphas, ds_cif_peak, us_alphas, us_peaks
+</code></pre></details></div><div class="mblk"><code>.encode(speech, speech_lengths, **kwargs)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/monotonic_aligner/model.py#L153" target="_blank" class="gh-link">L153</a><div class="mdoc"><p>Encoder. Note that this method is used by asr_inference.py</p>
+<h4>Args:</h4>
+<ul>
+<li><code>speech</code> — (Batch, Length, ...)</li>
+<li><code>speech_lengths</code> — (Batch, )</li>
+<li><code>ind</code> — int</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def encode(
+ self,
+ speech: torch.Tensor,
+ speech_lengths: torch.Tensor,
+ **kwargs,
+ ) -&gt; Tuple[torch.Tensor, torch.Tensor]:
+ &quot;&quot;&quot;Encoder. Note that this method is used by asr_inference.py
+ Args:
+ speech: (Batch, Length, ...)
+ speech_lengths: (Batch, )
+ ind: int
+ &quot;&quot;&quot;
+ with autocast(False):
+
+ # Data augmentation
+ if self.specaug is not None and self.training:
+ speech, speech_lengths = self.specaug(speech, speech_lengths)
+
+ # Normalization for feature: e.g. Global-CMVN, Utterance-CMVN
+ if self.normalize is not None:
+ speech, speech_lengths = self.normalize(speech, speech_lengths)
+
+ # Forward encoder
+ encoder_out, encoder_out_lens, _ = self.encoder(speech, speech_lengths)
+ if isinstance(encoder_out, tuple):
+ encoder_out = encoder_out[0]
+
+ return encoder_out, encoder_out_lens
+</code></pre></details></div><div class="mblk"><code>.inference(data_in, data_lengths, key, tokenizer, frontend, **kwargs)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/monotonic_aligner/model.py#L182" target="_blank" class="gh-link">L182</a><div class="mdoc"><p>Run inference on input data.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>data_in</code> — Input data (audio samples, file paths, or text).</li>
+<li><code>data_lengths</code> — Lengths of each input sample in the batch.</li>
+<li><code>key</code> — Sample identifiers.</li>
+<li><code>tokenizer</code> — Tokenizer instance for text encoding/decoding.</li>
+<li><code>frontend</code> — Audio frontend for feature extraction.</li>
+<li><code>**kwargs</code> — Additional keyword arguments.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def inference(
+ self,
+ data_in,
+ data_lengths=None,
+ key: list = None,
+ tokenizer=None,
+ frontend=None,
+ **kwargs,
+ ):
+ &quot;&quot;&quot;Run inference on input data.
+
+ Args:
+ data_in: Input data (audio samples, file paths, or text).
+ data_lengths: Lengths of each input sample in the batch.
+ key: Sample identifiers.
+ tokenizer: Tokenizer instance for text encoding/decoding.
+ frontend: Audio frontend for feature extraction.
+ **kwargs: Additional keyword arguments.
+ &quot;&quot;&quot;
+ meta_data = {}
+ # extract fbank feats
+ time1 = time.perf_counter()
+ audio_list, text_token_int_list = load_audio_text_image_video(
+ data_in,
+ fs=frontend.fs,
+ audio_fs=kwargs.get(&quot;fs&quot;, 16000),
+ data_type=kwargs.get(&quot;data_type&quot;, &quot;sound&quot;),
+ tokenizer=tokenizer,
+ )
+ time2 = time.perf_counter()
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/monotonic_aligner/model.py#L182" target="_blank" class="src-more">View full source →</a></details></div></div>
+<div class="api-detail" id="e153"><span class="dtag badge-m">method</span><h2><code>MonotonicAligner.forward(speech, speech_lengths, text, text_lengths)</code></h2><div class="dmod">funasr.models.monotonic_aligner.MonotonicAligner · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/monotonic_aligner/model.py#L86" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Frontend + Encoder + Decoder + Calc loss</p>
+<h4>Args:</h4>
+<ul>
+<li><code>speech</code> — (Batch, Length, ...)</li>
+<li><code>speech_lengths</code> — (Batch, )</li>
+<li><code>text</code> — (Batch, Length)</li>
+<li><code>text_lengths</code> — (Batch,)</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def forward(
+ self,
+ speech: torch.Tensor,
+ speech_lengths: torch.Tensor,
+ text: torch.Tensor,
+ text_lengths: torch.Tensor,
+ ) -&gt; Tuple[torch.Tensor, Dict[str, torch.Tensor], torch.Tensor]:
+ &quot;&quot;&quot;Frontend + Encoder + Decoder + Calc loss
+ Args:
+ speech: (Batch, Length, ...)
+ speech_lengths: (Batch, )
+ text: (Batch, Length)
+ text_lengths: (Batch,)
+ &quot;&quot;&quot;
+ assert text_lengths.dim() == 1, text_lengths.shape
+ # Check that batch_size is unified
+ assert (
+ speech.shape[0] == speech_lengths.shape[0] == text.shape[0] == text_lengths.shape[0]
+ ), (speech.shape, speech_lengths.shape, text.shape, text_lengths.shape)
+ batch_size = speech.shape[0]
+ # for data-parallel
+ text = text[:, : text_lengths.max()]
+ speech = speech[:, : speech_lengths.max()]
+
+ # 1. Encoder
+ encoder_out, encoder_out_lens = self.encode(speech, speech_lengths)
+
+ encoder_out_mask = (
+ ~make_pad_mask(encoder_out_lens, maxlen=encoder_out.size(1))[:, None, :]
+ ).to(encoder_out.device)
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/monotonic_aligner/model.py#L86" target="_blank" class="src-more">View full source on GitHub →</a></details></div>
+<div class="api-detail" id="e154"><span class="dtag badge-m">method</span><h2><code>MonotonicAligner.calc_predictor_timestamp(encoder_out, encoder_out_lens, token_num)</code></h2><div class="dmod">funasr.models.monotonic_aligner.MonotonicAligner · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/monotonic_aligner/model.py#L137" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Calc predictor timestamp.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>encoder_out</code> — Encoder output tensor.</li>
+<li><code>encoder_out_lens</code> — Encoder output lengths.</li>
+<li><code>token_num</code> — TODO.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def calc_predictor_timestamp(self, encoder_out, encoder_out_lens, token_num):
+ &quot;&quot;&quot;Calc predictor timestamp.
+
+ Args:
+ encoder_out: Encoder output tensor.
+ encoder_out_lens: Encoder output lengths.
+ token_num: TODO.
+ &quot;&quot;&quot;
+ encoder_out_mask = (
+ ~make_pad_mask(encoder_out_lens, maxlen=encoder_out.size(1))[:, None, :]
+ ).to(encoder_out.device)
+ ds_alphas, ds_cif_peak, us_alphas, us_peaks = self.predictor.get_upsample_timestamp(
+ encoder_out, encoder_out_mask, token_num
+ )
+ return ds_alphas, ds_cif_peak, us_alphas, us_peaks
+</code></pre></details></div>
+<div class="api-detail" id="e155"><span class="dtag badge-m">method</span><h2><code>MonotonicAligner.encode(speech, speech_lengths, **kwargs)</code></h2><div class="dmod">funasr.models.monotonic_aligner.MonotonicAligner · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/monotonic_aligner/model.py#L153" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Encoder. Note that this method is used by asr_inference.py</p>
+<h4>Args:</h4>
+<ul>
+<li><code>speech</code> — (Batch, Length, ...)</li>
+<li><code>speech_lengths</code> — (Batch, )</li>
+<li><code>ind</code> — int</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def encode(
+ self,
+ speech: torch.Tensor,
+ speech_lengths: torch.Tensor,
+ **kwargs,
+ ) -&gt; Tuple[torch.Tensor, torch.Tensor]:
+ &quot;&quot;&quot;Encoder. Note that this method is used by asr_inference.py
+ Args:
+ speech: (Batch, Length, ...)
+ speech_lengths: (Batch, )
+ ind: int
+ &quot;&quot;&quot;
+ with autocast(False):
+
+ # Data augmentation
+ if self.specaug is not None and self.training:
+ speech, speech_lengths = self.specaug(speech, speech_lengths)
+
+ # Normalization for feature: e.g. Global-CMVN, Utterance-CMVN
+ if self.normalize is not None:
+ speech, speech_lengths = self.normalize(speech, speech_lengths)
+
+ # Forward encoder
+ encoder_out, encoder_out_lens, _ = self.encoder(speech, speech_lengths)
+ if isinstance(encoder_out, tuple):
+ encoder_out = encoder_out[0]
+
+ return encoder_out, encoder_out_lens
+</code></pre></details></div>
+<div class="api-detail" id="e156"><span class="dtag badge-m">method</span><h2><code>MonotonicAligner.inference(data_in, data_lengths, key, tokenizer, frontend, **kwargs)</code></h2><div class="dmod">funasr.models.monotonic_aligner.MonotonicAligner · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/monotonic_aligner/model.py#L182" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Run inference on input data.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>data_in</code> — Input data (audio samples, file paths, or text).</li>
+<li><code>data_lengths</code> — Lengths of each input sample in the batch.</li>
+<li><code>key</code> — Sample identifiers.</li>
+<li><code>tokenizer</code> — Tokenizer instance for text encoding/decoding.</li>
+<li><code>frontend</code> — Audio frontend for feature extraction.</li>
+<li><code>**kwargs</code> — Additional keyword arguments.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def inference(
+ self,
+ data_in,
+ data_lengths=None,
+ key: list = None,
+ tokenizer=None,
+ frontend=None,
+ **kwargs,
+ ):
+ &quot;&quot;&quot;Run inference on input data.
+
+ Args:
+ data_in: Input data (audio samples, file paths, or text).
+ data_lengths: Lengths of each input sample in the batch.
+ key: Sample identifiers.
+ tokenizer: Tokenizer instance for text encoding/decoding.
+ frontend: Audio frontend for feature extraction.
+ **kwargs: Additional keyword arguments.
+ &quot;&quot;&quot;
+ meta_data = {}
+ # extract fbank feats
+ time1 = time.perf_counter()
+ audio_list, text_token_int_list = load_audio_text_image_video(
+ data_in,
+ fs=frontend.fs,
+ audio_fs=kwargs.get(&quot;fs&quot;, 16000),
+ data_type=kwargs.get(&quot;data_type&quot;, &quot;sound&quot;),
+ tokenizer=tokenizer,
+ )
+ time2 = time.perf_counter()
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/monotonic_aligner/model.py#L182" target="_blank" class="src-more">View full source on GitHub →</a></details></div>
+<div class="api-detail" id="e157"><span class="dtag badge-c">class</span><h2>Paraformer</h2><div class="dmod">funasr.models.paraformer · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/paraformer/model.py#L30" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><ul>
+<li><code>Paraformer</code> — Non-autoregressive End-to-End ASR Model.</li>
+<li><code>High</code> — accuracy speech recognition for Chinese/English. The production workhorse.</li>
+</ul>
+<h4>Features:</h4>
+<ul>
+<li>Non-autoregressive (parallel decoding, fast inference)</li>
+<li>Character-level timestamps via CIF predictor</li>
+<li>Streaming and offline modes</li>
+<li>Hotword customization</li>
+<li>Speaker diarization (with spk_model)</li>
+<li>ONNX export support</li>
+<li><code>Output</code> — {&quot;key&quot;: &quot;...&quot;, &quot;text&quot;: &quot;recognized text&quot;, &quot;timestamp&quot;: [[start_ms, end_ms], ...]}</li>
+<li><code>Note</code> — Requires punc_model=&quot;ct-punc&quot; for punctuation (unlike Fun-ASR-Nano/SenseVoice</li>
+</ul>
+<p>which output punctuation natively).</p>
+<ul>
+<li><code>Author</code> — Speech Lab of DAMO Academy, Alibaba Group</li>
+<li><code>Paraformer</code> — Fast and Accurate Parallel Transformer for Non-autoregressive End-to-End Speech Recognition</li>
+</ul>
+<p>https://arxiv.org/abs/2206.08317</p>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code>class Paraformer(torch.nn.Module):
+ &quot;&quot;&quot;Paraformer: Non-autoregressive End-to-End ASR Model.
+
+ High-accuracy speech recognition for Chinese/English. The production workhorse.
+
+ Features:
+ - Non-autoregressive (parallel decoding, fast inference)
+ - Character-level timestamps via CIF predictor
+ - Streaming and offline modes
+ - Hotword customization
+ - Speaker diarization (with spk_model)
+ - ONNX export support
+
+ Output: {&quot;key&quot;: &quot;...&quot;, &quot;text&quot;: &quot;recognized text&quot;, &quot;timestamp&quot;: [[start_ms, end_ms], ...]}
+
+ Note: Requires punc_model=&quot;ct-punc&quot; for punctuation (unlike Fun-ASR-Nano/SenseVoice
+ which output punctuation natively).
+
+ Author: Speech Lab of DAMO Academy, Alibaba Group
+ Paraformer: Fast and Accurate Parallel Transformer for Non-autoregressive End-to-End Speech Recognition
+ https://arxiv.org/abs/2206.08317
+ &quot;&quot;&quot;
+
+ def __init__(
+ self,
+ specaug: Optional[str] = None,
+ specaug_conf: Optional[Dict] = None,
+ normalize: str = None,
+ normalize_conf: Optional[Dict] = None,
+ encoder: str = None,
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/paraformer/model.py#L30" target="_blank" class="src-more">View full source on GitHub →</a></details><h3 class="mt">Methods</h3><div class="mblk"><code>.forward(speech, speech_lengths, text, text_lengths, **kwargs)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/paraformer/model.py#L215" target="_blank" class="gh-link">L215</a><div class="mdoc"><p>Encoder + Decoder + Calc loss</p>
+<h4>Args:</h4>
+<ul>
+<li><code>speech</code> — (Batch, Length, ...)</li>
+<li><code>speech_lengths</code> — (Batch, )</li>
+<li><code>text</code> — (Batch, Length)</li>
+<li><code>text_lengths</code> — (Batch,)</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def forward(
+ self,
+ speech: torch.Tensor,
+ speech_lengths: torch.Tensor,
+ text: torch.Tensor,
+ text_lengths: torch.Tensor,
+ **kwargs,
+ ) -&gt; Tuple[torch.Tensor, Dict[str, torch.Tensor], torch.Tensor]:
+ &quot;&quot;&quot;Encoder + Decoder + Calc loss
+ Args:
+ speech: (Batch, Length, ...)
+ speech_lengths: (Batch, )
+ text: (Batch, Length)
+ text_lengths: (Batch,)
+ &quot;&quot;&quot;
+ if len(text_lengths.size()) &gt; 1:
+ text_lengths = text_lengths[:, 0]
+ if len(speech_lengths.size()) &gt; 1:
+ speech_lengths = speech_lengths[:, 0]
+
+ batch_size = speech.shape[0]
+
+ # Encoder
+ encoder_out, encoder_out_lens = self.encode(speech, speech_lengths)
+
+ loss_ctc, cer_ctc = None, None
+ loss_pre = None
+ stats = dict()
+
+ # decoder: CTC branch
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/paraformer/model.py#L215" target="_blank" class="src-more">View full source →</a></details></div><div class="mblk"><code>.encode(speech, speech_lengths, **kwargs)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/paraformer/model.py#L286" target="_blank" class="gh-link">L286</a><div class="mdoc"><p>Encoder. Note that this method is used by asr_inference.py</p>
+<h4>Args:</h4>
+<ul>
+<li><code>speech</code> — (Batch, Length, ...)</li>
+<li><code>speech_lengths</code> — (Batch, )</li>
+<li><code>ind</code> — int</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def encode(
+ self,
+ speech: torch.Tensor,
+ speech_lengths: torch.Tensor,
+ **kwargs,
+ ) -&gt; Tuple[torch.Tensor, torch.Tensor]:
+ &quot;&quot;&quot;Encoder. Note that this method is used by asr_inference.py
+ Args:
+ speech: (Batch, Length, ...)
+ speech_lengths: (Batch, )
+ ind: int
+ &quot;&quot;&quot;
+ with autocast(False):
+
+ # Data augmentation
+ if self.specaug is not None and self.training:
+ speech, speech_lengths = self.specaug(speech, speech_lengths)
+
+ # Normalization for feature: e.g. Global-CMVN, Utterance-CMVN
+ if self.normalize is not None:
+ speech, speech_lengths = self.normalize(speech, speech_lengths)
+
+ # Forward encoder
+ encoder_out, encoder_out_lens, _ = self.encoder(speech, speech_lengths)
+ if isinstance(encoder_out, tuple):
+ encoder_out = encoder_out[0]
+
+ return encoder_out, encoder_out_lens
+</code></pre></details></div><div class="mblk"><code>.calc_predictor(encoder_out, encoder_out_lens)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/paraformer/model.py#L315" target="_blank" class="gh-link">L315</a><div class="mdoc"><p>Calc predictor.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>encoder_out</code> — Encoder output tensor.</li>
+<li><code>encoder_out_lens</code> — Encoder output lengths.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def calc_predictor(self, encoder_out, encoder_out_lens):
+
+ &quot;&quot;&quot;Calc predictor.
+
+ Args:
+ encoder_out: Encoder output tensor.
+ encoder_out_lens: Encoder output lengths.
+ &quot;&quot;&quot;
+ encoder_out_mask = (
+ ~make_pad_mask(encoder_out_lens, maxlen=encoder_out.size(1))[:, None, :]
+ ).to(encoder_out.device)
+ pre_acoustic_embeds, pre_token_length, alphas, pre_peak_index = self.predictor(
+ encoder_out, None, encoder_out_mask, ignore_id=self.ignore_id
+ )
+ return pre_acoustic_embeds, pre_token_length, alphas, pre_peak_index
+</code></pre></details></div><div class="mblk"><code>.cal_decoder_with_predictor(encoder_out, encoder_out_lens, sematic_embeds, ys_pad_lens)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/paraformer/model.py#L331" target="_blank" class="gh-link">L331</a><div class="mdoc"><p>Cal decoder with predictor.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>encoder_out</code> — Encoder output tensor.</li>
+<li><code>encoder_out_lens</code> — Encoder output lengths.</li>
+<li><code>sematic_embeds</code> — TODO.</li>
+<li><code>ys_pad_lens</code> — Lengths of ys_pad.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def cal_decoder_with_predictor(
+ self, encoder_out, encoder_out_lens, sematic_embeds, ys_pad_lens
+ ):
+
+ &quot;&quot;&quot;Cal decoder with predictor.
+
+ Args:
+ encoder_out: Encoder output tensor.
+ encoder_out_lens: Encoder output lengths.
+ sematic_embeds: TODO.
+ ys_pad_lens: Lengths of ys_pad.
+ &quot;&quot;&quot;
+ decoder_outs = self.decoder(encoder_out, encoder_out_lens, sematic_embeds, ys_pad_lens)
+ decoder_out = decoder_outs[0]
+ decoder_out = torch.log_softmax(decoder_out, dim=-1)
+ return decoder_out, ys_pad_lens
+</code></pre></details></div><div class="mblk"><code>.sampler(encoder_out, encoder_out_lens, ys_pad, ys_pad_lens, pre_acoustic_embeds)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/paraformer/model.py#L408" target="_blank" class="gh-link">L408</a><div class="mdoc"><p>Sampler.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>encoder_out</code> — Encoder output tensor.</li>
+<li><code>encoder_out_lens</code> — Encoder output lengths.</li>
+<li><code>ys_pad</code> — TODO.</li>
+<li><code>ys_pad_lens</code> — Lengths of ys_pad.</li>
+<li><code>pre_acoustic_embeds</code> — TODO.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def sampler(self, encoder_out, encoder_out_lens, ys_pad, ys_pad_lens, pre_acoustic_embeds):
+
+ &quot;&quot;&quot;Sampler.
+
+ Args:
+ encoder_out: Encoder output tensor.
+ encoder_out_lens: Encoder output lengths.
+ ys_pad: TODO.
+ ys_pad_lens: Lengths of ys_pad.
+ pre_acoustic_embeds: TODO.
+ &quot;&quot;&quot;
+ tgt_mask = (~make_pad_mask(ys_pad_lens, maxlen=ys_pad_lens.max())[:, :, None]).to(
+ ys_pad.device
+ )
+ ys_pad_masked = ys_pad * tgt_mask[:, :, 0]
+ if self.share_embedding:
+ ys_pad_embed = self.decoder.output_layer.weight[ys_pad_masked]
+ else:
+ ys_pad_embed = self.decoder.embed(ys_pad_masked)
+ with torch.no_grad():
+ decoder_outs = self.decoder(
+ encoder_out, encoder_out_lens, pre_acoustic_embeds, ys_pad_lens
+ )
+ decoder_out, _ = decoder_outs[0], decoder_outs[1]
+ pred_tokens = decoder_out.argmax(-1)
+ nonpad_positions = ys_pad.ne(self.ignore_id)
+ seq_lens = (nonpad_positions).sum(1)
+ same_num = ((pred_tokens == ys_pad) &amp; nonpad_positions).sum(1)
+ input_mask = torch.ones_like(nonpad_positions)
+ bsz, seq_len = ys_pad.size()
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/paraformer/model.py#L408" target="_blank" class="src-more">View full source →</a></details></div><div class="mblk"><code>.init_beam_search(**kwargs)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/paraformer/model.py#L482" target="_blank" class="gh-link">L482</a><div class="mdoc"><p>Init beam search.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>**kwargs</code> — Additional keyword arguments.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def init_beam_search(
+ self,
+ **kwargs,
+ ):
+ &quot;&quot;&quot;Init beam search.
+
+ Args:
+ **kwargs: Additional keyword arguments.
+ &quot;&quot;&quot;
+ from funasr.models.paraformer.search import BeamSearchPara
+ from funasr.models.transformer.scorers.ctc import CTCPrefixScorer
+ from funasr.models.transformer.scorers.length_bonus import LengthBonus
+
+ # 1. Build ASR model
+ scorers = {}
+
+ if self.ctc != None:
+ ctc = CTCPrefixScorer(ctc=self.ctc, eos=self.eos)
+ scorers.update(ctc=ctc)
+ token_list = kwargs.get(&quot;token_list&quot;)
+ scorers.update(
+ length_bonus=LengthBonus(len(token_list)),
+ )
+
+ # 3. Build ngram model
+ # ngram is not supported now
+ ngram = None
+ scorers[&quot;ngram&quot;] = ngram
+
+ weights = dict(
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/paraformer/model.py#L482" target="_blank" class="src-more">View full source →</a></details></div><div class="mblk"><code>.inference(data_in, data_lengths, key, tokenizer, frontend, **kwargs)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/paraformer/model.py#L534" target="_blank" class="gh-link">L534</a><div class="mdoc"><p>Run inference on input data.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>data_in</code> — Input data (audio samples, file paths, or text).</li>
+<li><code>data_lengths</code> — Lengths of each input sample in the batch.</li>
+<li><code>key</code> — Sample identifiers.</li>
+<li><code>tokenizer</code> — Tokenizer instance for text encoding/decoding.</li>
+<li><code>frontend</code> — Audio frontend for feature extraction.</li>
+<li><code>**kwargs</code> — Additional keyword arguments.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def inference(
+ self,
+ data_in,
+ data_lengths=None,
+ key: list = None,
+ tokenizer=None,
+ frontend=None,
+ **kwargs,
+ ):
+ # init beamsearch
+ &quot;&quot;&quot;Run inference on input data.
+
+ Args:
+ data_in: Input data (audio samples, file paths, or text).
+ data_lengths: Lengths of each input sample in the batch.
+ key: Sample identifiers.
+ tokenizer: Tokenizer instance for text encoding/decoding.
+ frontend: Audio frontend for feature extraction.
+ **kwargs: Additional keyword arguments.
+ &quot;&quot;&quot;
+ is_use_ctc = kwargs.get(&quot;decoding_ctc_weight&quot;, 0.0) &gt; 0.00001 and self.ctc != None
+ is_use_lm = (
+ kwargs.get(&quot;lm_weight&quot;, 0.0) &gt; 0.00001 and kwargs.get(&quot;lm_file&quot;, None) is not None
+ )
+ pred_timestamp = kwargs.get(&quot;pred_timestamp&quot;, False)
+ if self.beam_search is None and (is_use_lm or is_use_ctc):
+ logging.info(&quot;enable beam_search&quot;)
+ self.init_beam_search(**kwargs)
+ self.nbest = kwargs.get(&quot;nbest&quot;, 1)
+
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/paraformer/model.py#L534" target="_blank" class="src-more">View full source →</a></details></div><div class="mblk"><code>.export(**kwargs)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/paraformer/model.py#L699" target="_blank" class="gh-link">L699</a><div class="mdoc"><p>Export.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>**kwargs</code> — Additional keyword arguments.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def export(self, **kwargs):
+ &quot;&quot;&quot;Export.
+
+ Args:
+ **kwargs: Additional keyword arguments.
+ &quot;&quot;&quot;
+ from .export_meta import export_rebuild_model
+
+ if &quot;max_seq_len&quot; not in kwargs:
+ kwargs[&quot;max_seq_len&quot;] = 512
+ models = export_rebuild_model(model=self, **kwargs)
+ return models
+</code></pre></details></div></div>
+<div class="api-detail" id="e158"><span class="dtag badge-m">method</span><h2><code>Paraformer.forward(speech, speech_lengths, text, text_lengths, **kwargs)</code></h2><div class="dmod">funasr.models.paraformer.Paraformer · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/paraformer/model.py#L215" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Encoder + Decoder + Calc loss</p>
+<h4>Args:</h4>
+<ul>
+<li><code>speech</code> — (Batch, Length, ...)</li>
+<li><code>speech_lengths</code> — (Batch, )</li>
+<li><code>text</code> — (Batch, Length)</li>
+<li><code>text_lengths</code> — (Batch,)</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def forward(
+ self,
+ speech: torch.Tensor,
+ speech_lengths: torch.Tensor,
+ text: torch.Tensor,
+ text_lengths: torch.Tensor,
+ **kwargs,
+ ) -&gt; Tuple[torch.Tensor, Dict[str, torch.Tensor], torch.Tensor]:
+ &quot;&quot;&quot;Encoder + Decoder + Calc loss
+ Args:
+ speech: (Batch, Length, ...)
+ speech_lengths: (Batch, )
+ text: (Batch, Length)
+ text_lengths: (Batch,)
+ &quot;&quot;&quot;
+ if len(text_lengths.size()) &gt; 1:
+ text_lengths = text_lengths[:, 0]
+ if len(speech_lengths.size()) &gt; 1:
+ speech_lengths = speech_lengths[:, 0]
+
+ batch_size = speech.shape[0]
+
+ # Encoder
+ encoder_out, encoder_out_lens = self.encode(speech, speech_lengths)
+
+ loss_ctc, cer_ctc = None, None
+ loss_pre = None
+ stats = dict()
+
+ # decoder: CTC branch
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/paraformer/model.py#L215" target="_blank" class="src-more">View full source on GitHub →</a></details></div>
+<div class="api-detail" id="e159"><span class="dtag badge-m">method</span><h2><code>Paraformer.encode(speech, speech_lengths, **kwargs)</code></h2><div class="dmod">funasr.models.paraformer.Paraformer · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/paraformer/model.py#L286" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Encoder. Note that this method is used by asr_inference.py</p>
+<h4>Args:</h4>
+<ul>
+<li><code>speech</code> — (Batch, Length, ...)</li>
+<li><code>speech_lengths</code> — (Batch, )</li>
+<li><code>ind</code> — int</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def encode(
+ self,
+ speech: torch.Tensor,
+ speech_lengths: torch.Tensor,
+ **kwargs,
+ ) -&gt; Tuple[torch.Tensor, torch.Tensor]:
+ &quot;&quot;&quot;Encoder. Note that this method is used by asr_inference.py
+ Args:
+ speech: (Batch, Length, ...)
+ speech_lengths: (Batch, )
+ ind: int
+ &quot;&quot;&quot;
+ with autocast(False):
+
+ # Data augmentation
+ if self.specaug is not None and self.training:
+ speech, speech_lengths = self.specaug(speech, speech_lengths)
+
+ # Normalization for feature: e.g. Global-CMVN, Utterance-CMVN
+ if self.normalize is not None:
+ speech, speech_lengths = self.normalize(speech, speech_lengths)
+
+ # Forward encoder
+ encoder_out, encoder_out_lens, _ = self.encoder(speech, speech_lengths)
+ if isinstance(encoder_out, tuple):
+ encoder_out = encoder_out[0]
+
+ return encoder_out, encoder_out_lens
+</code></pre></details></div>
+<div class="api-detail" id="e160"><span class="dtag badge-m">method</span><h2><code>Paraformer.calc_predictor(encoder_out, encoder_out_lens)</code></h2><div class="dmod">funasr.models.paraformer.Paraformer · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/paraformer/model.py#L315" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Calc predictor.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>encoder_out</code> — Encoder output tensor.</li>
+<li><code>encoder_out_lens</code> — Encoder output lengths.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def calc_predictor(self, encoder_out, encoder_out_lens):
+
+ &quot;&quot;&quot;Calc predictor.
+
+ Args:
+ encoder_out: Encoder output tensor.
+ encoder_out_lens: Encoder output lengths.
+ &quot;&quot;&quot;
+ encoder_out_mask = (
+ ~make_pad_mask(encoder_out_lens, maxlen=encoder_out.size(1))[:, None, :]
+ ).to(encoder_out.device)
+ pre_acoustic_embeds, pre_token_length, alphas, pre_peak_index = self.predictor(
+ encoder_out, None, encoder_out_mask, ignore_id=self.ignore_id
+ )
+ return pre_acoustic_embeds, pre_token_length, alphas, pre_peak_index
+</code></pre></details></div>
+<div class="api-detail" id="e161"><span class="dtag badge-m">method</span><h2><code>Paraformer.cal_decoder_with_predictor(encoder_out, encoder_out_lens, sematic_embeds, ys_pad_lens)</code></h2><div class="dmod">funasr.models.paraformer.Paraformer · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/paraformer/model.py#L331" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Cal decoder with predictor.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>encoder_out</code> — Encoder output tensor.</li>
+<li><code>encoder_out_lens</code> — Encoder output lengths.</li>
+<li><code>sematic_embeds</code> — TODO.</li>
+<li><code>ys_pad_lens</code> — Lengths of ys_pad.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def cal_decoder_with_predictor(
+ self, encoder_out, encoder_out_lens, sematic_embeds, ys_pad_lens
+ ):
+
+ &quot;&quot;&quot;Cal decoder with predictor.
+
+ Args:
+ encoder_out: Encoder output tensor.
+ encoder_out_lens: Encoder output lengths.
+ sematic_embeds: TODO.
+ ys_pad_lens: Lengths of ys_pad.
+ &quot;&quot;&quot;
+ decoder_outs = self.decoder(encoder_out, encoder_out_lens, sematic_embeds, ys_pad_lens)
+ decoder_out = decoder_outs[0]
+ decoder_out = torch.log_softmax(decoder_out, dim=-1)
+ return decoder_out, ys_pad_lens
+</code></pre></details></div>
+<div class="api-detail" id="e162"><span class="dtag badge-m">method</span><h2><code>Paraformer.sampler(encoder_out, encoder_out_lens, ys_pad, ys_pad_lens, pre_acoustic_embeds)</code></h2><div class="dmod">funasr.models.paraformer.Paraformer · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/paraformer/model.py#L408" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Sampler.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>encoder_out</code> — Encoder output tensor.</li>
+<li><code>encoder_out_lens</code> — Encoder output lengths.</li>
+<li><code>ys_pad</code> — TODO.</li>
+<li><code>ys_pad_lens</code> — Lengths of ys_pad.</li>
+<li><code>pre_acoustic_embeds</code> — TODO.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def sampler(self, encoder_out, encoder_out_lens, ys_pad, ys_pad_lens, pre_acoustic_embeds):
+
+ &quot;&quot;&quot;Sampler.
+
+ Args:
+ encoder_out: Encoder output tensor.
+ encoder_out_lens: Encoder output lengths.
+ ys_pad: TODO.
+ ys_pad_lens: Lengths of ys_pad.
+ pre_acoustic_embeds: TODO.
+ &quot;&quot;&quot;
+ tgt_mask = (~make_pad_mask(ys_pad_lens, maxlen=ys_pad_lens.max())[:, :, None]).to(
+ ys_pad.device
+ )
+ ys_pad_masked = ys_pad * tgt_mask[:, :, 0]
+ if self.share_embedding:
+ ys_pad_embed = self.decoder.output_layer.weight[ys_pad_masked]
+ else:
+ ys_pad_embed = self.decoder.embed(ys_pad_masked)
+ with torch.no_grad():
+ decoder_outs = self.decoder(
+ encoder_out, encoder_out_lens, pre_acoustic_embeds, ys_pad_lens
+ )
+ decoder_out, _ = decoder_outs[0], decoder_outs[1]
+ pred_tokens = decoder_out.argmax(-1)
+ nonpad_positions = ys_pad.ne(self.ignore_id)
+ seq_lens = (nonpad_positions).sum(1)
+ same_num = ((pred_tokens == ys_pad) &amp; nonpad_positions).sum(1)
+ input_mask = torch.ones_like(nonpad_positions)
+ bsz, seq_len = ys_pad.size()
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/paraformer/model.py#L408" target="_blank" class="src-more">View full source on GitHub →</a></details></div>
+<div class="api-detail" id="e163"><span class="dtag badge-m">method</span><h2><code>Paraformer.init_beam_search(**kwargs)</code></h2><div class="dmod">funasr.models.paraformer.Paraformer · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/paraformer/model.py#L482" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Init beam search.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>**kwargs</code> — Additional keyword arguments.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def init_beam_search(
+ self,
+ **kwargs,
+ ):
+ &quot;&quot;&quot;Init beam search.
+
+ Args:
+ **kwargs: Additional keyword arguments.
+ &quot;&quot;&quot;
+ from funasr.models.paraformer.search import BeamSearchPara
+ from funasr.models.transformer.scorers.ctc import CTCPrefixScorer
+ from funasr.models.transformer.scorers.length_bonus import LengthBonus
+
+ # 1. Build ASR model
+ scorers = {}
+
+ if self.ctc != None:
+ ctc = CTCPrefixScorer(ctc=self.ctc, eos=self.eos)
+ scorers.update(ctc=ctc)
+ token_list = kwargs.get(&quot;token_list&quot;)
+ scorers.update(
+ length_bonus=LengthBonus(len(token_list)),
+ )
+
+ # 3. Build ngram model
+ # ngram is not supported now
+ ngram = None
+ scorers[&quot;ngram&quot;] = ngram
+
+ weights = dict(
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/paraformer/model.py#L482" target="_blank" class="src-more">View full source on GitHub →</a></details></div>
+<div class="api-detail" id="e164"><span class="dtag badge-m">method</span><h2><code>Paraformer.inference(data_in, data_lengths, key, tokenizer, frontend, **kwargs)</code></h2><div class="dmod">funasr.models.paraformer.Paraformer · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/paraformer/model.py#L534" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Run inference on input data.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>data_in</code> — Input data (audio samples, file paths, or text).</li>
+<li><code>data_lengths</code> — Lengths of each input sample in the batch.</li>
+<li><code>key</code> — Sample identifiers.</li>
+<li><code>tokenizer</code> — Tokenizer instance for text encoding/decoding.</li>
+<li><code>frontend</code> — Audio frontend for feature extraction.</li>
+<li><code>**kwargs</code> — Additional keyword arguments.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def inference(
+ self,
+ data_in,
+ data_lengths=None,
+ key: list = None,
+ tokenizer=None,
+ frontend=None,
+ **kwargs,
+ ):
+ # init beamsearch
+ &quot;&quot;&quot;Run inference on input data.
+
+ Args:
+ data_in: Input data (audio samples, file paths, or text).
+ data_lengths: Lengths of each input sample in the batch.
+ key: Sample identifiers.
+ tokenizer: Tokenizer instance for text encoding/decoding.
+ frontend: Audio frontend for feature extraction.
+ **kwargs: Additional keyword arguments.
+ &quot;&quot;&quot;
+ is_use_ctc = kwargs.get(&quot;decoding_ctc_weight&quot;, 0.0) &gt; 0.00001 and self.ctc != None
+ is_use_lm = (
+ kwargs.get(&quot;lm_weight&quot;, 0.0) &gt; 0.00001 and kwargs.get(&quot;lm_file&quot;, None) is not None
+ )
+ pred_timestamp = kwargs.get(&quot;pred_timestamp&quot;, False)
+ if self.beam_search is None and (is_use_lm or is_use_ctc):
+ logging.info(&quot;enable beam_search&quot;)
+ self.init_beam_search(**kwargs)
+ self.nbest = kwargs.get(&quot;nbest&quot;, 1)
+
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/paraformer/model.py#L534" target="_blank" class="src-more">View full source on GitHub →</a></details></div>
+<div class="api-detail" id="e165"><span class="dtag badge-m">method</span><h2><code>Paraformer.export(**kwargs)</code></h2><div class="dmod">funasr.models.paraformer.Paraformer · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/paraformer/model.py#L699" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Export.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>**kwargs</code> — Additional keyword arguments.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def export(self, **kwargs):
+ &quot;&quot;&quot;Export.
+
+ Args:
+ **kwargs: Additional keyword arguments.
+ &quot;&quot;&quot;
+ from .export_meta import export_rebuild_model
+
+ if &quot;max_seq_len&quot; not in kwargs:
+ kwargs[&quot;max_seq_len&quot;] = 512
+ models = export_rebuild_model(model=self, **kwargs)
+ return models
+</code></pre></details></div>
+<div class="api-detail" id="e166"><span class="dtag badge-c">class</span><h2>ParaformerStreaming</h2><div class="dmod">funasr.models.paraformer_streaming · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/paraformer_streaming/model.py#L43" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><ul>
+<li><code>ParaformerStreaming</code> — Streaming (online) version of Paraformer.</li>
+</ul>
+<p>Processes audio chunk-by-chunk with encoder lookback for real-time</p>
+<p>transcription. Uses cache mechanism to maintain state across chunks.</p>
+<ul>
+<li><code>Usage</code> — generate(input=chunk, cache=cache, is_final=bool, chunk_size=[0,10,5])</li>
+<li><code>Output</code> — {&quot;key&quot;: str, &quot;text&quot;: str} (partial results per chunk)</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code>class ParaformerStreaming(Paraformer):
+ &quot;&quot;&quot;ParaformerStreaming: Streaming (online) version of Paraformer.
+
+ Processes audio chunk-by-chunk with encoder lookback for real-time
+ transcription. Uses cache mechanism to maintain state across chunks.
+
+ Usage: generate(input=chunk, cache=cache, is_final=bool, chunk_size=[0,10,5])
+ Output: {&quot;key&quot;: str, &quot;text&quot;: str} (partial results per chunk)
+ &quot;&quot;&quot;
+
+ def __init__(
+ self,
+ *args,
+ **kwargs,
+ ):
+
+ &quot;&quot;&quot;Initialize ParaformerStreaming.
+
+ Args:
+ *args: Variable positional arguments.
+ **kwargs: Additional keyword arguments.
+ &quot;&quot;&quot;
+ super().__init__(*args, **kwargs)
+
+ self.sampling_ratio = kwargs.get(&quot;sampling_ratio&quot;, 0.2)
+
+ self.scama_mask = None
+ if (
+ hasattr(self.encoder, &quot;overlap_chunk_cls&quot;)
+ and self.encoder.overlap_chunk_cls is not None
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/paraformer_streaming/model.py#L43" target="_blank" class="src-more">View full source on GitHub →</a></details><h3 class="mt">Methods</h3><div class="mblk"><code>.forward(speech, speech_lengths, text, text_lengths, **kwargs)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/paraformer_streaming/model.py#L83" target="_blank" class="gh-link">L83</a><div class="mdoc"><p>Encoder + Decoder + Calc loss</p>
+<h4>Args:</h4>
+<ul>
+<li><code>speech</code> — (Batch, Length, ...)</li>
+<li><code>speech_lengths</code> — (Batch, )</li>
+<li><code>text</code> — (Batch, Length)</li>
+<li><code>text_lengths</code> — (Batch,)</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def forward(
+ self,
+ speech: torch.Tensor,
+ speech_lengths: torch.Tensor,
+ text: torch.Tensor,
+ text_lengths: torch.Tensor,
+ **kwargs,
+ ) -&gt; Tuple[torch.Tensor, Dict[str, torch.Tensor], torch.Tensor]:
+ &quot;&quot;&quot;Encoder + Decoder + Calc loss
+ Args:
+ speech: (Batch, Length, ...)
+ speech_lengths: (Batch, )
+ text: (Batch, Length)
+ text_lengths: (Batch,)
+ &quot;&quot;&quot;
+ decoding_ind = kwargs.get(&quot;decoding_ind&quot;)
+ if len(text_lengths.size()) &gt; 1:
+ text_lengths = text_lengths[:, 0]
+ if len(speech_lengths.size()) &gt; 1:
+ speech_lengths = speech_lengths[:, 0]
+
+ batch_size = speech.shape[0]
+
+ # Encoder
+ if hasattr(self.encoder, &quot;overlap_chunk_cls&quot;):
+ ind = self.encoder.overlap_chunk_cls.random_choice(self.training, decoding_ind)
+ encoder_out, encoder_out_lens = self.encode(speech, speech_lengths, ind=ind)
+ else:
+ encoder_out, encoder_out_lens = self.encode(speech, speech_lengths)
+
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/paraformer_streaming/model.py#L83" target="_blank" class="src-more">View full source →</a></details></div><div class="mblk"><code>.encode_chunk(speech, speech_lengths, cache, **kwargs)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/paraformer_streaming/model.py#L165" target="_blank" class="gh-link">L165</a><div class="mdoc"><p>Encode chunk.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>speech</code> — Speech audio tensor, shape (batch, time).</li>
+<li><code>speech_lengths</code> — Length of each speech sample.</li>
+<li><code>cache</code> — State cache dict for streaming inference.</li>
+<li><code>**kwargs</code> — Additional keyword arguments.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def encode_chunk(
+ self,
+ speech: torch.Tensor,
+ speech_lengths: torch.Tensor,
+ cache: dict = None,
+ **kwargs,
+ ) -&gt; Tuple[torch.Tensor, torch.Tensor]:
+ &quot;&quot;&quot;Encode chunk.
+
+ Args:
+ speech: Speech audio tensor, shape (batch, time).
+ speech_lengths: Length of each speech sample.
+ cache: State cache dict for streaming inference.
+ **kwargs: Additional keyword arguments.
+ &quot;&quot;&quot;
+ if cache is None:
+ cache = {}
+ &quot;&quot;&quot;Frontend + Encoder. Note that this method is used by asr_inference.py
+ Args:
+ speech: (Batch, Length, ...)
+ speech_lengths: (Batch, )
+ ind: int
+ &quot;&quot;&quot;
+ with autocast(False):
+
+ # Data augmentation
+ if self.specaug is not None and self.training:
+ speech, speech_lengths = self.specaug(speech, speech_lengths)
+
+ # Normalization for feature: e.g. Global-CMVN, Utterance-CMVN
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/paraformer_streaming/model.py#L165" target="_blank" class="src-more">View full source →</a></details></div><div class="mblk"><code>.sampler(encoder_out, encoder_out_lens, ys_pad, ys_pad_lens, pre_acoustic_embeds, chunk_mask)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/paraformer_streaming/model.py#L336" target="_blank" class="gh-link">L336</a><div class="mdoc"><p>Sampler.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>encoder_out</code> — Encoder output tensor.</li>
+<li><code>encoder_out_lens</code> — Encoder output lengths.</li>
+<li><code>ys_pad</code> — TODO.</li>
+<li><code>ys_pad_lens</code> — Lengths of ys_pad.</li>
+<li><code>pre_acoustic_embeds</code> — TODO.</li>
+<li><code>chunk_mask</code> — TODO.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def sampler(
+ self,
+ encoder_out,
+ encoder_out_lens,
+ ys_pad,
+ ys_pad_lens,
+ pre_acoustic_embeds,
+ chunk_mask=None,
+ ):
+
+ &quot;&quot;&quot;Sampler.
+
+ Args:
+ encoder_out: Encoder output tensor.
+ encoder_out_lens: Encoder output lengths.
+ ys_pad: TODO.
+ ys_pad_lens: Lengths of ys_pad.
+ pre_acoustic_embeds: TODO.
+ chunk_mask: TODO.
+ &quot;&quot;&quot;
+ tgt_mask = (~make_pad_mask(ys_pad_lens, maxlen=ys_pad_lens.max())[:, :, None]).to(
+ ys_pad.device
+ )
+ ys_pad_masked = ys_pad * tgt_mask[:, :, 0]
+ if self.share_embedding:
+ ys_pad_embed = self.decoder.output_layer.weight[ys_pad_masked]
+ else:
+ ys_pad_embed = self.decoder.embed(ys_pad_masked)
+ with torch.no_grad():
+ decoder_outs = self.decoder(
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/paraformer_streaming/model.py#L336" target="_blank" class="src-more">View full source →</a></details></div><div class="mblk"><code>.calc_predictor(encoder_out, encoder_out_lens)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/paraformer_streaming/model.py#L392" target="_blank" class="gh-link">L392</a><div class="mdoc"><p>Calc predictor.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>encoder_out</code> — Encoder output tensor.</li>
+<li><code>encoder_out_lens</code> — Encoder output lengths.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def calc_predictor(self, encoder_out, encoder_out_lens):
+
+ &quot;&quot;&quot;Calc predictor.
+
+ Args:
+ encoder_out: Encoder output tensor.
+ encoder_out_lens: Encoder output lengths.
+ &quot;&quot;&quot;
+ encoder_out_mask = (
+ ~make_pad_mask(encoder_out_lens, maxlen=encoder_out.size(1))[:, None, :]
+ ).to(encoder_out.device)
+ mask_chunk_predictor = None
+ if self.encoder.overlap_chunk_cls is not None:
+ mask_chunk_predictor = self.encoder.overlap_chunk_cls.get_mask_chunk_predictor(
+ None, device=encoder_out.device, batch_size=encoder_out.size(0)
+ )
+ mask_shfit_chunk = self.encoder.overlap_chunk_cls.get_mask_shfit_chunk(
+ None, device=encoder_out.device, batch_size=encoder_out.size(0)
+ )
+ encoder_out = encoder_out * mask_shfit_chunk
+ pre_acoustic_embeds, pre_token_length, pre_alphas, pre_peak_index = self.predictor(
+ encoder_out,
+ None,
+ encoder_out_mask,
+ ignore_id=self.ignore_id,
+ mask_chunk_predictor=mask_chunk_predictor,
+ target_label_length=None,
+ )
+ predictor_alignments, predictor_alignments_len = self.predictor.gen_frame_alignments(
+ pre_alphas,
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/paraformer_streaming/model.py#L392" target="_blank" class="src-more">View full source →</a></details></div><div class="mblk"><code>.calc_predictor_chunk(encoder_out, encoder_out_lens, cache, **kwargs)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/paraformer_streaming/model.py#L460" target="_blank" class="gh-link">L460</a><div class="mdoc"><p>Calc predictor chunk.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>encoder_out</code> — Encoder output tensor.</li>
+<li><code>encoder_out_lens</code> — Encoder output lengths.</li>
+<li><code>cache</code> — State cache dict for streaming inference.</li>
+<li><code>**kwargs</code> — Additional keyword arguments.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def calc_predictor_chunk(self, encoder_out, encoder_out_lens, cache=None, **kwargs):
+ &quot;&quot;&quot;Calc predictor chunk.
+
+ Args:
+ encoder_out: Encoder output tensor.
+ encoder_out_lens: Encoder output lengths.
+ cache: State cache dict for streaming inference.
+ **kwargs: Additional keyword arguments.
+ &quot;&quot;&quot;
+ is_final = kwargs.get(&quot;is_final&quot;, False)
+
+ return self.predictor.forward_chunk(encoder_out, cache[&quot;encoder&quot;], is_final=is_final)
+</code></pre></details></div><div class="mblk"><code>.cal_decoder_with_predictor(encoder_out, encoder_out_lens, sematic_embeds, ys_pad_lens)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/paraformer_streaming/model.py#L473" target="_blank" class="gh-link">L473</a><div class="mdoc"><p>Cal decoder with predictor.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>encoder_out</code> — Encoder output tensor.</li>
+<li><code>encoder_out_lens</code> — Encoder output lengths.</li>
+<li><code>sematic_embeds</code> — TODO.</li>
+<li><code>ys_pad_lens</code> — Lengths of ys_pad.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def cal_decoder_with_predictor(
+ self, encoder_out, encoder_out_lens, sematic_embeds, ys_pad_lens
+ ):
+ &quot;&quot;&quot;Cal decoder with predictor.
+
+ Args:
+ encoder_out: Encoder output tensor.
+ encoder_out_lens: Encoder output lengths.
+ sematic_embeds: TODO.
+ ys_pad_lens: Lengths of ys_pad.
+ &quot;&quot;&quot;
+ decoder_outs = self.decoder(
+ encoder_out, encoder_out_lens, sematic_embeds, ys_pad_lens, self.scama_mask
+ )
+ decoder_out = decoder_outs[0]
+ decoder_out = torch.log_softmax(decoder_out, dim=-1)
+ return decoder_out, ys_pad_lens
+</code></pre></details></div><div class="mblk"><code>.cal_decoder_with_predictor_chunk(encoder_out, encoder_out_lens, sematic_embeds, ys_pad_lens, cache)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/paraformer_streaming/model.py#L491" target="_blank" class="gh-link">L491</a><div class="mdoc"><p>Cal decoder with predictor chunk.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>encoder_out</code> — Encoder output tensor.</li>
+<li><code>encoder_out_lens</code> — Encoder output lengths.</li>
+<li><code>sematic_embeds</code> — TODO.</li>
+<li><code>ys_pad_lens</code> — Lengths of ys_pad.</li>
+<li><code>cache</code> — State cache dict for streaming inference.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def cal_decoder_with_predictor_chunk(
+ self, encoder_out, encoder_out_lens, sematic_embeds, ys_pad_lens, cache=None
+ ):
+ &quot;&quot;&quot;Cal decoder with predictor chunk.
+
+ Args:
+ encoder_out: Encoder output tensor.
+ encoder_out_lens: Encoder output lengths.
+ sematic_embeds: TODO.
+ ys_pad_lens: Lengths of ys_pad.
+ cache: State cache dict for streaming inference.
+ &quot;&quot;&quot;
+ decoder_outs = self.decoder.forward_chunk(encoder_out, sematic_embeds, cache[&quot;decoder&quot;])
+ decoder_out = decoder_outs
+ decoder_out = torch.log_softmax(decoder_out, dim=-1)
+ return decoder_out, ys_pad_lens
+</code></pre></details></div><div class="mblk"><code>.init_cache(cache, **kwargs)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/paraformer_streaming/model.py#L508" target="_blank" class="gh-link">L508</a><div class="mdoc"><p>Init cache.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>cache</code> — State cache dict for streaming inference.</li>
+<li><code>**kwargs</code> — Additional keyword arguments.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def init_cache(self, cache: dict = None, **kwargs):
+ &quot;&quot;&quot;Init cache.
+
+ Args:
+ cache: State cache dict for streaming inference.
+ **kwargs: Additional keyword arguments.
+ &quot;&quot;&quot;
+ if cache is None:
+ cache = {}
+ chunk_size = kwargs.get(&quot;chunk_size&quot;, [0, 10, 5])
+ encoder_chunk_look_back = kwargs.get(&quot;encoder_chunk_look_back&quot;, 0)
+ decoder_chunk_look_back = kwargs.get(&quot;decoder_chunk_look_back&quot;, 0)
+ batch_size = 1
+
+ enc_output_size = kwargs[&quot;encoder_conf&quot;][&quot;output_size&quot;]
+ feats_dims = kwargs[&quot;frontend_conf&quot;][&quot;n_mels&quot;] * kwargs[&quot;frontend_conf&quot;][&quot;lfr_m&quot;]
+ cache_encoder = {
+ &quot;start_idx&quot;: 0,
+ &quot;cif_hidden&quot;: torch.zeros((batch_size, 1, enc_output_size)),
+ &quot;cif_alphas&quot;: torch.zeros((batch_size, 1)),
+ &quot;chunk_size&quot;: chunk_size,
+ &quot;encoder_chunk_look_back&quot;: encoder_chunk_look_back,
+ &quot;last_chunk&quot;: False,
+ &quot;opt&quot;: None,
+ &quot;feats&quot;: torch.zeros((batch_size, chunk_size[0] + chunk_size[2], feats_dims)),
+ &quot;tail_chunk&quot;: False,
+ }
+ cache[&quot;encoder&quot;] = cache_encoder
+
+ cache_decoder = {
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/paraformer_streaming/model.py#L508" target="_blank" class="src-more">View full source →</a></details></div><div class="mblk"><code>.generate_chunk(speech, speech_lengths, key, tokenizer, frontend, **kwargs)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/paraformer_streaming/model.py#L549" target="_blank" class="gh-link">L549</a><div class="mdoc"><p>Generate chunk.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>speech</code> — Speech audio tensor, shape (batch, time).</li>
+<li><code>speech_lengths</code> — Length of each speech sample.</li>
+<li><code>key</code> — Sample identifiers.</li>
+<li><code>tokenizer</code> — Tokenizer instance for text encoding/decoding.</li>
+<li><code>frontend</code> — Audio frontend for feature extraction.</li>
+<li><code>**kwargs</code> — Additional keyword arguments.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def generate_chunk(
+ self,
+ speech,
+ speech_lengths=None,
+ key: list = None,
+ tokenizer=None,
+ frontend=None,
+ **kwargs,
+ ):
+ &quot;&quot;&quot;Generate chunk.
+
+ Args:
+ speech: Speech audio tensor, shape (batch, time).
+ speech_lengths: Length of each speech sample.
+ key: Sample identifiers.
+ tokenizer: Tokenizer instance for text encoding/decoding.
+ frontend: Audio frontend for feature extraction.
+ **kwargs: Additional keyword arguments.
+ &quot;&quot;&quot;
+ cache = kwargs.get(&quot;cache&quot;, {})
+ speech = speech.to(device=kwargs[&quot;device&quot;])
+ speech_lengths = speech_lengths.to(device=kwargs[&quot;device&quot;])
+
+ # Encoder
+ encoder_out, encoder_out_lens = self.encode_chunk(
+ speech, speech_lengths, cache=cache, is_final=kwargs.get(&quot;is_final&quot;, False)
+ )
+ if isinstance(encoder_out, tuple):
+ encoder_out = encoder_out[0]
+
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/paraformer_streaming/model.py#L549" target="_blank" class="src-more">View full source →</a></details></div><div class="mblk"><code>.inference(data_in, data_lengths, key, tokenizer, frontend, cache, **kwargs)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/paraformer_streaming/model.py#L647" target="_blank" class="gh-link">L647</a><div class="mdoc"><p>Run inference on input data.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>data_in</code> — Input data (audio samples, file paths, or text).</li>
+<li><code>data_lengths</code> — Lengths of each input sample in the batch.</li>
+<li><code>key</code> — Sample identifiers.</li>
+<li><code>tokenizer</code> — Tokenizer instance for text encoding/decoding.</li>
+<li><code>frontend</code> — Audio frontend for feature extraction.</li>
+<li><code>cache</code> — State cache dict for streaming inference.</li>
+<li><code>**kwargs</code> — Additional keyword arguments.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def inference(
+ self,
+ data_in,
+ data_lengths=None,
+ key: list = None,
+ tokenizer=None,
+ frontend=None,
+ cache: dict = None,
+ **kwargs,
+ ):
+ &quot;&quot;&quot;Run inference on input data.
+
+ Args:
+ data_in: Input data (audio samples, file paths, or text).
+ data_lengths: Lengths of each input sample in the batch.
+ key: Sample identifiers.
+ tokenizer: Tokenizer instance for text encoding/decoding.
+ frontend: Audio frontend for feature extraction.
+ cache: State cache dict for streaming inference.
+ **kwargs: Additional keyword arguments.
+ &quot;&quot;&quot;
+ if cache is None:
+ cache = {}
+
+ # init beamsearch
+ is_use_ctc = kwargs.get(&quot;decoding_ctc_weight&quot;, 0.0) &gt; 0.00001 and self.ctc != None
+ is_use_lm = (
+ kwargs.get(&quot;lm_weight&quot;, 0.0) &gt; 0.00001 and kwargs.get(&quot;lm_file&quot;, None) is not None
+ )
+ if self.beam_search is None and (is_use_lm or is_use_ctc):
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/paraformer_streaming/model.py#L647" target="_blank" class="src-more">View full source →</a></details></div><div class="mblk"><code>.export(**kwargs)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/paraformer_streaming/model.py#L762" target="_blank" class="gh-link">L762</a><div class="mdoc"><p>Export.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>**kwargs</code> — Additional keyword arguments.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def export(self, **kwargs):
+ &quot;&quot;&quot;Export.
+
+ Args:
+ **kwargs: Additional keyword arguments.
+ &quot;&quot;&quot;
+ from .export_meta import export_rebuild_model
+
+ models = export_rebuild_model(model=self, **kwargs)
+ return models
+</code></pre></details></div></div>
+<div class="api-detail" id="e167"><span class="dtag badge-m">method</span><h2><code>ParaformerStreaming.forward(speech, speech_lengths, text, text_lengths, **kwargs)</code></h2><div class="dmod">funasr.models.paraformer_streaming.ParaformerStreaming · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/paraformer_streaming/model.py#L83" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Encoder + Decoder + Calc loss</p>
+<h4>Args:</h4>
+<ul>
+<li><code>speech</code> — (Batch, Length, ...)</li>
+<li><code>speech_lengths</code> — (Batch, )</li>
+<li><code>text</code> — (Batch, Length)</li>
+<li><code>text_lengths</code> — (Batch,)</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def forward(
+ self,
+ speech: torch.Tensor,
+ speech_lengths: torch.Tensor,
+ text: torch.Tensor,
+ text_lengths: torch.Tensor,
+ **kwargs,
+ ) -&gt; Tuple[torch.Tensor, Dict[str, torch.Tensor], torch.Tensor]:
+ &quot;&quot;&quot;Encoder + Decoder + Calc loss
+ Args:
+ speech: (Batch, Length, ...)
+ speech_lengths: (Batch, )
+ text: (Batch, Length)
+ text_lengths: (Batch,)
+ &quot;&quot;&quot;
+ decoding_ind = kwargs.get(&quot;decoding_ind&quot;)
+ if len(text_lengths.size()) &gt; 1:
+ text_lengths = text_lengths[:, 0]
+ if len(speech_lengths.size()) &gt; 1:
+ speech_lengths = speech_lengths[:, 0]
+
+ batch_size = speech.shape[0]
+
+ # Encoder
+ if hasattr(self.encoder, &quot;overlap_chunk_cls&quot;):
+ ind = self.encoder.overlap_chunk_cls.random_choice(self.training, decoding_ind)
+ encoder_out, encoder_out_lens = self.encode(speech, speech_lengths, ind=ind)
+ else:
+ encoder_out, encoder_out_lens = self.encode(speech, speech_lengths)
+
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/paraformer_streaming/model.py#L83" target="_blank" class="src-more">View full source on GitHub →</a></details></div>
+<div class="api-detail" id="e168"><span class="dtag badge-m">method</span><h2><code>ParaformerStreaming.encode_chunk(speech, speech_lengths, cache, **kwargs)</code></h2><div class="dmod">funasr.models.paraformer_streaming.ParaformerStreaming · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/paraformer_streaming/model.py#L165" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Encode chunk.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>speech</code> — Speech audio tensor, shape (batch, time).</li>
+<li><code>speech_lengths</code> — Length of each speech sample.</li>
+<li><code>cache</code> — State cache dict for streaming inference.</li>
+<li><code>**kwargs</code> — Additional keyword arguments.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def encode_chunk(
+ self,
+ speech: torch.Tensor,
+ speech_lengths: torch.Tensor,
+ cache: dict = None,
+ **kwargs,
+ ) -&gt; Tuple[torch.Tensor, torch.Tensor]:
+ &quot;&quot;&quot;Encode chunk.
+
+ Args:
+ speech: Speech audio tensor, shape (batch, time).
+ speech_lengths: Length of each speech sample.
+ cache: State cache dict for streaming inference.
+ **kwargs: Additional keyword arguments.
+ &quot;&quot;&quot;
+ if cache is None:
+ cache = {}
+ &quot;&quot;&quot;Frontend + Encoder. Note that this method is used by asr_inference.py
+ Args:
+ speech: (Batch, Length, ...)
+ speech_lengths: (Batch, )
+ ind: int
+ &quot;&quot;&quot;
+ with autocast(False):
+
+ # Data augmentation
+ if self.specaug is not None and self.training:
+ speech, speech_lengths = self.specaug(speech, speech_lengths)
+
+ # Normalization for feature: e.g. Global-CMVN, Utterance-CMVN
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/paraformer_streaming/model.py#L165" target="_blank" class="src-more">View full source on GitHub →</a></details></div>
+<div class="api-detail" id="e169"><span class="dtag badge-m">method</span><h2><code>ParaformerStreaming.sampler(encoder_out, encoder_out_lens, ys_pad, ys_pad_lens, pre_acoustic_embeds, chunk_mask)</code></h2><div class="dmod">funasr.models.paraformer_streaming.ParaformerStreaming · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/paraformer_streaming/model.py#L336" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Sampler.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>encoder_out</code> — Encoder output tensor.</li>
+<li><code>encoder_out_lens</code> — Encoder output lengths.</li>
+<li><code>ys_pad</code> — TODO.</li>
+<li><code>ys_pad_lens</code> — Lengths of ys_pad.</li>
+<li><code>pre_acoustic_embeds</code> — TODO.</li>
+<li><code>chunk_mask</code> — TODO.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def sampler(
+ self,
+ encoder_out,
+ encoder_out_lens,
+ ys_pad,
+ ys_pad_lens,
+ pre_acoustic_embeds,
+ chunk_mask=None,
+ ):
+
+ &quot;&quot;&quot;Sampler.
+
+ Args:
+ encoder_out: Encoder output tensor.
+ encoder_out_lens: Encoder output lengths.
+ ys_pad: TODO.
+ ys_pad_lens: Lengths of ys_pad.
+ pre_acoustic_embeds: TODO.
+ chunk_mask: TODO.
+ &quot;&quot;&quot;
+ tgt_mask = (~make_pad_mask(ys_pad_lens, maxlen=ys_pad_lens.max())[:, :, None]).to(
+ ys_pad.device
+ )
+ ys_pad_masked = ys_pad * tgt_mask[:, :, 0]
+ if self.share_embedding:
+ ys_pad_embed = self.decoder.output_layer.weight[ys_pad_masked]
+ else:
+ ys_pad_embed = self.decoder.embed(ys_pad_masked)
+ with torch.no_grad():
+ decoder_outs = self.decoder(
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/paraformer_streaming/model.py#L336" target="_blank" class="src-more">View full source on GitHub →</a></details></div>
+<div class="api-detail" id="e170"><span class="dtag badge-m">method</span><h2><code>ParaformerStreaming.calc_predictor(encoder_out, encoder_out_lens)</code></h2><div class="dmod">funasr.models.paraformer_streaming.ParaformerStreaming · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/paraformer_streaming/model.py#L392" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Calc predictor.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>encoder_out</code> — Encoder output tensor.</li>
+<li><code>encoder_out_lens</code> — Encoder output lengths.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def calc_predictor(self, encoder_out, encoder_out_lens):
+
+ &quot;&quot;&quot;Calc predictor.
+
+ Args:
+ encoder_out: Encoder output tensor.
+ encoder_out_lens: Encoder output lengths.
+ &quot;&quot;&quot;
+ encoder_out_mask = (
+ ~make_pad_mask(encoder_out_lens, maxlen=encoder_out.size(1))[:, None, :]
+ ).to(encoder_out.device)
+ mask_chunk_predictor = None
+ if self.encoder.overlap_chunk_cls is not None:
+ mask_chunk_predictor = self.encoder.overlap_chunk_cls.get_mask_chunk_predictor(
+ None, device=encoder_out.device, batch_size=encoder_out.size(0)
+ )
+ mask_shfit_chunk = self.encoder.overlap_chunk_cls.get_mask_shfit_chunk(
+ None, device=encoder_out.device, batch_size=encoder_out.size(0)
+ )
+ encoder_out = encoder_out * mask_shfit_chunk
+ pre_acoustic_embeds, pre_token_length, pre_alphas, pre_peak_index = self.predictor(
+ encoder_out,
+ None,
+ encoder_out_mask,
+ ignore_id=self.ignore_id,
+ mask_chunk_predictor=mask_chunk_predictor,
+ target_label_length=None,
+ )
+ predictor_alignments, predictor_alignments_len = self.predictor.gen_frame_alignments(
+ pre_alphas,
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/paraformer_streaming/model.py#L392" target="_blank" class="src-more">View full source on GitHub →</a></details></div>
+<div class="api-detail" id="e171"><span class="dtag badge-m">method</span><h2><code>ParaformerStreaming.calc_predictor_chunk(encoder_out, encoder_out_lens, cache, **kwargs)</code></h2><div class="dmod">funasr.models.paraformer_streaming.ParaformerStreaming · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/paraformer_streaming/model.py#L460" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Calc predictor chunk.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>encoder_out</code> — Encoder output tensor.</li>
+<li><code>encoder_out_lens</code> — Encoder output lengths.</li>
+<li><code>cache</code> — State cache dict for streaming inference.</li>
+<li><code>**kwargs</code> — Additional keyword arguments.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def calc_predictor_chunk(self, encoder_out, encoder_out_lens, cache=None, **kwargs):
+ &quot;&quot;&quot;Calc predictor chunk.
+
+ Args:
+ encoder_out: Encoder output tensor.
+ encoder_out_lens: Encoder output lengths.
+ cache: State cache dict for streaming inference.
+ **kwargs: Additional keyword arguments.
+ &quot;&quot;&quot;
+ is_final = kwargs.get(&quot;is_final&quot;, False)
+
+ return self.predictor.forward_chunk(encoder_out, cache[&quot;encoder&quot;], is_final=is_final)
+</code></pre></details></div>
+<div class="api-detail" id="e172"><span class="dtag badge-m">method</span><h2><code>ParaformerStreaming.cal_decoder_with_predictor(encoder_out, encoder_out_lens, sematic_embeds, ys_pad_lens)</code></h2><div class="dmod">funasr.models.paraformer_streaming.ParaformerStreaming · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/paraformer_streaming/model.py#L473" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Cal decoder with predictor.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>encoder_out</code> — Encoder output tensor.</li>
+<li><code>encoder_out_lens</code> — Encoder output lengths.</li>
+<li><code>sematic_embeds</code> — TODO.</li>
+<li><code>ys_pad_lens</code> — Lengths of ys_pad.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def cal_decoder_with_predictor(
+ self, encoder_out, encoder_out_lens, sematic_embeds, ys_pad_lens
+ ):
+ &quot;&quot;&quot;Cal decoder with predictor.
+
+ Args:
+ encoder_out: Encoder output tensor.
+ encoder_out_lens: Encoder output lengths.
+ sematic_embeds: TODO.
+ ys_pad_lens: Lengths of ys_pad.
+ &quot;&quot;&quot;
+ decoder_outs = self.decoder(
+ encoder_out, encoder_out_lens, sematic_embeds, ys_pad_lens, self.scama_mask
+ )
+ decoder_out = decoder_outs[0]
+ decoder_out = torch.log_softmax(decoder_out, dim=-1)
+ return decoder_out, ys_pad_lens
+</code></pre></details></div>
+<div class="api-detail" id="e173"><span class="dtag badge-m">method</span><h2><code>ParaformerStreaming.cal_decoder_with_predictor_chunk(encoder_out, encoder_out_lens, sematic_embeds, ys_pad_lens, cache)</code></h2><div class="dmod">funasr.models.paraformer_streaming.ParaformerStreaming · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/paraformer_streaming/model.py#L491" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Cal decoder with predictor chunk.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>encoder_out</code> — Encoder output tensor.</li>
+<li><code>encoder_out_lens</code> — Encoder output lengths.</li>
+<li><code>sematic_embeds</code> — TODO.</li>
+<li><code>ys_pad_lens</code> — Lengths of ys_pad.</li>
+<li><code>cache</code> — State cache dict for streaming inference.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def cal_decoder_with_predictor_chunk(
+ self, encoder_out, encoder_out_lens, sematic_embeds, ys_pad_lens, cache=None
+ ):
+ &quot;&quot;&quot;Cal decoder with predictor chunk.
+
+ Args:
+ encoder_out: Encoder output tensor.
+ encoder_out_lens: Encoder output lengths.
+ sematic_embeds: TODO.
+ ys_pad_lens: Lengths of ys_pad.
+ cache: State cache dict for streaming inference.
+ &quot;&quot;&quot;
+ decoder_outs = self.decoder.forward_chunk(encoder_out, sematic_embeds, cache[&quot;decoder&quot;])
+ decoder_out = decoder_outs
+ decoder_out = torch.log_softmax(decoder_out, dim=-1)
+ return decoder_out, ys_pad_lens
+</code></pre></details></div>
+<div class="api-detail" id="e174"><span class="dtag badge-m">method</span><h2><code>ParaformerStreaming.init_cache(cache, **kwargs)</code></h2><div class="dmod">funasr.models.paraformer_streaming.ParaformerStreaming · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/paraformer_streaming/model.py#L508" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Init cache.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>cache</code> — State cache dict for streaming inference.</li>
+<li><code>**kwargs</code> — Additional keyword arguments.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def init_cache(self, cache: dict = None, **kwargs):
+ &quot;&quot;&quot;Init cache.
+
+ Args:
+ cache: State cache dict for streaming inference.
+ **kwargs: Additional keyword arguments.
+ &quot;&quot;&quot;
+ if cache is None:
+ cache = {}
+ chunk_size = kwargs.get(&quot;chunk_size&quot;, [0, 10, 5])
+ encoder_chunk_look_back = kwargs.get(&quot;encoder_chunk_look_back&quot;, 0)
+ decoder_chunk_look_back = kwargs.get(&quot;decoder_chunk_look_back&quot;, 0)
+ batch_size = 1
+
+ enc_output_size = kwargs[&quot;encoder_conf&quot;][&quot;output_size&quot;]
+ feats_dims = kwargs[&quot;frontend_conf&quot;][&quot;n_mels&quot;] * kwargs[&quot;frontend_conf&quot;][&quot;lfr_m&quot;]
+ cache_encoder = {
+ &quot;start_idx&quot;: 0,
+ &quot;cif_hidden&quot;: torch.zeros((batch_size, 1, enc_output_size)),
+ &quot;cif_alphas&quot;: torch.zeros((batch_size, 1)),
+ &quot;chunk_size&quot;: chunk_size,
+ &quot;encoder_chunk_look_back&quot;: encoder_chunk_look_back,
+ &quot;last_chunk&quot;: False,
+ &quot;opt&quot;: None,
+ &quot;feats&quot;: torch.zeros((batch_size, chunk_size[0] + chunk_size[2], feats_dims)),
+ &quot;tail_chunk&quot;: False,
+ }
+ cache[&quot;encoder&quot;] = cache_encoder
+
+ cache_decoder = {
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/paraformer_streaming/model.py#L508" target="_blank" class="src-more">View full source on GitHub →</a></details></div>
+<div class="api-detail" id="e175"><span class="dtag badge-m">method</span><h2><code>ParaformerStreaming.generate_chunk(speech, speech_lengths, key, tokenizer, frontend, **kwargs)</code></h2><div class="dmod">funasr.models.paraformer_streaming.ParaformerStreaming · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/paraformer_streaming/model.py#L549" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Generate chunk.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>speech</code> — Speech audio tensor, shape (batch, time).</li>
+<li><code>speech_lengths</code> — Length of each speech sample.</li>
+<li><code>key</code> — Sample identifiers.</li>
+<li><code>tokenizer</code> — Tokenizer instance for text encoding/decoding.</li>
+<li><code>frontend</code> — Audio frontend for feature extraction.</li>
+<li><code>**kwargs</code> — Additional keyword arguments.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def generate_chunk(
+ self,
+ speech,
+ speech_lengths=None,
+ key: list = None,
+ tokenizer=None,
+ frontend=None,
+ **kwargs,
+ ):
+ &quot;&quot;&quot;Generate chunk.
+
+ Args:
+ speech: Speech audio tensor, shape (batch, time).
+ speech_lengths: Length of each speech sample.
+ key: Sample identifiers.
+ tokenizer: Tokenizer instance for text encoding/decoding.
+ frontend: Audio frontend for feature extraction.
+ **kwargs: Additional keyword arguments.
+ &quot;&quot;&quot;
+ cache = kwargs.get(&quot;cache&quot;, {})
+ speech = speech.to(device=kwargs[&quot;device&quot;])
+ speech_lengths = speech_lengths.to(device=kwargs[&quot;device&quot;])
+
+ # Encoder
+ encoder_out, encoder_out_lens = self.encode_chunk(
+ speech, speech_lengths, cache=cache, is_final=kwargs.get(&quot;is_final&quot;, False)
+ )
+ if isinstance(encoder_out, tuple):
+ encoder_out = encoder_out[0]
+
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/paraformer_streaming/model.py#L549" target="_blank" class="src-more">View full source on GitHub →</a></details></div>
+<div class="api-detail" id="e176"><span class="dtag badge-m">method</span><h2><code>ParaformerStreaming.inference(data_in, data_lengths, key, tokenizer, frontend, cache, **kwargs)</code></h2><div class="dmod">funasr.models.paraformer_streaming.ParaformerStreaming · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/paraformer_streaming/model.py#L647" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Run inference on input data.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>data_in</code> — Input data (audio samples, file paths, or text).</li>
+<li><code>data_lengths</code> — Lengths of each input sample in the batch.</li>
+<li><code>key</code> — Sample identifiers.</li>
+<li><code>tokenizer</code> — Tokenizer instance for text encoding/decoding.</li>
+<li><code>frontend</code> — Audio frontend for feature extraction.</li>
+<li><code>cache</code> — State cache dict for streaming inference.</li>
+<li><code>**kwargs</code> — Additional keyword arguments.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def inference(
+ self,
+ data_in,
+ data_lengths=None,
+ key: list = None,
+ tokenizer=None,
+ frontend=None,
+ cache: dict = None,
+ **kwargs,
+ ):
+ &quot;&quot;&quot;Run inference on input data.
+
+ Args:
+ data_in: Input data (audio samples, file paths, or text).
+ data_lengths: Lengths of each input sample in the batch.
+ key: Sample identifiers.
+ tokenizer: Tokenizer instance for text encoding/decoding.
+ frontend: Audio frontend for feature extraction.
+ cache: State cache dict for streaming inference.
+ **kwargs: Additional keyword arguments.
+ &quot;&quot;&quot;
+ if cache is None:
+ cache = {}
+
+ # init beamsearch
+ is_use_ctc = kwargs.get(&quot;decoding_ctc_weight&quot;, 0.0) &gt; 0.00001 and self.ctc != None
+ is_use_lm = (
+ kwargs.get(&quot;lm_weight&quot;, 0.0) &gt; 0.00001 and kwargs.get(&quot;lm_file&quot;, None) is not None
+ )
+ if self.beam_search is None and (is_use_lm or is_use_ctc):
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/paraformer_streaming/model.py#L647" target="_blank" class="src-more">View full source on GitHub →</a></details></div>
+<div class="api-detail" id="e177"><span class="dtag badge-m">method</span><h2><code>ParaformerStreaming.export(**kwargs)</code></h2><div class="dmod">funasr.models.paraformer_streaming.ParaformerStreaming · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/paraformer_streaming/model.py#L762" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Export.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>**kwargs</code> — Additional keyword arguments.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def export(self, **kwargs):
+ &quot;&quot;&quot;Export.
+
+ Args:
+ **kwargs: Additional keyword arguments.
+ &quot;&quot;&quot;
+ from .export_meta import export_rebuild_model
+
+ models = export_rebuild_model(model=self, **kwargs)
+ return models
+</code></pre></details></div>
+<div class="api-detail" id="e178"><span class="dtag badge-c">class</span><h2>Paraformer</h2><div class="dmod">funasr.models.paraformer_v2_community · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/paraformer_v2_community/model.py#L31" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><ul>
+<li><code>Author</code> — Speech Lab of DAMO Academy, Alibaba Group</li>
+<li><code>Paraformer</code> — Fast and Accurate Parallel Transformer for Non-autoregressive End-to-End Speech Recognition</li>
+</ul>
+<p>https://arxiv.org/abs/2206.08317</p>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code>class Paraformer(torch.nn.Module):
+ &quot;&quot;&quot;
+ Author: Speech Lab of DAMO Academy, Alibaba Group
+ Paraformer: Fast and Accurate Parallel Transformer for Non-autoregressive End-to-End Speech Recognition
+ https://arxiv.org/abs/2206.08317
+ &quot;&quot;&quot;
+
+ def __init__(
+ self,
+ specaug: Optional[str] = None,
+ specaug_conf: Optional[Dict] = None,
+ normalize: str = None,
+ normalize_conf: Optional[Dict] = None,
+ encoder: str = None,
+ encoder_conf: Optional[Dict] = None,
+ decoder: str = None,
+ decoder_conf: Optional[Dict] = None,
+ ctc: str = None,
+ ctc_conf: Optional[Dict] = None,
+ ctc_weight: float = 0.5,
+ input_size: int = 80,
+ vocab_size: int = -1,
+ ignore_id: int = -1,
+ blank_id: int = 0,
+ sos: int = 1,
+ eos: int = 2,
+ lsm_weight: float = 0.0,
+ length_normalized_loss: bool = False,
+ # report_cer: bool = True,
+ # report_wer: bool = True,
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/paraformer_v2_community/model.py#L31" target="_blank" class="src-more">View full source on GitHub →</a></details><h3 class="mt">Methods</h3><div class="mblk"><code>.forward(speech, speech_lengths, text, text_lengths, **kwargs)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/paraformer_v2_community/model.py#L182" target="_blank" class="gh-link">L182</a><div class="mdoc"><p>Encoder + Decoder + Calc loss</p>
+<h4>Args:</h4>
+<ul>
+<li><code>speech</code> — (Batch, Length, ...)</li>
+<li><code>speech_lengths</code> — (Batch, )</li>
+<li><code>text</code> — (Batch, Length)</li>
+<li><code>text_lengths</code> — (Batch,)</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def forward(
+ self,
+ speech: torch.Tensor,
+ speech_lengths: torch.Tensor,
+ text: torch.Tensor,
+ text_lengths: torch.Tensor,
+ **kwargs,
+ ) -&gt; Tuple[torch.Tensor, Dict[str, torch.Tensor], torch.Tensor]:
+ &quot;&quot;&quot;Encoder + Decoder + Calc loss
+ Args:
+ speech: (Batch, Length, ...)
+ speech_lengths: (Batch, )
+ text: (Batch, Length)
+ text_lengths: (Batch,)
+ &quot;&quot;&quot;
+ if len(text_lengths.size()) &gt; 1:
+ text_lengths = text_lengths[:, 0]
+ if len(speech_lengths.size()) &gt; 1:
+ speech_lengths = speech_lengths[:, 0]
+
+ batch_size = speech.shape[0]
+
+ # Encoder
+ encoder_out, encoder_out_lens = self.encode(speech, speech_lengths)
+
+ loss_ctc, cer_ctc = None, None
+ loss_pre = None
+ stats = dict()
+
+ # decoder: CTC branch
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/paraformer_v2_community/model.py#L182" target="_blank" class="src-more">View full source →</a></details></div><div class="mblk"><code>.encode(speech, speech_lengths, **kwargs)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/paraformer_v2_community/model.py#L250" target="_blank" class="gh-link">L250</a><div class="mdoc"><p>Encoder. Note that this method is used by asr_inference.py</p>
+<h4>Args:</h4>
+<ul>
+<li><code>speech</code> — (Batch, Length, ...)</li>
+<li><code>speech_lengths</code> — (Batch, )</li>
+<li><code>ind</code> — int</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def encode(
+ self,
+ speech: torch.Tensor,
+ speech_lengths: torch.Tensor,
+ **kwargs,
+ ) -&gt; Tuple[torch.Tensor, torch.Tensor]:
+ &quot;&quot;&quot;Encoder. Note that this method is used by asr_inference.py
+ Args:
+ speech: (Batch, Length, ...)
+ speech_lengths: (Batch, )
+ ind: int
+ &quot;&quot;&quot;
+ with autocast(False):
+
+ # Data augmentation
+ if self.specaug is not None and self.training:
+ speech, speech_lengths = self.specaug(speech, speech_lengths)
+
+ # Normalization for feature: e.g. Global-CMVN, Utterance-CMVN
+ if self.normalize is not None:
+ speech, speech_lengths = self.normalize(speech, speech_lengths)
+
+ # Forward encoder
+ encoder_out, encoder_out_lens, _ = self.encoder(speech, speech_lengths)
+ if isinstance(encoder_out, tuple):
+ encoder_out = encoder_out[0]
+
+ return encoder_out, encoder_out_lens
+</code></pre></details></div><div class="mblk"><code>.map_alignment_to_target_index(align_path, blank_id)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/paraformer_v2_community/model.py#L366" target="_blank" class="gh-link">L366</a><div class="mdoc"><p>Robustly map CTC alignment path (Token IDs) to Target Indices.</p>
+<p>Logic:</p>
+<p>Detect boundaries where a new token segment begins.</p>
+<p>A segment starts if the current frame is a Token AND it is different from the previous frame</p>
+<p>(considering CTC topology where repeats are separated by blanks or are distinct tokens).</p>
+<p>Example:</p>
+<ul>
+<li><code>Text</code> — [A, B]</li>
+</ul>
+<p>Align Path: [A, A, _, B, B]</p>
+<ul>
+<li><code>Output</code> — [0, 0, -1, 1, 1]</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def map_alignment_to_target_index(self, align_path, blank_id):
+ &quot;&quot;&quot;
+ Robustly map CTC alignment path (Token IDs) to Target Indices.
+
+ Logic:
+ Detect boundaries where a new token segment begins.
+ A segment starts if the current frame is a Token AND it is different from the previous frame
+ (considering CTC topology where repeats are separated by blanks or are distinct tokens).
+
+ Example:
+ Text: [A, B]
+ Align Path: [A, A, _, B, B]
+ Output: [0, 0, -1, 1, 1]
+ &quot;&quot;&quot;
+ # 1. Identify where the path is NOT blank
+ is_token = align_path != blank_id
+
+ # 2. Identify transitions
+ prev_path = torch.roll(align_path, 1)
+ # Handle the very first frame: if it&#x27;s a token, it must be the start of segment 0.
+ prev_path[0] = blank_id # force mismatch for the first element
+
+ # A new segment starts if: It&#x27;s a token AND (it differs from prev OR prev was blank)
+ # Note: If align_path[i] == align_path[i-1] (and not blank), it&#x27;s the same segment.
+ new_segment_start = is_token &amp; (align_path != prev_path)
+
+ # 3. Cumulative sum to assign indices (1..U)
+ segment_ids = torch.cumsum(new_segment_start.long(), dim=0) - 1
+
+ # 4. Mask out blank positions with -1
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/paraformer_v2_community/model.py#L366" target="_blank" class="src-more">View full source →</a></details></div><div class="mblk"><code>.force_align(ctc_probs, y, blank_id)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/paraformer_v2_community/model.py#L399" target="_blank" class="gh-link">L399</a><div class="mdoc"><p>ctc forced alignment.</p>
+<h4>Args:</h4>
+<p>torch.Tensor ctc_probs: hidden state sequence, 2d tensor (T, D)</p>
+<p>torch.Tensor y: id sequence tensor 1d tensor (L)</p>
+<p>int blank_id: blank symbol index</p>
+<h4>Returns:</h4>
+<ul>
+<li><code>torch.Tensor</code> — alignment result</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def force_align(self, ctc_probs: torch.Tensor, y: torch.Tensor, blank_id=0) -&gt; list:
+ &quot;&quot;&quot;ctc forced alignment.
+
+ Args:
+ torch.Tensor ctc_probs: hidden state sequence, 2d tensor (T, D)
+ torch.Tensor y: id sequence tensor 1d tensor (L)
+ int blank_id: blank symbol index
+ Returns:
+ torch.Tensor: alignment result
+ &quot;&quot;&quot;
+ ctc_probs = ctc_probs[None].cpu()
+ y = y[None].cpu()
+ alignments, _ = torchaudio.functional.forced_align(ctc_probs, y, blank=blank_id)
+ return alignments[0]
+</code></pre></details></div><div class="mblk"><code>.average_repeats_training(ctc_probs, target_idx_path, target_len)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/paraformer_v2_community/model.py#L414" target="_blank" class="gh-link">L414</a><div class="mdoc"><p>Aggregates frames belonging to the same target index using scatter_add.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>ctc_probs</code> — [T, V]</li>
+<li><code>target_idx_path</code> — [T], values in [-1, 0, ... U-1]</li>
+<li><code>target_len</code> — U</li>
+</ul>
+<h4>Returns:</h4>
+<ul>
+<li><code>compressed</code> — [U, V]</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def average_repeats_training(self, ctc_probs, target_idx_path, target_len):
+ &quot;&quot;&quot;
+ Aggregates frames belonging to the same target index using scatter_add.
+
+ Args:
+ ctc_probs: [T, V]
+ target_idx_path: [T], values in [-1, 0, ... U-1]
+ target_len: U
+ Returns:
+ compressed: [U, V]
+ &quot;&quot;&quot;
+ U = target_len
+ V = ctc_probs.size(1)
+
+ compressed = torch.zeros((U, V), device=ctc_probs.device, dtype=ctc_probs.dtype)
+ counts = torch.zeros((U, 1), device=ctc_probs.device, dtype=ctc_probs.dtype)
+
+ # Filter valid frames (non-blank)
+ mask = target_idx_path != -1
+ valid_indices = target_idx_path[mask] # [T_valid]
+ valid_probs = ctc_probs[mask] # [T_valid, V]
+
+ if valid_indices.numel() == 0:
+ return compressed
+
+ # Scatter Add Probs
+ index_expanded = valid_indices.unsqueeze(1).repeat(1, V)
+ compressed.scatter_add_(0, index_expanded, valid_probs)
+
+ # Scatter Add Counts
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/paraformer_v2_community/model.py#L414" target="_blank" class="src-more">View full source →</a></details></div><div class="mblk"><code>.average_repeats_inference(ctc_probs, greedy_path)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/paraformer_v2_community/model.py#L451" target="_blank" class="gh-link">L451</a><div class="mdoc"><h4>Returns:</h4>
+<ul>
+<li><code>merged_probs</code> — [U&#x27;, V]</li>
+<li><code>timestamps</code> — List[Tuple[int, int]] -&gt; [(start_frame, end_frame), ...]</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def average_repeats_inference(self, ctc_probs, greedy_path):
+ &quot;&quot;&quot;
+ Returns:
+ merged_probs: [U&#x27;, V]
+ timestamps: List[Tuple[int, int]] -&gt; [(start_frame, end_frame), ...]
+ &quot;&quot;&quot;
+ if greedy_path.numel() == 0:
+ return torch.zeros((0, ctc_probs.size(1)), device=ctc_probs.device)
+
+ # Find consecutive segments in the greedy path
+ unique_tokens, counts = torch.unique_consecutive(greedy_path, return_counts=True)
+
+ # Compute start and end indices for each segment
+ end_indices = torch.cumsum(counts, dim=0)
+ start_indices = torch.cat([torch.tensor([0], device=counts.device), end_indices[:-1]])
+
+ merged_probs = []
+
+ for i, token in enumerate(unique_tokens):
+ if token != self.blank_id:
+ start = start_indices[i].item()
+ end = end_indices[i].item()
+
+ # Extract and average probabilities for the decoder
+ avg_prob = ctc_probs[start:end].mean(dim=0)
+ merged_probs.append(avg_prob)
+
+
+ if not merged_probs:
+ return torch.zeros((0, ctc_probs.size(1)), device=ctc_probs.device)
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/paraformer_v2_community/model.py#L451" target="_blank" class="src-more">View full source →</a></details></div><div class="mblk"><code>.inference(data_in, data_lengths, key, tokenizer, frontend, **kwargs)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/paraformer_v2_community/model.py#L484" target="_blank" class="gh-link">L484</a><div class="mdoc"><p>Run inference on input data.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>data_in</code> — Input data (audio samples, file paths, or text).</li>
+<li><code>data_lengths</code> — Lengths of each input sample in the batch.</li>
+<li><code>key</code> — Sample identifiers.</li>
+<li><code>tokenizer</code> — Tokenizer instance for text encoding/decoding.</li>
+<li><code>frontend</code> — Audio frontend for feature extraction.</li>
+<li><code>**kwargs</code> — Additional keyword arguments.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def inference(
+ self,
+ data_in,
+ data_lengths=None,
+ key: list = None,
+ tokenizer=None,
+ frontend=None,
+ **kwargs,
+ ):
+
+ &quot;&quot;&quot;Run inference on input data.
+
+ Args:
+ data_in: Input data (audio samples, file paths, or text).
+ data_lengths: Lengths of each input sample in the batch.
+ key: Sample identifiers.
+ tokenizer: Tokenizer instance for text encoding/decoding.
+ frontend: Audio frontend for feature extraction.
+ **kwargs: Additional keyword arguments.
+ &quot;&quot;&quot;
+ meta_data = {}
+ if (
+ isinstance(data_in, torch.Tensor) and kwargs.get(&quot;data_type&quot;, &quot;sound&quot;) == &quot;fbank&quot;
+ ): # fbank
+ speech, speech_lengths = data_in, data_lengths
+ if len(speech.shape) &lt; 3:
+ speech = speech[None, :, :]
+ if speech_lengths is not None:
+ speech_lengths = speech_lengths.squeeze(-1)
+ else:
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/paraformer_v2_community/model.py#L484" target="_blank" class="src-more">View full source →</a></details></div><div class="mblk"><code>.export(**kwargs)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/paraformer_v2_community/model.py#L592" target="_blank" class="gh-link">L592</a><div class="mdoc"><p>Export.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>**kwargs</code> — Additional keyword arguments.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def export(self, **kwargs):
+ &quot;&quot;&quot;Export.
+
+ Args:
+ **kwargs: Additional keyword arguments.
+ &quot;&quot;&quot;
+ from .export_meta import export_rebuild_model
+
+ if &quot;max_seq_len&quot; not in kwargs:
+ kwargs[&quot;max_seq_len&quot;] = 512
+ models = export_rebuild_model(model=self, **kwargs)
+ return models
+</code></pre></details></div></div>
+<div class="api-detail" id="e179"><span class="dtag badge-m">method</span><h2><code>Paraformer.forward(speech, speech_lengths, text, text_lengths, **kwargs)</code></h2><div class="dmod">funasr.models.paraformer_v2_community.Paraformer · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/paraformer_v2_community/model.py#L182" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Encoder + Decoder + Calc loss</p>
+<h4>Args:</h4>
+<ul>
+<li><code>speech</code> — (Batch, Length, ...)</li>
+<li><code>speech_lengths</code> — (Batch, )</li>
+<li><code>text</code> — (Batch, Length)</li>
+<li><code>text_lengths</code> — (Batch,)</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def forward(
+ self,
+ speech: torch.Tensor,
+ speech_lengths: torch.Tensor,
+ text: torch.Tensor,
+ text_lengths: torch.Tensor,
+ **kwargs,
+ ) -&gt; Tuple[torch.Tensor, Dict[str, torch.Tensor], torch.Tensor]:
+ &quot;&quot;&quot;Encoder + Decoder + Calc loss
+ Args:
+ speech: (Batch, Length, ...)
+ speech_lengths: (Batch, )
+ text: (Batch, Length)
+ text_lengths: (Batch,)
+ &quot;&quot;&quot;
+ if len(text_lengths.size()) &gt; 1:
+ text_lengths = text_lengths[:, 0]
+ if len(speech_lengths.size()) &gt; 1:
+ speech_lengths = speech_lengths[:, 0]
+
+ batch_size = speech.shape[0]
+
+ # Encoder
+ encoder_out, encoder_out_lens = self.encode(speech, speech_lengths)
+
+ loss_ctc, cer_ctc = None, None
+ loss_pre = None
+ stats = dict()
+
+ # decoder: CTC branch
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/paraformer_v2_community/model.py#L182" target="_blank" class="src-more">View full source on GitHub →</a></details></div>
+<div class="api-detail" id="e180"><span class="dtag badge-m">method</span><h2><code>Paraformer.encode(speech, speech_lengths, **kwargs)</code></h2><div class="dmod">funasr.models.paraformer_v2_community.Paraformer · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/paraformer_v2_community/model.py#L250" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Encoder. Note that this method is used by asr_inference.py</p>
+<h4>Args:</h4>
+<ul>
+<li><code>speech</code> — (Batch, Length, ...)</li>
+<li><code>speech_lengths</code> — (Batch, )</li>
+<li><code>ind</code> — int</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def encode(
+ self,
+ speech: torch.Tensor,
+ speech_lengths: torch.Tensor,
+ **kwargs,
+ ) -&gt; Tuple[torch.Tensor, torch.Tensor]:
+ &quot;&quot;&quot;Encoder. Note that this method is used by asr_inference.py
+ Args:
+ speech: (Batch, Length, ...)
+ speech_lengths: (Batch, )
+ ind: int
+ &quot;&quot;&quot;
+ with autocast(False):
+
+ # Data augmentation
+ if self.specaug is not None and self.training:
+ speech, speech_lengths = self.specaug(speech, speech_lengths)
+
+ # Normalization for feature: e.g. Global-CMVN, Utterance-CMVN
+ if self.normalize is not None:
+ speech, speech_lengths = self.normalize(speech, speech_lengths)
+
+ # Forward encoder
+ encoder_out, encoder_out_lens, _ = self.encoder(speech, speech_lengths)
+ if isinstance(encoder_out, tuple):
+ encoder_out = encoder_out[0]
+
+ return encoder_out, encoder_out_lens
+</code></pre></details></div>
+<div class="api-detail" id="e181"><span class="dtag badge-m">method</span><h2><code>Paraformer.map_alignment_to_target_index(align_path, blank_id)</code></h2><div class="dmod">funasr.models.paraformer_v2_community.Paraformer · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/paraformer_v2_community/model.py#L366" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Robustly map CTC alignment path (Token IDs) to Target Indices.</p>
+<p>Logic:</p>
+<p>Detect boundaries where a new token segment begins.</p>
+<p>A segment starts if the current frame is a Token AND it is different from the previous frame</p>
+<p>(considering CTC topology where repeats are separated by blanks or are distinct tokens).</p>
+<p>Example:</p>
+<ul>
+<li><code>Text</code> — [A, B]</li>
+</ul>
+<p>Align Path: [A, A, _, B, B]</p>
+<ul>
+<li><code>Output</code> — [0, 0, -1, 1, 1]</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def map_alignment_to_target_index(self, align_path, blank_id):
+ &quot;&quot;&quot;
+ Robustly map CTC alignment path (Token IDs) to Target Indices.
+
+ Logic:
+ Detect boundaries where a new token segment begins.
+ A segment starts if the current frame is a Token AND it is different from the previous frame
+ (considering CTC topology where repeats are separated by blanks or are distinct tokens).
+
+ Example:
+ Text: [A, B]
+ Align Path: [A, A, _, B, B]
+ Output: [0, 0, -1, 1, 1]
+ &quot;&quot;&quot;
+ # 1. Identify where the path is NOT blank
+ is_token = align_path != blank_id
+
+ # 2. Identify transitions
+ prev_path = torch.roll(align_path, 1)
+ # Handle the very first frame: if it&#x27;s a token, it must be the start of segment 0.
+ prev_path[0] = blank_id # force mismatch for the first element
+
+ # A new segment starts if: It&#x27;s a token AND (it differs from prev OR prev was blank)
+ # Note: If align_path[i] == align_path[i-1] (and not blank), it&#x27;s the same segment.
+ new_segment_start = is_token &amp; (align_path != prev_path)
+
+ # 3. Cumulative sum to assign indices (1..U)
+ segment_ids = torch.cumsum(new_segment_start.long(), dim=0) - 1
+
+ # 4. Mask out blank positions with -1
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/paraformer_v2_community/model.py#L366" target="_blank" class="src-more">View full source on GitHub →</a></details></div>
+<div class="api-detail" id="e182"><span class="dtag badge-m">method</span><h2><code>Paraformer.force_align(ctc_probs, y, blank_id)</code></h2><div class="dmod">funasr.models.paraformer_v2_community.Paraformer · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/paraformer_v2_community/model.py#L399" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>ctc forced alignment.</p>
+<h4>Args:</h4>
+<p>torch.Tensor ctc_probs: hidden state sequence, 2d tensor (T, D)</p>
+<p>torch.Tensor y: id sequence tensor 1d tensor (L)</p>
+<p>int blank_id: blank symbol index</p>
+<h4>Returns:</h4>
+<ul>
+<li><code>torch.Tensor</code> — alignment result</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def force_align(self, ctc_probs: torch.Tensor, y: torch.Tensor, blank_id=0) -&gt; list:
+ &quot;&quot;&quot;ctc forced alignment.
+
+ Args:
+ torch.Tensor ctc_probs: hidden state sequence, 2d tensor (T, D)
+ torch.Tensor y: id sequence tensor 1d tensor (L)
+ int blank_id: blank symbol index
+ Returns:
+ torch.Tensor: alignment result
+ &quot;&quot;&quot;
+ ctc_probs = ctc_probs[None].cpu()
+ y = y[None].cpu()
+ alignments, _ = torchaudio.functional.forced_align(ctc_probs, y, blank=blank_id)
+ return alignments[0]
+</code></pre></details></div>
+<div class="api-detail" id="e183"><span class="dtag badge-m">method</span><h2><code>Paraformer.average_repeats_training(ctc_probs, target_idx_path, target_len)</code></h2><div class="dmod">funasr.models.paraformer_v2_community.Paraformer · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/paraformer_v2_community/model.py#L414" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Aggregates frames belonging to the same target index using scatter_add.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>ctc_probs</code> — [T, V]</li>
+<li><code>target_idx_path</code> — [T], values in [-1, 0, ... U-1]</li>
+<li><code>target_len</code> — U</li>
+</ul>
+<h4>Returns:</h4>
+<ul>
+<li><code>compressed</code> — [U, V]</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def average_repeats_training(self, ctc_probs, target_idx_path, target_len):
+ &quot;&quot;&quot;
+ Aggregates frames belonging to the same target index using scatter_add.
+
+ Args:
+ ctc_probs: [T, V]
+ target_idx_path: [T], values in [-1, 0, ... U-1]
+ target_len: U
+ Returns:
+ compressed: [U, V]
+ &quot;&quot;&quot;
+ U = target_len
+ V = ctc_probs.size(1)
+
+ compressed = torch.zeros((U, V), device=ctc_probs.device, dtype=ctc_probs.dtype)
+ counts = torch.zeros((U, 1), device=ctc_probs.device, dtype=ctc_probs.dtype)
+
+ # Filter valid frames (non-blank)
+ mask = target_idx_path != -1
+ valid_indices = target_idx_path[mask] # [T_valid]
+ valid_probs = ctc_probs[mask] # [T_valid, V]
+
+ if valid_indices.numel() == 0:
+ return compressed
+
+ # Scatter Add Probs
+ index_expanded = valid_indices.unsqueeze(1).repeat(1, V)
+ compressed.scatter_add_(0, index_expanded, valid_probs)
+
+ # Scatter Add Counts
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/paraformer_v2_community/model.py#L414" target="_blank" class="src-more">View full source on GitHub →</a></details></div>
+<div class="api-detail" id="e184"><span class="dtag badge-m">method</span><h2><code>Paraformer.average_repeats_inference(ctc_probs, greedy_path)</code></h2><div class="dmod">funasr.models.paraformer_v2_community.Paraformer · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/paraformer_v2_community/model.py#L451" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><h4>Returns:</h4>
+<ul>
+<li><code>merged_probs</code> — [U&#x27;, V]</li>
+<li><code>timestamps</code> — List[Tuple[int, int]] -&gt; [(start_frame, end_frame), ...]</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def average_repeats_inference(self, ctc_probs, greedy_path):
+ &quot;&quot;&quot;
+ Returns:
+ merged_probs: [U&#x27;, V]
+ timestamps: List[Tuple[int, int]] -&gt; [(start_frame, end_frame), ...]
+ &quot;&quot;&quot;
+ if greedy_path.numel() == 0:
+ return torch.zeros((0, ctc_probs.size(1)), device=ctc_probs.device)
+
+ # Find consecutive segments in the greedy path
+ unique_tokens, counts = torch.unique_consecutive(greedy_path, return_counts=True)
+
+ # Compute start and end indices for each segment
+ end_indices = torch.cumsum(counts, dim=0)
+ start_indices = torch.cat([torch.tensor([0], device=counts.device), end_indices[:-1]])
+
+ merged_probs = []
+
+ for i, token in enumerate(unique_tokens):
+ if token != self.blank_id:
+ start = start_indices[i].item()
+ end = end_indices[i].item()
+
+ # Extract and average probabilities for the decoder
+ avg_prob = ctc_probs[start:end].mean(dim=0)
+ merged_probs.append(avg_prob)
+
+
+ if not merged_probs:
+ return torch.zeros((0, ctc_probs.size(1)), device=ctc_probs.device)
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/paraformer_v2_community/model.py#L451" target="_blank" class="src-more">View full source on GitHub →</a></details></div>
+<div class="api-detail" id="e185"><span class="dtag badge-m">method</span><h2><code>Paraformer.inference(data_in, data_lengths, key, tokenizer, frontend, **kwargs)</code></h2><div class="dmod">funasr.models.paraformer_v2_community.Paraformer · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/paraformer_v2_community/model.py#L484" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Run inference on input data.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>data_in</code> — Input data (audio samples, file paths, or text).</li>
+<li><code>data_lengths</code> — Lengths of each input sample in the batch.</li>
+<li><code>key</code> — Sample identifiers.</li>
+<li><code>tokenizer</code> — Tokenizer instance for text encoding/decoding.</li>
+<li><code>frontend</code> — Audio frontend for feature extraction.</li>
+<li><code>**kwargs</code> — Additional keyword arguments.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def inference(
+ self,
+ data_in,
+ data_lengths=None,
+ key: list = None,
+ tokenizer=None,
+ frontend=None,
+ **kwargs,
+ ):
+
+ &quot;&quot;&quot;Run inference on input data.
+
+ Args:
+ data_in: Input data (audio samples, file paths, or text).
+ data_lengths: Lengths of each input sample in the batch.
+ key: Sample identifiers.
+ tokenizer: Tokenizer instance for text encoding/decoding.
+ frontend: Audio frontend for feature extraction.
+ **kwargs: Additional keyword arguments.
+ &quot;&quot;&quot;
+ meta_data = {}
+ if (
+ isinstance(data_in, torch.Tensor) and kwargs.get(&quot;data_type&quot;, &quot;sound&quot;) == &quot;fbank&quot;
+ ): # fbank
+ speech, speech_lengths = data_in, data_lengths
+ if len(speech.shape) &lt; 3:
+ speech = speech[None, :, :]
+ if speech_lengths is not None:
+ speech_lengths = speech_lengths.squeeze(-1)
+ else:
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/paraformer_v2_community/model.py#L484" target="_blank" class="src-more">View full source on GitHub →</a></details></div>
+<div class="api-detail" id="e186"><span class="dtag badge-m">method</span><h2><code>Paraformer.export(**kwargs)</code></h2><div class="dmod">funasr.models.paraformer_v2_community.Paraformer · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/paraformer_v2_community/model.py#L592" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Export.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>**kwargs</code> — Additional keyword arguments.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def export(self, **kwargs):
+ &quot;&quot;&quot;Export.
+
+ Args:
+ **kwargs: Additional keyword arguments.
+ &quot;&quot;&quot;
+ from .export_meta import export_rebuild_model
+
+ if &quot;max_seq_len&quot; not in kwargs:
+ kwargs[&quot;max_seq_len&quot;] = 512
+ models = export_rebuild_model(model=self, **kwargs)
+ return models
+</code></pre></details></div>
+<div class="api-detail" id="e187"><span class="dtag badge-c">class</span><h2>Qwen3ASR</h2><div class="dmod">funasr.models.qwen3_asr · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/qwen3_asr/model.py#L16" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><ul>
+<li><code>Qwen3</code> — ASR: Large Language Model based ASR supporting 52 languages.</li>
+</ul>
+<p>Wraps the qwen-asr package&#x27;s Qwen3ASRModel for use within FunASR&#x27;s AutoModel interface.</p>
+<p>Supports auto language detection, contextual recognition, and optional forced alignment</p>
+<p>for character-level timestamps.</p>
+<h4>Requirements:</h4>
+<p>pip install qwen-asr</p>
+<h4>Models:</h4>
+<ul>
+<li>Qwen/Qwen3-ASR-0.6B (lighter, ~4GB GPU memory)</li>
+<li>Qwen/Qwen3-ASR-1.7B (more accurate, ~8GB GPU memory)</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code>class Qwen3ASR(nn.Module):
+ &quot;&quot;&quot;Qwen3-ASR: Large Language Model based ASR supporting 52 languages.
+
+ Wraps the qwen-asr package&#x27;s Qwen3ASRModel for use within FunASR&#x27;s AutoModel interface.
+ Supports auto language detection, contextual recognition, and optional forced alignment
+ for character-level timestamps.
+
+ Requirements:
+ pip install qwen-asr
+
+ Models:
+ - Qwen/Qwen3-ASR-0.6B (lighter, ~4GB GPU memory)
+ - Qwen/Qwen3-ASR-1.7B (more accurate, ~8GB GPU memory)
+ &quot;&quot;&quot;
+
+ def __init__(self, **kwargs):
+ &quot;&quot;&quot;Initialize Qwen3ASR.
+
+ Args:
+ **kwargs: Additional keyword arguments.
+ &quot;&quot;&quot;
+ super().__init__()
+ model_path = kwargs.get(&quot;model_path&quot;, kwargs.get(&quot;model&quot;, &quot;Qwen/Qwen3-ASR-1.7B&quot;))
+ device = kwargs.get(&quot;device&quot;, &quot;cuda:0&quot;)
+ dtype = kwargs.get(&quot;dtype&quot;, &quot;bf16&quot;)
+ hub = kwargs.get(&quot;hub&quot;, &quot;ms&quot;)
+ max_new_tokens = kwargs.get(&quot;max_new_tokens&quot;, 512)
+ max_inference_batch_size = kwargs.get(&quot;max_inference_batch_size&quot;, 32)
+ forced_aligner = kwargs.get(&quot;forced_aligner&quot;, None)
+ forced_aligner_kwargs = kwargs.get(&quot;forced_aligner_kwargs&quot;, None)
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/qwen3_asr/model.py#L16" target="_blank" class="src-more">View full source on GitHub →</a></details><h3 class="mt">Methods</h3><div class="mblk"><code>.forward(**kwargs)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/qwen3_asr/model.py#L105" target="_blank" class="gh-link">L105</a><div class="mdoc"><p>Forward pass for training.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>**kwargs</code> — Additional keyword arguments.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def forward(self, **kwargs):
+ &quot;&quot;&quot;Forward pass for training.
+
+ Args:
+ **kwargs: Additional keyword arguments.
+ &quot;&quot;&quot;
+ raise NotImplementedError(&quot;Qwen3ASR only supports inference mode&quot;)
+</code></pre></details></div><div class="mblk"><code>.inference(data_in, data_lengths, key, tokenizer, frontend, **kwargs)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/qwen3_asr/model.py#L113" target="_blank" class="gh-link">L113</a><div class="mdoc"><p>Run Qwen3-ASR speech recognition.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>data_in</code> — Audio input. Accepts:</li>
+<li>list of file paths/URLs</li>
+<li>list of (numpy_array, sample_rate) tuples</li>
+<li>single numpy array or torch Tensor</li>
+<li><code>data_lengths</code> — Not used.</li>
+<li><code>key (list)</code> — Sample identifiers.</li>
+<li><code>tokenizer</code> — Not used (Qwen3-ASR has internal tokenizer).</li>
+<li><code>frontend</code> — Not used (Qwen3-ASR has internal audio processing).</li>
+<li><code>**kwargs</code> — Runtime parameters:</li>
+<li>language (str): Language hint (e.g. &quot;Chinese&quot;, &quot;English&quot;) or None for auto-detect.</li>
+<li>return_time_stamps (bool): Return character-level timestamps (requires forced_aligner).</li>
+<li>output_timestamp (bool): Same as return_time_stamps (for pipeline compatibility).</li>
+<li>context (str): Context prompt for contextual recognition.</li>
+</ul>
+<h4>Returns:</h4>
+<ul>
+<li><code>tuple</code> — (results, meta_data) where results is list of dicts:</li>
+<li>&quot;key&quot; (str): Sample ID</li>
+<li>&quot;text&quot; (str): Recognized text (with punctuation)</li>
+<li>&quot;language&quot; (str): Detected language (if available)</li>
+<li>&quot;timestamp&quot; (list): [[start_ms, end_ms], ...] (if timestamps enabled)</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def inference(
+ self,
+ data_in,
+ data_lengths=None,
+ key: list = None,
+ tokenizer=None,
+ frontend=None,
+ **kwargs,
+ ):
+ &quot;&quot;&quot;Run Qwen3-ASR speech recognition.
+
+ Args:
+ data_in: Audio input. Accepts:
+ - list of file paths/URLs
+ - list of (numpy_array, sample_rate) tuples
+ - single numpy array or torch Tensor
+ data_lengths: Not used.
+ key (list): Sample identifiers.
+ tokenizer: Not used (Qwen3-ASR has internal tokenizer).
+ frontend: Not used (Qwen3-ASR has internal audio processing).
+ **kwargs: Runtime parameters:
+ - language (str): Language hint (e.g. &quot;Chinese&quot;, &quot;English&quot;) or None for auto-detect.
+ - return_time_stamps (bool): Return character-level timestamps (requires forced_aligner).
+ - output_timestamp (bool): Same as return_time_stamps (for pipeline compatibility).
+ - context (str): Context prompt for contextual recognition.
+
+ Returns:
+ tuple: (results, meta_data) where results is list of dicts:
+ - &quot;key&quot; (str): Sample ID
+ - &quot;text&quot; (str): Recognized text (with punctuation)
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/qwen3_asr/model.py#L113" target="_blank" class="src-more">View full source →</a></details></div></div>
+<div class="api-detail" id="e188"><span class="dtag badge-m">method</span><h2><code>Qwen3ASR.forward(**kwargs)</code></h2><div class="dmod">funasr.models.qwen3_asr.Qwen3ASR · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/qwen3_asr/model.py#L105" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Forward pass for training.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>**kwargs</code> — Additional keyword arguments.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def forward(self, **kwargs):
+ &quot;&quot;&quot;Forward pass for training.
+
+ Args:
+ **kwargs: Additional keyword arguments.
+ &quot;&quot;&quot;
+ raise NotImplementedError(&quot;Qwen3ASR only supports inference mode&quot;)
+</code></pre></details></div>
+<div class="api-detail" id="e189"><span class="dtag badge-m">method</span><h2><code>Qwen3ASR.inference(data_in, data_lengths, key, tokenizer, frontend, **kwargs)</code></h2><div class="dmod">funasr.models.qwen3_asr.Qwen3ASR · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/qwen3_asr/model.py#L113" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Run Qwen3-ASR speech recognition.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>data_in</code> — Audio input. Accepts:</li>
+<li>list of file paths/URLs</li>
+<li>list of (numpy_array, sample_rate) tuples</li>
+<li>single numpy array or torch Tensor</li>
+<li><code>data_lengths</code> — Not used.</li>
+<li><code>key (list)</code> — Sample identifiers.</li>
+<li><code>tokenizer</code> — Not used (Qwen3-ASR has internal tokenizer).</li>
+<li><code>frontend</code> — Not used (Qwen3-ASR has internal audio processing).</li>
+<li><code>**kwargs</code> — Runtime parameters:</li>
+<li>language (str): Language hint (e.g. &quot;Chinese&quot;, &quot;English&quot;) or None for auto-detect.</li>
+<li>return_time_stamps (bool): Return character-level timestamps (requires forced_aligner).</li>
+<li>output_timestamp (bool): Same as return_time_stamps (for pipeline compatibility).</li>
+<li>context (str): Context prompt for contextual recognition.</li>
+</ul>
+<h4>Returns:</h4>
+<ul>
+<li><code>tuple</code> — (results, meta_data) where results is list of dicts:</li>
+<li>&quot;key&quot; (str): Sample ID</li>
+<li>&quot;text&quot; (str): Recognized text (with punctuation)</li>
+<li>&quot;language&quot; (str): Detected language (if available)</li>
+<li>&quot;timestamp&quot; (list): [[start_ms, end_ms], ...] (if timestamps enabled)</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def inference(
+ self,
+ data_in,
+ data_lengths=None,
+ key: list = None,
+ tokenizer=None,
+ frontend=None,
+ **kwargs,
+ ):
+ &quot;&quot;&quot;Run Qwen3-ASR speech recognition.
+
+ Args:
+ data_in: Audio input. Accepts:
+ - list of file paths/URLs
+ - list of (numpy_array, sample_rate) tuples
+ - single numpy array or torch Tensor
+ data_lengths: Not used.
+ key (list): Sample identifiers.
+ tokenizer: Not used (Qwen3-ASR has internal tokenizer).
+ frontend: Not used (Qwen3-ASR has internal audio processing).
+ **kwargs: Runtime parameters:
+ - language (str): Language hint (e.g. &quot;Chinese&quot;, &quot;English&quot;) or None for auto-detect.
+ - return_time_stamps (bool): Return character-level timestamps (requires forced_aligner).
+ - output_timestamp (bool): Same as return_time_stamps (for pipeline compatibility).
+ - context (str): Context prompt for contextual recognition.
+
+ Returns:
+ tuple: (results, meta_data) where results is list of dicts:
+ - &quot;key&quot; (str): Sample ID
+ - &quot;text&quot; (str): Recognized text (with punctuation)
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/qwen3_asr/model.py#L113" target="_blank" class="src-more">View full source on GitHub →</a></details></div>
+<div class="api-detail" id="e190"><span class="dtag badge-c">class</span><h2>QwenAudioWarp</h2><div class="dmod">funasr.models.qwen_audio · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/qwen_audio/model.py#L22" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><ul>
+<li><code>Qwen</code> — Audio: Advancing Universal Audio Understanding via Unified Large-Scale Audio-Language Models</li>
+</ul>
+<p>https://arxiv.org/abs/2311.07919</p>
+<p>Modified from https://github.com/QwenLM/Qwen-Audio</p>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code>class QwenAudioWarp(nn.Module):
+ &quot;&quot;&quot;
+ Qwen-Audio: Advancing Universal Audio Understanding via Unified Large-Scale Audio-Language Models
+ https://arxiv.org/abs/2311.07919
+ Modified from https://github.com/QwenLM/Qwen-Audio
+ &quot;&quot;&quot;
+
+ def __init__(self, *args, **kwargs):
+ &quot;&quot;&quot;Initialize QwenAudioWarp.
+
+ Args:
+ *args: Variable positional arguments.
+ **kwargs: Additional keyword arguments.
+ &quot;&quot;&quot;
+ super().__init__()
+ from transformers import AutoModelForCausalLM, AutoTokenizer
+ from transformers.generation import GenerationConfig
+
+ model_or_path = kwargs.get(&quot;model_path&quot;, &quot;QwenAudio&quot;)
+ model = AutoModelForCausalLM.from_pretrained(
+ model_or_path, device_map=&quot;cpu&quot;, trust_remote_code=True
+ )
+ tokenizer = AutoTokenizer.from_pretrained(model_or_path, trust_remote_code=True)
+
+ self.model = model
+ self.tokenizer = tokenizer
+
+ def forward(
+ self,
+ ):
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/qwen_audio/model.py#L22" target="_blank" class="src-more">View full source on GitHub →</a></details><h3 class="mt">Methods</h3><div class="mblk"><code>.forward()</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/qwen_audio/model.py#L49" target="_blank" class="gh-link">L49</a><div class="mdoc"><p>Forward pass for training.</p>
+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def forward(
+ self,
+ ):
+ &quot;&quot;&quot;Forward pass for training.&quot;&quot;&quot;
+ pass
+</code></pre></details></div><div class="mblk"><code>.inference(data_in, data_lengths, key, tokenizer, frontend, **kwargs)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/qwen_audio/model.py#L55" target="_blank" class="gh-link">L55</a><div class="mdoc"><p>Run inference on input data.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>data_in</code> — Input data (audio samples, file paths, or text).</li>
+<li><code>data_lengths</code> — Lengths of each input sample in the batch.</li>
+<li><code>key</code> — Sample identifiers.</li>
+<li><code>tokenizer</code> — Tokenizer instance for text encoding/decoding.</li>
+<li><code>frontend</code> — Audio frontend for feature extraction.</li>
+<li><code>**kwargs</code> — Additional keyword arguments.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def inference(
+ self,
+ data_in,
+ data_lengths=None,
+ key: list = None,
+ tokenizer=None,
+ frontend=None,
+ **kwargs,
+ ):
+ &quot;&quot;&quot;Run inference on input data.
+
+ Args:
+ data_in: Input data (audio samples, file paths, or text).
+ data_lengths: Lengths of each input sample in the batch.
+ key: Sample identifiers.
+ tokenizer: Tokenizer instance for text encoding/decoding.
+ frontend: Audio frontend for feature extraction.
+ **kwargs: Additional keyword arguments.
+ &quot;&quot;&quot;
+ if kwargs.get(&quot;batch_size&quot;, 1) &gt; 1:
+ raise NotImplementedError(&quot;batch decoding is not implemented&quot;)
+
+ meta_data = {}
+ # meta_data[&quot;batch_data_time&quot;] = -1
+ prompt = kwargs.get(
+ &quot;prompt&quot;, &quot;&lt;|startoftranscription|&gt;&lt;|en|&gt;&lt;|transcribe|&gt;&lt;|en|&gt;&lt;|notimestamps|&gt;&lt;|wo_itn|&gt;&quot;
+ )
+ query = f&quot;&lt;audio&gt;{data_in[0]}&lt;/audio&gt;{prompt}&quot;
+ audio_info = self.tokenizer.process_audio(query)
+ inputs = self.tokenizer(query, return_tensors=&quot;pt&quot;, audio_info=audio_info)
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/qwen_audio/model.py#L55" target="_blank" class="src-more">View full source →</a></details></div></div>
+<div class="api-detail" id="e191"><span class="dtag badge-m">method</span><h2><code>QwenAudioWarp.forward()</code></h2><div class="dmod">funasr.models.qwen_audio.QwenAudioWarp · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/qwen_audio/model.py#L49" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Forward pass for training.</p>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def forward(
+ self,
+ ):
+ &quot;&quot;&quot;Forward pass for training.&quot;&quot;&quot;
+ pass
+</code></pre></details></div>
+<div class="api-detail" id="e192"><span class="dtag badge-m">method</span><h2><code>QwenAudioWarp.inference(data_in, data_lengths, key, tokenizer, frontend, **kwargs)</code></h2><div class="dmod">funasr.models.qwen_audio.QwenAudioWarp · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/qwen_audio/model.py#L55" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Run inference on input data.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>data_in</code> — Input data (audio samples, file paths, or text).</li>
+<li><code>data_lengths</code> — Lengths of each input sample in the batch.</li>
+<li><code>key</code> — Sample identifiers.</li>
+<li><code>tokenizer</code> — Tokenizer instance for text encoding/decoding.</li>
+<li><code>frontend</code> — Audio frontend for feature extraction.</li>
+<li><code>**kwargs</code> — Additional keyword arguments.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def inference(
+ self,
+ data_in,
+ data_lengths=None,
+ key: list = None,
+ tokenizer=None,
+ frontend=None,
+ **kwargs,
+ ):
+ &quot;&quot;&quot;Run inference on input data.
+
+ Args:
+ data_in: Input data (audio samples, file paths, or text).
+ data_lengths: Lengths of each input sample in the batch.
+ key: Sample identifiers.
+ tokenizer: Tokenizer instance for text encoding/decoding.
+ frontend: Audio frontend for feature extraction.
+ **kwargs: Additional keyword arguments.
+ &quot;&quot;&quot;
+ if kwargs.get(&quot;batch_size&quot;, 1) &gt; 1:
+ raise NotImplementedError(&quot;batch decoding is not implemented&quot;)
+
+ meta_data = {}
+ # meta_data[&quot;batch_data_time&quot;] = -1
+ prompt = kwargs.get(
+ &quot;prompt&quot;, &quot;&lt;|startoftranscription|&gt;&lt;|en|&gt;&lt;|transcribe|&gt;&lt;|en|&gt;&lt;|notimestamps|&gt;&lt;|wo_itn|&gt;&quot;
+ )
+ query = f&quot;&lt;audio&gt;{data_in[0]}&lt;/audio&gt;{prompt}&quot;
+ audio_info = self.tokenizer.process_audio(query)
+ inputs = self.tokenizer(query, return_tensors=&quot;pt&quot;, audio_info=audio_info)
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/qwen_audio/model.py#L55" target="_blank" class="src-more">View full source on GitHub →</a></details></div>
+<div class="api-detail" id="e193"><span class="dtag badge-c">class</span><h2>QwenAudioChatWarp</h2><div class="dmod">funasr.models.qwen_audio · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/qwen_audio/model.py#L104" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><ul>
+<li><code>QwenAudioChat</code> — Qwen Audio Chat model wrapper.</li>
+</ul>
+<p>Interactive audio chat using the Qwen-Audio-Chat model.</p>
+<p>Supports multi-turn conversation about audio content.</p>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code>class QwenAudioChatWarp(nn.Module):
+ &quot;&quot;&quot;QwenAudioChat: Qwen Audio Chat model wrapper.
+
+ Interactive audio chat using the Qwen-Audio-Chat model.
+ Supports multi-turn conversation about audio content.
+ &quot;&quot;&quot;
+
+ def __init__(self, *args, **kwargs):
+ &quot;&quot;&quot;
+ Qwen-Audio: Advancing Universal Audio Understanding via Unified Large-Scale Audio-Language Models
+ https://arxiv.org/abs/2311.07919
+ Modified from https://github.com/QwenLM/Qwen-Audio
+ &quot;&quot;&quot;
+ super().__init__()
+ from transformers import AutoModelForCausalLM, AutoTokenizer
+ from transformers.generation import GenerationConfig
+
+ model_or_path = kwargs.get(&quot;model_path&quot;, &quot;QwenAudio&quot;)
+ bf16 = kwargs.get(&quot;bf16&quot;, False)
+ fp16 = kwargs.get(&quot;fp16&quot;, False)
+ model = AutoModelForCausalLM.from_pretrained(
+ model_or_path, device_map=&quot;cpu&quot;, bf16=bf16, fp16=fp16, trust_remote_code=True
+ )
+ tokenizer = AutoTokenizer.from_pretrained(model_or_path, trust_remote_code=True)
+
+ self.model = model
+ self.tokenizer = tokenizer
+
+ def forward(
+ self,
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/qwen_audio/model.py#L104" target="_blank" class="src-more">View full source on GitHub →</a></details><h3 class="mt">Methods</h3><div class="mblk"><code>.forward()</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/qwen_audio/model.py#L132" target="_blank" class="gh-link">L132</a><div class="mdoc"><p>Forward pass for training.</p>
+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def forward(
+ self,
+ ):
+ &quot;&quot;&quot;Forward pass for training.&quot;&quot;&quot;
+ pass
+</code></pre></details></div><div class="mblk"><code>.inference(data_in, data_lengths, key, tokenizer, frontend, **kwargs)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/qwen_audio/model.py#L138" target="_blank" class="gh-link">L138</a><div class="mdoc"><p>Run inference on input data.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>data_in</code> — Input data (audio samples, file paths, or text).</li>
+<li><code>data_lengths</code> — Lengths of each input sample in the batch.</li>
+<li><code>key</code> — Sample identifiers.</li>
+<li><code>tokenizer</code> — Tokenizer instance for text encoding/decoding.</li>
+<li><code>frontend</code> — Audio frontend for feature extraction.</li>
+<li><code>**kwargs</code> — Additional keyword arguments.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def inference(
+ self,
+ data_in,
+ data_lengths=None,
+ key: list = None,
+ tokenizer=None,
+ frontend=None,
+ **kwargs,
+ ):
+ &quot;&quot;&quot;Run inference on input data.
+
+ Args:
+ data_in: Input data (audio samples, file paths, or text).
+ data_lengths: Lengths of each input sample in the batch.
+ key: Sample identifiers.
+ tokenizer: Tokenizer instance for text encoding/decoding.
+ frontend: Audio frontend for feature extraction.
+ **kwargs: Additional keyword arguments.
+ &quot;&quot;&quot;
+ if kwargs.get(&quot;batch_size&quot;, 1) &gt; 1:
+ raise NotImplementedError(&quot;batch decoding is not implemented&quot;)
+
+ meta_data = {}
+
+ prompt = kwargs.get(&quot;prompt&quot;, &quot;what does the person say?&quot;)
+ cache = kwargs.get(&quot;cache&quot;, {})
+ history = cache.get(&quot;history&quot;, None)
+ if data_in[0] is not None:
+ # 1st dialogue turn
+ query = self.tokenizer.from_list_format(
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/qwen_audio/model.py#L138" target="_blank" class="src-more">View full source →</a></details></div></div>
+<div class="api-detail" id="e194"><span class="dtag badge-m">method</span><h2><code>QwenAudioChatWarp.forward()</code></h2><div class="dmod">funasr.models.qwen_audio.QwenAudioChatWarp · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/qwen_audio/model.py#L132" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Forward pass for training.</p>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def forward(
+ self,
+ ):
+ &quot;&quot;&quot;Forward pass for training.&quot;&quot;&quot;
+ pass
+</code></pre></details></div>
+<div class="api-detail" id="e195"><span class="dtag badge-m">method</span><h2><code>QwenAudioChatWarp.inference(data_in, data_lengths, key, tokenizer, frontend, **kwargs)</code></h2><div class="dmod">funasr.models.qwen_audio.QwenAudioChatWarp · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/qwen_audio/model.py#L138" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Run inference on input data.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>data_in</code> — Input data (audio samples, file paths, or text).</li>
+<li><code>data_lengths</code> — Lengths of each input sample in the batch.</li>
+<li><code>key</code> — Sample identifiers.</li>
+<li><code>tokenizer</code> — Tokenizer instance for text encoding/decoding.</li>
+<li><code>frontend</code> — Audio frontend for feature extraction.</li>
+<li><code>**kwargs</code> — Additional keyword arguments.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def inference(
+ self,
+ data_in,
+ data_lengths=None,
+ key: list = None,
+ tokenizer=None,
+ frontend=None,
+ **kwargs,
+ ):
+ &quot;&quot;&quot;Run inference on input data.
+
+ Args:
+ data_in: Input data (audio samples, file paths, or text).
+ data_lengths: Lengths of each input sample in the batch.
+ key: Sample identifiers.
+ tokenizer: Tokenizer instance for text encoding/decoding.
+ frontend: Audio frontend for feature extraction.
+ **kwargs: Additional keyword arguments.
+ &quot;&quot;&quot;
+ if kwargs.get(&quot;batch_size&quot;, 1) &gt; 1:
+ raise NotImplementedError(&quot;batch decoding is not implemented&quot;)
+
+ meta_data = {}
+
+ prompt = kwargs.get(&quot;prompt&quot;, &quot;what does the person say?&quot;)
+ cache = kwargs.get(&quot;cache&quot;, {})
+ history = cache.get(&quot;history&quot;, None)
+ if data_in[0] is not None:
+ # 1st dialogue turn
+ query = self.tokenizer.from_list_format(
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/qwen_audio/model.py#L138" target="_blank" class="src-more">View full source on GitHub →</a></details></div>
+<div class="api-detail" id="e196"><span class="dtag badge-c">class</span><h2>SANM</h2><div class="dmod">funasr.models.sanm · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/sanm/model.py#L15" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><ul>
+<li><code>Author</code> — Zhifu Gao, Shiliang Zhang, Ming Lei, Ian McLoughlin</li>
+<li><code>San</code> — m: Memory equipped self-attention for end-to-end speech recognition</li>
+</ul>
+<p>https://arxiv.org/abs/2006.01713</p>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code>class SANM(Transformer):
+ &quot;&quot;&quot;
+ Author: Zhifu Gao, Shiliang Zhang, Ming Lei, Ian McLoughlin
+ San-m: Memory equipped self-attention for end-to-end speech recognition
+ https://arxiv.org/abs/2006.01713
+ &quot;&quot;&quot;
+
+ def __init__(
+ self,
+ *args,
+ **kwargs,
+ ):
+
+ &quot;&quot;&quot;Initialize SANM.
+
+ Args:
+ *args: Variable positional arguments.
+ **kwargs: Additional keyword arguments.
+ &quot;&quot;&quot;
+ super().__init__(*args, **kwargs)
+</code></pre></details></div>
+<div class="api-detail" id="e197"><span class="dtag badge-c">class</span><h2>SanmKWS</h2><div class="dmod">funasr.models.sanm_kws · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/sanm_kws/model.py#L28" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><ul>
+<li><code>SANM</code> — KWS: Self-Attention Neural Memory based Keyword Spotting.</li>
+</ul>
+<p>Advanced keyword spotting using self-attention mechanism</p>
+<p>for better context modeling of keyword patterns.</p>
+<ul>
+<li><code>Output</code> — {&quot;key&quot;: str, &quot;value&quot;: detected_keyword_info}</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code>class SanmKWS(torch.nn.Module):
+ &quot;&quot;&quot;SANM-KWS: Self-Attention Neural Memory based Keyword Spotting.
+
+ Advanced keyword spotting using self-attention mechanism
+ for better context modeling of keyword patterns.
+
+ Output: {&quot;key&quot;: str, &quot;value&quot;: detected_keyword_info}
+ &quot;&quot;&quot;
+
+ def __init__(
+ self,
+ specaug: Optional[str] = None,
+ specaug_conf: Optional[Dict] = None,
+ normalize: str = None,
+ normalize_conf: Optional[Dict] = None,
+ encoder: str = None,
+ encoder_conf: Optional[Dict] = None,
+ ctc: str = None,
+ ctc_conf: Optional[Dict] = None,
+ ctc_weight: float = 1.0,
+ input_size: int = 360,
+ vocab_size: int = -1,
+ ignore_id: int = -1,
+ blank_id: int = 0,
+ sos: int = 1,
+ eos: int = 2,
+ **kwargs,
+ ):
+
+ &quot;&quot;&quot;Initialize SanmKWS.
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/sanm_kws/model.py#L28" target="_blank" class="src-more">View full source on GitHub →</a></details><h3 class="mt">Methods</h3><div class="mblk"><code>.forward(speech, speech_lengths, text, text_lengths, **kwargs)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/sanm_kws/model.py#L112" target="_blank" class="gh-link">L112</a><div class="mdoc"><p>Encoder + Decoder + Calc loss</p>
+<h4>Args:</h4>
+<ul>
+<li><code>speech</code> — (Batch, Length, ...)</li>
+<li><code>speech_lengths</code> — (Batch, )</li>
+<li><code>text</code> — (Batch, Length)</li>
+<li><code>text_lengths</code> — (Batch,)</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def forward(
+ self,
+ speech: torch.Tensor,
+ speech_lengths: torch.Tensor,
+ text: torch.Tensor,
+ text_lengths: torch.Tensor,
+ **kwargs,
+ ) -&gt; Tuple[torch.Tensor, Dict[str, torch.Tensor], torch.Tensor]:
+ &quot;&quot;&quot;Encoder + Decoder + Calc loss
+ Args:
+ speech: (Batch, Length, ...)
+ speech_lengths: (Batch, )
+ text: (Batch, Length)
+ text_lengths: (Batch,)
+ &quot;&quot;&quot;
+ if len(text_lengths.size()) &gt; 1:
+ text_lengths = text_lengths[:, 0]
+ if len(speech_lengths.size()) &gt; 1:
+ speech_lengths = speech_lengths[:, 0]
+ batch_size = speech.shape[0]
+
+ # Encoder
+ encoder_out, encoder_out_lens = self.encode(speech, speech_lengths)
+
+ # decoder: CTC branch
+ loss_ctc, cer_ctc = self._calc_ctc_loss(
+ encoder_out, encoder_out_lens, text, text_lengths
+ )
+
+ # Collect CTC branch stats
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/sanm_kws/model.py#L112" target="_blank" class="src-more">View full source →</a></details></div><div class="mblk"><code>.encode(speech, speech_lengths, **kwargs)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/sanm_kws/model.py#L156" target="_blank" class="gh-link">L156</a><div class="mdoc"><p>Encoder. Note that this method is used by asr_inference.py</p>
+<h4>Args:</h4>
+<ul>
+<li><code>speech</code> — (Batch, Length, ...)</li>
+<li><code>speech_lengths</code> — (Batch, )</li>
+<li><code>ind</code> — int</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def encode(
+ self,
+ speech: torch.Tensor,
+ speech_lengths: torch.Tensor,
+ **kwargs,
+ ) -&gt; Tuple[torch.Tensor, torch.Tensor]:
+ &quot;&quot;&quot;Encoder. Note that this method is used by asr_inference.py
+ Args:
+ speech: (Batch, Length, ...)
+ speech_lengths: (Batch, )
+ ind: int
+ &quot;&quot;&quot;
+ with autocast(False):
+ # Data augmentation
+ if self.specaug is not None and self.training:
+ speech, speech_lengths = self.specaug(speech, speech_lengths)
+
+ # Normalization for feature: e.g. Global-CMVN, Utterance-CMVN
+ if self.normalize is not None:
+ speech, speech_lengths = self.normalize(speech, speech_lengths)
+
+ # Forward encoder
+ encoder_out, encoder_out_lens, _ = self.encoder(speech, speech_lengths)
+ if isinstance(encoder_out, tuple):
+ encoder_out = encoder_out[0]
+
+ return encoder_out, encoder_out_lens
+</code></pre></details></div><div class="mblk"><code>.inference(data_in, data_lengths, key, tokenizer, frontend, **kwargs)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/sanm_kws/model.py#L209" target="_blank" class="gh-link">L209</a><div class="mdoc"><p>Run inference on input data.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>data_in</code> — Input data (audio samples, file paths, or text).</li>
+<li><code>data_lengths</code> — Lengths of each input sample in the batch.</li>
+<li><code>key</code> — Sample identifiers.</li>
+<li><code>tokenizer</code> — Tokenizer instance for text encoding/decoding.</li>
+<li><code>frontend</code> — Audio frontend for feature extraction.</li>
+<li><code>**kwargs</code> — Additional keyword arguments.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def inference(
+ self,
+ data_in,
+ data_lengths=None,
+ key: list = None,
+ tokenizer=None,
+ frontend=None,
+ **kwargs,
+ ):
+ &quot;&quot;&quot;Run inference on input data.
+
+ Args:
+ data_in: Input data (audio samples, file paths, or text).
+ data_lengths: Lengths of each input sample in the batch.
+ key: Sample identifiers.
+ tokenizer: Tokenizer instance for text encoding/decoding.
+ frontend: Audio frontend for feature extraction.
+ **kwargs: Additional keyword arguments.
+ &quot;&quot;&quot;
+ keywords = kwargs.get(&quot;keywords&quot;)
+ from funasr.utils.kws_utils import KwsCtcPrefixDecoder
+ self.kws_decoder = KwsCtcPrefixDecoder(
+ ctc=self.ctc,
+ keywords=keywords,
+ token_list=tokenizer.token_list,
+ seg_dict=tokenizer.seg_dict,
+ )
+
+ meta_data = {}
+ if (
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/sanm_kws/model.py#L209" target="_blank" class="src-more">View full source →</a></details></div><div class="mblk"><code>.export(**kwargs)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/sanm_kws/model.py#L300" target="_blank" class="gh-link">L300</a><div class="mdoc"><p>Export.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>**kwargs</code> — Additional keyword arguments.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def export(self, **kwargs):
+ &quot;&quot;&quot;Export.
+
+ Args:
+ **kwargs: Additional keyword arguments.
+ &quot;&quot;&quot;
+ from .export_meta import export_rebuild_model
+
+ if &quot;max_seq_len&quot; not in kwargs:
+ kwargs[&quot;max_seq_len&quot;] = 512
+ models = export_rebuild_model(model=self, **kwargs)
+ return models
+</code></pre></details></div></div>
+<div class="api-detail" id="e198"><span class="dtag badge-m">method</span><h2><code>SanmKWS.forward(speech, speech_lengths, text, text_lengths, **kwargs)</code></h2><div class="dmod">funasr.models.sanm_kws.SanmKWS · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/sanm_kws/model.py#L112" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Encoder + Decoder + Calc loss</p>
+<h4>Args:</h4>
+<ul>
+<li><code>speech</code> — (Batch, Length, ...)</li>
+<li><code>speech_lengths</code> — (Batch, )</li>
+<li><code>text</code> — (Batch, Length)</li>
+<li><code>text_lengths</code> — (Batch,)</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def forward(
+ self,
+ speech: torch.Tensor,
+ speech_lengths: torch.Tensor,
+ text: torch.Tensor,
+ text_lengths: torch.Tensor,
+ **kwargs,
+ ) -&gt; Tuple[torch.Tensor, Dict[str, torch.Tensor], torch.Tensor]:
+ &quot;&quot;&quot;Encoder + Decoder + Calc loss
+ Args:
+ speech: (Batch, Length, ...)
+ speech_lengths: (Batch, )
+ text: (Batch, Length)
+ text_lengths: (Batch,)
+ &quot;&quot;&quot;
+ if len(text_lengths.size()) &gt; 1:
+ text_lengths = text_lengths[:, 0]
+ if len(speech_lengths.size()) &gt; 1:
+ speech_lengths = speech_lengths[:, 0]
+ batch_size = speech.shape[0]
+
+ # Encoder
+ encoder_out, encoder_out_lens = self.encode(speech, speech_lengths)
+
+ # decoder: CTC branch
+ loss_ctc, cer_ctc = self._calc_ctc_loss(
+ encoder_out, encoder_out_lens, text, text_lengths
+ )
+
+ # Collect CTC branch stats
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/sanm_kws/model.py#L112" target="_blank" class="src-more">View full source on GitHub →</a></details></div>
+<div class="api-detail" id="e199"><span class="dtag badge-m">method</span><h2><code>SanmKWS.encode(speech, speech_lengths, **kwargs)</code></h2><div class="dmod">funasr.models.sanm_kws.SanmKWS · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/sanm_kws/model.py#L156" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Encoder. Note that this method is used by asr_inference.py</p>
+<h4>Args:</h4>
+<ul>
+<li><code>speech</code> — (Batch, Length, ...)</li>
+<li><code>speech_lengths</code> — (Batch, )</li>
+<li><code>ind</code> — int</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def encode(
+ self,
+ speech: torch.Tensor,
+ speech_lengths: torch.Tensor,
+ **kwargs,
+ ) -&gt; Tuple[torch.Tensor, torch.Tensor]:
+ &quot;&quot;&quot;Encoder. Note that this method is used by asr_inference.py
+ Args:
+ speech: (Batch, Length, ...)
+ speech_lengths: (Batch, )
+ ind: int
+ &quot;&quot;&quot;
+ with autocast(False):
+ # Data augmentation
+ if self.specaug is not None and self.training:
+ speech, speech_lengths = self.specaug(speech, speech_lengths)
+
+ # Normalization for feature: e.g. Global-CMVN, Utterance-CMVN
+ if self.normalize is not None:
+ speech, speech_lengths = self.normalize(speech, speech_lengths)
+
+ # Forward encoder
+ encoder_out, encoder_out_lens, _ = self.encoder(speech, speech_lengths)
+ if isinstance(encoder_out, tuple):
+ encoder_out = encoder_out[0]
+
+ return encoder_out, encoder_out_lens
+</code></pre></details></div>
+<div class="api-detail" id="e200"><span class="dtag badge-m">method</span><h2><code>SanmKWS.inference(data_in, data_lengths, key, tokenizer, frontend, **kwargs)</code></h2><div class="dmod">funasr.models.sanm_kws.SanmKWS · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/sanm_kws/model.py#L209" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Run inference on input data.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>data_in</code> — Input data (audio samples, file paths, or text).</li>
+<li><code>data_lengths</code> — Lengths of each input sample in the batch.</li>
+<li><code>key</code> — Sample identifiers.</li>
+<li><code>tokenizer</code> — Tokenizer instance for text encoding/decoding.</li>
+<li><code>frontend</code> — Audio frontend for feature extraction.</li>
+<li><code>**kwargs</code> — Additional keyword arguments.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def inference(
+ self,
+ data_in,
+ data_lengths=None,
+ key: list = None,
+ tokenizer=None,
+ frontend=None,
+ **kwargs,
+ ):
+ &quot;&quot;&quot;Run inference on input data.
+
+ Args:
+ data_in: Input data (audio samples, file paths, or text).
+ data_lengths: Lengths of each input sample in the batch.
+ key: Sample identifiers.
+ tokenizer: Tokenizer instance for text encoding/decoding.
+ frontend: Audio frontend for feature extraction.
+ **kwargs: Additional keyword arguments.
+ &quot;&quot;&quot;
+ keywords = kwargs.get(&quot;keywords&quot;)
+ from funasr.utils.kws_utils import KwsCtcPrefixDecoder
+ self.kws_decoder = KwsCtcPrefixDecoder(
+ ctc=self.ctc,
+ keywords=keywords,
+ token_list=tokenizer.token_list,
+ seg_dict=tokenizer.seg_dict,
+ )
+
+ meta_data = {}
+ if (
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/sanm_kws/model.py#L209" target="_blank" class="src-more">View full source on GitHub →</a></details></div>
+<div class="api-detail" id="e201"><span class="dtag badge-m">method</span><h2><code>SanmKWS.export(**kwargs)</code></h2><div class="dmod">funasr.models.sanm_kws.SanmKWS · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/sanm_kws/model.py#L300" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Export.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>**kwargs</code> — Additional keyword arguments.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def export(self, **kwargs):
+ &quot;&quot;&quot;Export.
+
+ Args:
+ **kwargs: Additional keyword arguments.
+ &quot;&quot;&quot;
+ from .export_meta import export_rebuild_model
+
+ if &quot;max_seq_len&quot; not in kwargs:
+ kwargs[&quot;max_seq_len&quot;] = 512
+ models = export_rebuild_model(model=self, **kwargs)
+ return models
+</code></pre></details></div>
+<div class="api-detail" id="e202"><span class="dtag badge-c">class</span><h2>SanmKWSStreaming</h2><div class="dmod">funasr.models.sanm_kws_streaming · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/sanm_kws_streaming/model.py#L43" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><ul>
+<li><code>Author</code> — Speech Lab of DAMO Academy, Alibaba Group</li>
+<li><code>Paraformer</code> — Fast and Accurate Parallel Transformer for Non-autoregressive End-to-End Speech Recognition</li>
+</ul>
+<p>https://arxiv.org/abs/2206.08317</p>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code>class SanmKWSStreaming(SanmKWS):
+ &quot;&quot;&quot;
+ Author: Speech Lab of DAMO Academy, Alibaba Group
+ Paraformer: Fast and Accurate Parallel Transformer for Non-autoregressive End-to-End Speech Recognition
+ https://arxiv.org/abs/2206.08317
+ &quot;&quot;&quot;
+
+ def __init__(
+ self,
+ *args,
+ **kwargs,
+ ):
+ &quot;&quot;&quot;Initialize SanmKWSStreaming.
+
+ Args:
+ *args: Variable positional arguments.
+ **kwargs: Additional keyword arguments.
+ &quot;&quot;&quot;
+ super().__init__(*args, **kwargs)
+
+ def forward(
+ self,
+ speech: torch.Tensor,
+ speech_lengths: torch.Tensor,
+ text: torch.Tensor,
+ text_lengths: torch.Tensor,
+ **kwargs,
+ ) -&gt; Tuple[torch.Tensor, Dict[str, torch.Tensor], torch.Tensor]:
+ &quot;&quot;&quot;Encoder + Decoder + Calc loss
+ Args:
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/sanm_kws_streaming/model.py#L43" target="_blank" class="src-more">View full source on GitHub →</a></details><h3 class="mt">Methods</h3><div class="mblk"><code>.forward(speech, speech_lengths, text, text_lengths, **kwargs)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/sanm_kws_streaming/model.py#L63" target="_blank" class="gh-link">L63</a><div class="mdoc"><p>Encoder + Decoder + Calc loss</p>
+<h4>Args:</h4>
+<ul>
+<li><code>speech</code> — (Batch, Length, ...)</li>
+<li><code>speech_lengths</code> — (Batch, )</li>
+<li><code>text</code> — (Batch, Length)</li>
+<li><code>text_lengths</code> — (Batch,)</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def forward(
+ self,
+ speech: torch.Tensor,
+ speech_lengths: torch.Tensor,
+ text: torch.Tensor,
+ text_lengths: torch.Tensor,
+ **kwargs,
+ ) -&gt; Tuple[torch.Tensor, Dict[str, torch.Tensor], torch.Tensor]:
+ &quot;&quot;&quot;Encoder + Decoder + Calc loss
+ Args:
+ speech: (Batch, Length, ...)
+ speech_lengths: (Batch, )
+ text: (Batch, Length)
+ text_lengths: (Batch,)
+ &quot;&quot;&quot;
+ decoding_ind = kwargs.get(&quot;decoding_ind&quot;)
+ if len(text_lengths.size()) &gt; 1:
+ text_lengths = text_lengths[:, 0]
+ if len(speech_lengths.size()) &gt; 1:
+ speech_lengths = speech_lengths[:, 0]
+
+ batch_size = speech.shape[0]
+
+ # Encoder
+ if hasattr(self.encoder, &quot;overlap_chunk_cls&quot;):
+ ind = self.encoder.overlap_chunk_cls.random_choice(self.training, decoding_ind)
+ encoder_out, encoder_out_lens = self.encode(speech, speech_lengths, ind=ind)
+ else:
+ encoder_out, encoder_out_lens = self.encode(speech, speech_lengths)
+
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/sanm_kws_streaming/model.py#L63" target="_blank" class="src-more">View full source →</a></details></div><div class="mblk"><code>.encode_chunk(speech, speech_lengths, cache, **kwargs)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/sanm_kws_streaming/model.py#L119" target="_blank" class="gh-link">L119</a><div class="mdoc"><p>Encode chunk.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>speech</code> — Speech audio tensor, shape (batch, time).</li>
+<li><code>speech_lengths</code> — Length of each speech sample.</li>
+<li><code>cache</code> — State cache dict for streaming inference.</li>
+<li><code>**kwargs</code> — Additional keyword arguments.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def encode_chunk(
+ self,
+ speech: torch.Tensor,
+ speech_lengths: torch.Tensor,
+ cache: dict = None,
+ **kwargs,
+ ) -&gt; Tuple[torch.Tensor, torch.Tensor]:
+ &quot;&quot;&quot;Encode chunk.
+
+ Args:
+ speech: Speech audio tensor, shape (batch, time).
+ speech_lengths: Length of each speech sample.
+ cache: State cache dict for streaming inference.
+ **kwargs: Additional keyword arguments.
+ &quot;&quot;&quot;
+ if cache is None:
+ cache = {}
+ &quot;&quot;&quot;Frontend + Encoder. Note that this method is used by asr_inference.py
+ Args:
+ speech: (Batch, Length, ...)
+ speech_lengths: (Batch, )
+ ind: int
+ &quot;&quot;&quot;
+ with autocast(False):
+ # Data augmentation
+ if self.specaug is not None and self.training:
+ speech, speech_lengths = self.specaug(speech, speech_lengths)
+
+ # Normalization for feature: e.g. Global-CMVN, Utterance-CMVN
+ if self.normalize is not None:
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/sanm_kws_streaming/model.py#L119" target="_blank" class="src-more">View full source →</a></details></div><div class="mblk"><code>.init_cache(cache, **kwargs)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/sanm_kws_streaming/model.py#L161" target="_blank" class="gh-link">L161</a><div class="mdoc"><p>Init cache.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>cache</code> — State cache dict for streaming inference.</li>
+<li><code>**kwargs</code> — Additional keyword arguments.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def init_cache(self, cache: dict = None, **kwargs):
+ &quot;&quot;&quot;Init cache.
+
+ Args:
+ cache: State cache dict for streaming inference.
+ **kwargs: Additional keyword arguments.
+ &quot;&quot;&quot;
+ if cache is None:
+ cache = {}
+ chunk_size = kwargs.get(&quot;chunk_size&quot;, [0, 10, 5])
+ encoder_chunk_look_back = kwargs.get(&quot;encoder_chunk_look_back&quot;, 0)
+ decoder_chunk_look_back = kwargs.get(&quot;decoder_chunk_look_back&quot;, 0)
+ batch_size = 1
+
+ enc_output_size = kwargs[&quot;encoder_conf&quot;][&quot;output_size&quot;]
+ feats_dims = kwargs[&quot;frontend_conf&quot;][&quot;n_mels&quot;] * kwargs[&quot;frontend_conf&quot;][&quot;lfr_m&quot;]
+ cache_encoder = {
+ &quot;start_idx&quot;: 0,
+ &quot;cif_hidden&quot;: torch.zeros((batch_size, 1, enc_output_size)),
+ &quot;cif_alphas&quot;: torch.zeros((batch_size, 1)),
+ &quot;encoder_out&quot;: None,
+ &quot;encoder_out_lens&quot;: None,
+ &quot;chunk_size&quot;: chunk_size,
+ &quot;encoder_chunk_look_back&quot;: encoder_chunk_look_back,
+ &quot;last_chunk&quot;: False,
+ &quot;opt&quot;: None,
+ &quot;feats&quot;: torch.zeros((batch_size, chunk_size[0] + chunk_size[2], feats_dims)),
+ &quot;tail_chunk&quot;: False,
+ }
+ cache[&quot;encoder&quot;] = cache_encoder
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/sanm_kws_streaming/model.py#L161" target="_blank" class="src-more">View full source →</a></details></div><div class="mblk"><code>.generate_chunk(speech, speech_lengths, key, tokenizer, frontend, **kwargs)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/sanm_kws_streaming/model.py#L204" target="_blank" class="gh-link">L204</a><div class="mdoc"><p>Generate chunk.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>speech</code> — Speech audio tensor, shape (batch, time).</li>
+<li><code>speech_lengths</code> — Length of each speech sample.</li>
+<li><code>key</code> — Sample identifiers.</li>
+<li><code>tokenizer</code> — Tokenizer instance for text encoding/decoding.</li>
+<li><code>frontend</code> — Audio frontend for feature extraction.</li>
+<li><code>**kwargs</code> — Additional keyword arguments.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def generate_chunk(
+ self,
+ speech,
+ speech_lengths=None,
+ key: list = None,
+ tokenizer=None,
+ frontend=None,
+ **kwargs,
+ ):
+ &quot;&quot;&quot;Generate chunk.
+
+ Args:
+ speech: Speech audio tensor, shape (batch, time).
+ speech_lengths: Length of each speech sample.
+ key: Sample identifiers.
+ tokenizer: Tokenizer instance for text encoding/decoding.
+ frontend: Audio frontend for feature extraction.
+ **kwargs: Additional keyword arguments.
+ &quot;&quot;&quot;
+ cache = kwargs.get(&quot;cache&quot;, {})
+ speech = speech.to(device=kwargs[&quot;device&quot;])
+ speech_lengths = speech_lengths.to(device=kwargs[&quot;device&quot;])
+
+ # Encoder
+ is_final = kwargs.get(&quot;is_final&quot;, False)
+ encoder_out, encoder_out_lens = self.encode_chunk(
+ speech, speech_lengths, cache=cache, is_final=is_final
+ )
+ if isinstance(encoder_out, tuple):
+ encoder_out = encoder_out[0]
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/sanm_kws_streaming/model.py#L204" target="_blank" class="src-more">View full source →</a></details></div><div class="mblk"><code>.inference(data_in, data_lengths, key, tokenizer, frontend, cache, **kwargs)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/sanm_kws_streaming/model.py#L287" target="_blank" class="gh-link">L287</a><div class="mdoc"><p>Run inference on input data.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>data_in</code> — Input data (audio samples, file paths, or text).</li>
+<li><code>data_lengths</code> — Lengths of each input sample in the batch.</li>
+<li><code>key</code> — Sample identifiers.</li>
+<li><code>tokenizer</code> — Tokenizer instance for text encoding/decoding.</li>
+<li><code>frontend</code> — Audio frontend for feature extraction.</li>
+<li><code>cache</code> — State cache dict for streaming inference.</li>
+<li><code>**kwargs</code> — Additional keyword arguments.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def inference(
+ self,
+ data_in,
+ data_lengths=None,
+ key: list = None,
+ tokenizer=None,
+ frontend=None,
+ cache: dict = None,
+ **kwargs,
+ ):
+ &quot;&quot;&quot;Run inference on input data.
+
+ Args:
+ data_in: Input data (audio samples, file paths, or text).
+ data_lengths: Lengths of each input sample in the batch.
+ key: Sample identifiers.
+ tokenizer: Tokenizer instance for text encoding/decoding.
+ frontend: Audio frontend for feature extraction.
+ cache: State cache dict for streaming inference.
+ **kwargs: Additional keyword arguments.
+ &quot;&quot;&quot;
+ if cache is None:
+ cache = {}
+ keywords = kwargs.get(&quot;keywords&quot;)
+ from funasr.utils.kws_utils import KwsCtcPrefixDecoder
+ self.kws_decoder = KwsCtcPrefixDecoder(
+ ctc=self.ctc,
+ keywords=keywords,
+ token_list=tokenizer.token_list,
+ seg_dict=tokenizer.seg_dict,
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/sanm_kws_streaming/model.py#L287" target="_blank" class="src-more">View full source →</a></details></div><div class="mblk"><code>.export(**kwargs)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/sanm_kws_streaming/model.py#L490" target="_blank" class="gh-link">L490</a><div class="mdoc"><p>Export.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>**kwargs</code> — Additional keyword arguments.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def export(self, **kwargs):
+ &quot;&quot;&quot;Export.
+
+ Args:
+ **kwargs: Additional keyword arguments.
+ &quot;&quot;&quot;
+ from .export_meta import export_rebuild_model
+
+ models = export_rebuild_model(model=self, **kwargs)
+ return models
+</code></pre></details></div></div>
+<div class="api-detail" id="e203"><span class="dtag badge-m">method</span><h2><code>SanmKWSStreaming.forward(speech, speech_lengths, text, text_lengths, **kwargs)</code></h2><div class="dmod">funasr.models.sanm_kws_streaming.SanmKWSStreaming · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/sanm_kws_streaming/model.py#L63" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Encoder + Decoder + Calc loss</p>
+<h4>Args:</h4>
+<ul>
+<li><code>speech</code> — (Batch, Length, ...)</li>
+<li><code>speech_lengths</code> — (Batch, )</li>
+<li><code>text</code> — (Batch, Length)</li>
+<li><code>text_lengths</code> — (Batch,)</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def forward(
+ self,
+ speech: torch.Tensor,
+ speech_lengths: torch.Tensor,
+ text: torch.Tensor,
+ text_lengths: torch.Tensor,
+ **kwargs,
+ ) -&gt; Tuple[torch.Tensor, Dict[str, torch.Tensor], torch.Tensor]:
+ &quot;&quot;&quot;Encoder + Decoder + Calc loss
+ Args:
+ speech: (Batch, Length, ...)
+ speech_lengths: (Batch, )
+ text: (Batch, Length)
+ text_lengths: (Batch,)
+ &quot;&quot;&quot;
+ decoding_ind = kwargs.get(&quot;decoding_ind&quot;)
+ if len(text_lengths.size()) &gt; 1:
+ text_lengths = text_lengths[:, 0]
+ if len(speech_lengths.size()) &gt; 1:
+ speech_lengths = speech_lengths[:, 0]
+
+ batch_size = speech.shape[0]
+
+ # Encoder
+ if hasattr(self.encoder, &quot;overlap_chunk_cls&quot;):
+ ind = self.encoder.overlap_chunk_cls.random_choice(self.training, decoding_ind)
+ encoder_out, encoder_out_lens = self.encode(speech, speech_lengths, ind=ind)
+ else:
+ encoder_out, encoder_out_lens = self.encode(speech, speech_lengths)
+
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/sanm_kws_streaming/model.py#L63" target="_blank" class="src-more">View full source on GitHub →</a></details></div>
+<div class="api-detail" id="e204"><span class="dtag badge-m">method</span><h2><code>SanmKWSStreaming.encode_chunk(speech, speech_lengths, cache, **kwargs)</code></h2><div class="dmod">funasr.models.sanm_kws_streaming.SanmKWSStreaming · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/sanm_kws_streaming/model.py#L119" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Encode chunk.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>speech</code> — Speech audio tensor, shape (batch, time).</li>
+<li><code>speech_lengths</code> — Length of each speech sample.</li>
+<li><code>cache</code> — State cache dict for streaming inference.</li>
+<li><code>**kwargs</code> — Additional keyword arguments.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def encode_chunk(
+ self,
+ speech: torch.Tensor,
+ speech_lengths: torch.Tensor,
+ cache: dict = None,
+ **kwargs,
+ ) -&gt; Tuple[torch.Tensor, torch.Tensor]:
+ &quot;&quot;&quot;Encode chunk.
+
+ Args:
+ speech: Speech audio tensor, shape (batch, time).
+ speech_lengths: Length of each speech sample.
+ cache: State cache dict for streaming inference.
+ **kwargs: Additional keyword arguments.
+ &quot;&quot;&quot;
+ if cache is None:
+ cache = {}
+ &quot;&quot;&quot;Frontend + Encoder. Note that this method is used by asr_inference.py
+ Args:
+ speech: (Batch, Length, ...)
+ speech_lengths: (Batch, )
+ ind: int
+ &quot;&quot;&quot;
+ with autocast(False):
+ # Data augmentation
+ if self.specaug is not None and self.training:
+ speech, speech_lengths = self.specaug(speech, speech_lengths)
+
+ # Normalization for feature: e.g. Global-CMVN, Utterance-CMVN
+ if self.normalize is not None:
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/sanm_kws_streaming/model.py#L119" target="_blank" class="src-more">View full source on GitHub →</a></details></div>
+<div class="api-detail" id="e205"><span class="dtag badge-m">method</span><h2><code>SanmKWSStreaming.init_cache(cache, **kwargs)</code></h2><div class="dmod">funasr.models.sanm_kws_streaming.SanmKWSStreaming · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/sanm_kws_streaming/model.py#L161" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Init cache.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>cache</code> — State cache dict for streaming inference.</li>
+<li><code>**kwargs</code> — Additional keyword arguments.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def init_cache(self, cache: dict = None, **kwargs):
+ &quot;&quot;&quot;Init cache.
+
+ Args:
+ cache: State cache dict for streaming inference.
+ **kwargs: Additional keyword arguments.
+ &quot;&quot;&quot;
+ if cache is None:
+ cache = {}
+ chunk_size = kwargs.get(&quot;chunk_size&quot;, [0, 10, 5])
+ encoder_chunk_look_back = kwargs.get(&quot;encoder_chunk_look_back&quot;, 0)
+ decoder_chunk_look_back = kwargs.get(&quot;decoder_chunk_look_back&quot;, 0)
+ batch_size = 1
+
+ enc_output_size = kwargs[&quot;encoder_conf&quot;][&quot;output_size&quot;]
+ feats_dims = kwargs[&quot;frontend_conf&quot;][&quot;n_mels&quot;] * kwargs[&quot;frontend_conf&quot;][&quot;lfr_m&quot;]
+ cache_encoder = {
+ &quot;start_idx&quot;: 0,
+ &quot;cif_hidden&quot;: torch.zeros((batch_size, 1, enc_output_size)),
+ &quot;cif_alphas&quot;: torch.zeros((batch_size, 1)),
+ &quot;encoder_out&quot;: None,
+ &quot;encoder_out_lens&quot;: None,
+ &quot;chunk_size&quot;: chunk_size,
+ &quot;encoder_chunk_look_back&quot;: encoder_chunk_look_back,
+ &quot;last_chunk&quot;: False,
+ &quot;opt&quot;: None,
+ &quot;feats&quot;: torch.zeros((batch_size, chunk_size[0] + chunk_size[2], feats_dims)),
+ &quot;tail_chunk&quot;: False,
+ }
+ cache[&quot;encoder&quot;] = cache_encoder
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/sanm_kws_streaming/model.py#L161" target="_blank" class="src-more">View full source on GitHub →</a></details></div>
+<div class="api-detail" id="e206"><span class="dtag badge-m">method</span><h2><code>SanmKWSStreaming.generate_chunk(speech, speech_lengths, key, tokenizer, frontend, **kwargs)</code></h2><div class="dmod">funasr.models.sanm_kws_streaming.SanmKWSStreaming · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/sanm_kws_streaming/model.py#L204" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Generate chunk.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>speech</code> — Speech audio tensor, shape (batch, time).</li>
+<li><code>speech_lengths</code> — Length of each speech sample.</li>
+<li><code>key</code> — Sample identifiers.</li>
+<li><code>tokenizer</code> — Tokenizer instance for text encoding/decoding.</li>
+<li><code>frontend</code> — Audio frontend for feature extraction.</li>
+<li><code>**kwargs</code> — Additional keyword arguments.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def generate_chunk(
+ self,
+ speech,
+ speech_lengths=None,
+ key: list = None,
+ tokenizer=None,
+ frontend=None,
+ **kwargs,
+ ):
+ &quot;&quot;&quot;Generate chunk.
+
+ Args:
+ speech: Speech audio tensor, shape (batch, time).
+ speech_lengths: Length of each speech sample.
+ key: Sample identifiers.
+ tokenizer: Tokenizer instance for text encoding/decoding.
+ frontend: Audio frontend for feature extraction.
+ **kwargs: Additional keyword arguments.
+ &quot;&quot;&quot;
+ cache = kwargs.get(&quot;cache&quot;, {})
+ speech = speech.to(device=kwargs[&quot;device&quot;])
+ speech_lengths = speech_lengths.to(device=kwargs[&quot;device&quot;])
+
+ # Encoder
+ is_final = kwargs.get(&quot;is_final&quot;, False)
+ encoder_out, encoder_out_lens = self.encode_chunk(
+ speech, speech_lengths, cache=cache, is_final=is_final
+ )
+ if isinstance(encoder_out, tuple):
+ encoder_out = encoder_out[0]
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/sanm_kws_streaming/model.py#L204" target="_blank" class="src-more">View full source on GitHub →</a></details></div>
+<div class="api-detail" id="e207"><span class="dtag badge-m">method</span><h2><code>SanmKWSStreaming.inference(data_in, data_lengths, key, tokenizer, frontend, cache, **kwargs)</code></h2><div class="dmod">funasr.models.sanm_kws_streaming.SanmKWSStreaming · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/sanm_kws_streaming/model.py#L287" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Run inference on input data.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>data_in</code> — Input data (audio samples, file paths, or text).</li>
+<li><code>data_lengths</code> — Lengths of each input sample in the batch.</li>
+<li><code>key</code> — Sample identifiers.</li>
+<li><code>tokenizer</code> — Tokenizer instance for text encoding/decoding.</li>
+<li><code>frontend</code> — Audio frontend for feature extraction.</li>
+<li><code>cache</code> — State cache dict for streaming inference.</li>
+<li><code>**kwargs</code> — Additional keyword arguments.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def inference(
+ self,
+ data_in,
+ data_lengths=None,
+ key: list = None,
+ tokenizer=None,
+ frontend=None,
+ cache: dict = None,
+ **kwargs,
+ ):
+ &quot;&quot;&quot;Run inference on input data.
+
+ Args:
+ data_in: Input data (audio samples, file paths, or text).
+ data_lengths: Lengths of each input sample in the batch.
+ key: Sample identifiers.
+ tokenizer: Tokenizer instance for text encoding/decoding.
+ frontend: Audio frontend for feature extraction.
+ cache: State cache dict for streaming inference.
+ **kwargs: Additional keyword arguments.
+ &quot;&quot;&quot;
+ if cache is None:
+ cache = {}
+ keywords = kwargs.get(&quot;keywords&quot;)
+ from funasr.utils.kws_utils import KwsCtcPrefixDecoder
+ self.kws_decoder = KwsCtcPrefixDecoder(
+ ctc=self.ctc,
+ keywords=keywords,
+ token_list=tokenizer.token_list,
+ seg_dict=tokenizer.seg_dict,
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/sanm_kws_streaming/model.py#L287" target="_blank" class="src-more">View full source on GitHub →</a></details></div>
+<div class="api-detail" id="e208"><span class="dtag badge-m">method</span><h2><code>SanmKWSStreaming.export(**kwargs)</code></h2><div class="dmod">funasr.models.sanm_kws_streaming.SanmKWSStreaming · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/sanm_kws_streaming/model.py#L490" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Export.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>**kwargs</code> — Additional keyword arguments.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def export(self, **kwargs):
+ &quot;&quot;&quot;Export.
+
+ Args:
+ **kwargs: Additional keyword arguments.
+ &quot;&quot;&quot;
+ from .export_meta import export_rebuild_model
+
+ models = export_rebuild_model(model=self, **kwargs)
+ return models
+</code></pre></details></div>
+<div class="api-detail" id="e209"><span class="dtag badge-c">class</span><h2>SCAMA</h2><div class="dmod">funasr.models.scama · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/scama/model.py#L45" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><ul>
+<li><code>SCAMA</code> — Streaming Chunk-Aware Multi-head Attention ASR.</li>
+</ul>
+<p>Streaming ASR using chunk-based encoder with configurable latency.</p>
+<p>Supports 2-pass decoding for accuracy refinement.</p>
+<ul>
+<li><code>Output</code> — {&quot;key&quot;: str, &quot;text&quot;: str}</li>
+<li><code>Author</code> — Shiliang Zhang, Zhifu Gao, Haoneng Luo, Ming Lei, Jie Gao, Zhijie Yan, Lei Xie</li>
+<li><code>SCAMA</code> — Streaming chunk-aware multihead attention for online end-to-end speech recognition</li>
+</ul>
+<p>https://arxiv.org/abs/2006.01712</p>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code>class SCAMA(nn.Module):
+ &quot;&quot;&quot;SCAMA: Streaming Chunk-Aware Multi-head Attention ASR.
+
+ Streaming ASR using chunk-based encoder with configurable latency.
+ Supports 2-pass decoding for accuracy refinement.
+
+ Output: {&quot;key&quot;: str, &quot;text&quot;: str}
+
+ Author: Shiliang Zhang, Zhifu Gao, Haoneng Luo, Ming Lei, Jie Gao, Zhijie Yan, Lei Xie
+ SCAMA: Streaming chunk-aware multihead attention for online end-to-end speech recognition
+ https://arxiv.org/abs/2006.01712
+ &quot;&quot;&quot;
+
+ def __init__(
+ self,
+ specaug: str = None,
+ specaug_conf: dict = None,
+ normalize: str = None,
+ normalize_conf: dict = None,
+ encoder: str = None,
+ encoder_conf: dict = None,
+ decoder: str = None,
+ decoder_conf: dict = None,
+ ctc: str = None,
+ ctc_conf: dict = None,
+ ctc_weight: float = 0.5,
+ predictor: str = None,
+ predictor_conf: dict = None,
+ predictor_bias: int = 0,
+ predictor_weight: float = 0.0,
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/scama/model.py#L45" target="_blank" class="src-more">View full source on GitHub →</a></details><h3 class="mt">Methods</h3><div class="mblk"><code>.forward(speech, speech_lengths, text, text_lengths, **kwargs)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/scama/model.py#L200" target="_blank" class="gh-link">L200</a><div class="mdoc"><p>Encoder + Decoder + Calc loss</p>
+<h4>Args:</h4>
+<ul>
+<li><code>speech</code> — (Batch, Length, ...)</li>
+<li><code>speech_lengths</code> — (Batch, )</li>
+<li><code>text</code> — (Batch, Length)</li>
+<li><code>text_lengths</code> — (Batch,)</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def forward(
+ self,
+ speech: torch.Tensor,
+ speech_lengths: torch.Tensor,
+ text: torch.Tensor,
+ text_lengths: torch.Tensor,
+ **kwargs,
+ ) -&gt; Tuple[torch.Tensor, Dict[str, torch.Tensor], torch.Tensor]:
+ &quot;&quot;&quot;Encoder + Decoder + Calc loss
+ Args:
+ speech: (Batch, Length, ...)
+ speech_lengths: (Batch, )
+ text: (Batch, Length)
+ text_lengths: (Batch,)
+ &quot;&quot;&quot;
+
+ decoding_ind = kwargs.get(&quot;decoding_ind&quot;)
+ if len(text_lengths.size()) &gt; 1:
+ text_lengths = text_lengths[:, 0]
+ if len(speech_lengths.size()) &gt; 1:
+ speech_lengths = speech_lengths[:, 0]
+
+ batch_size = speech.shape[0]
+
+ # Encoder
+ ind = self.encoder.overlap_chunk_cls.random_choice(self.training, decoding_ind)
+ encoder_out, encoder_out_lens = self.encode(speech, speech_lengths, ind=ind)
+
+ loss_ctc, cer_ctc = None, None
+ loss_pre = None
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/scama/model.py#L200" target="_blank" class="src-more">View full source →</a></details></div><div class="mblk"><code>.encode(speech, speech_lengths, **kwargs)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/scama/model.py#L277" target="_blank" class="gh-link">L277</a><div class="mdoc"><p>Encoder. Note that this method is used by asr_inference.py</p>
+<h4>Args:</h4>
+<ul>
+<li><code>speech</code> — (Batch, Length, ...)</li>
+<li><code>speech_lengths</code> — (Batch, )</li>
+<li><code>ind</code> — int</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def encode(
+ self,
+ speech: torch.Tensor,
+ speech_lengths: torch.Tensor,
+ **kwargs,
+ ) -&gt; Tuple[torch.Tensor, torch.Tensor]:
+ &quot;&quot;&quot;Encoder. Note that this method is used by asr_inference.py
+ Args:
+ speech: (Batch, Length, ...)
+ speech_lengths: (Batch, )
+ ind: int
+ &quot;&quot;&quot;
+ with autocast(False):
+
+ # Data augmentation
+ if self.specaug is not None and self.training:
+ speech, speech_lengths = self.specaug(speech, speech_lengths)
+
+ # Normalization for feature: e.g. Global-CMVN, Utterance-CMVN
+ if self.normalize is not None:
+ speech, speech_lengths = self.normalize(speech, speech_lengths)
+
+ # Forward encoder
+ encoder_out, encoder_out_lens, _ = self.encoder(speech, speech_lengths)
+ if isinstance(encoder_out, tuple):
+ encoder_out = encoder_out[0]
+
+ return encoder_out, encoder_out_lens
+</code></pre></details></div><div class="mblk"><code>.encode_chunk(speech, speech_lengths, cache, **kwargs)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/scama/model.py#L306" target="_blank" class="gh-link">L306</a><div class="mdoc"><p>Encode chunk.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>speech</code> — Speech audio tensor, shape (batch, time).</li>
+<li><code>speech_lengths</code> — Length of each speech sample.</li>
+<li><code>cache</code> — State cache dict for streaming inference.</li>
+<li><code>**kwargs</code> — Additional keyword arguments.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def encode_chunk(
+ self,
+ speech: torch.Tensor,
+ speech_lengths: torch.Tensor,
+ cache: dict = None,
+ **kwargs,
+ ) -&gt; Tuple[torch.Tensor, torch.Tensor]:
+ &quot;&quot;&quot;Encode chunk.
+
+ Args:
+ speech: Speech audio tensor, shape (batch, time).
+ speech_lengths: Length of each speech sample.
+ cache: State cache dict for streaming inference.
+ **kwargs: Additional keyword arguments.
+ &quot;&quot;&quot;
+ if cache is None:
+ cache = {}
+ &quot;&quot;&quot;Frontend + Encoder. Note that this method is used by asr_inference.py
+ Args:
+ speech: (Batch, Length, ...)
+ speech_lengths: (Batch, )
+ ind: int
+ &quot;&quot;&quot;
+ with autocast(False):
+
+ # Data augmentation
+ if self.specaug is not None and self.training:
+ speech, speech_lengths = self.specaug(speech, speech_lengths)
+
+ # Normalization for feature: e.g. Global-CMVN, Utterance-CMVN
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/scama/model.py#L306" target="_blank" class="src-more">View full source →</a></details></div><div class="mblk"><code>.calc_predictor_chunk(encoder_out, encoder_out_lens, cache, **kwargs)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/scama/model.py#L348" target="_blank" class="gh-link">L348</a><div class="mdoc"><p>Calc predictor chunk.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>encoder_out</code> — Encoder output tensor.</li>
+<li><code>encoder_out_lens</code> — Encoder output lengths.</li>
+<li><code>cache</code> — State cache dict for streaming inference.</li>
+<li><code>**kwargs</code> — Additional keyword arguments.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def calc_predictor_chunk(self, encoder_out, encoder_out_lens, cache=None, **kwargs):
+ &quot;&quot;&quot;Calc predictor chunk.
+
+ Args:
+ encoder_out: Encoder output tensor.
+ encoder_out_lens: Encoder output lengths.
+ cache: State cache dict for streaming inference.
+ **kwargs: Additional keyword arguments.
+ &quot;&quot;&quot;
+ is_final = kwargs.get(&quot;is_final&quot;, False)
+
+ return self.predictor.forward_chunk(encoder_out, cache[&quot;encoder&quot;], is_final=is_final)
+</code></pre></details></div><div class="mblk"><code>.calc_predictor_mask(encoder_out, encoder_out_lens, ys_pad, ys_pad_lens)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/scama/model.py#L462" target="_blank" class="gh-link">L462</a><div class="mdoc"><p>Calc predictor mask.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>encoder_out</code> — Encoder output tensor.</li>
+<li><code>encoder_out_lens</code> — Encoder output lengths.</li>
+<li><code>ys_pad</code> — TODO.</li>
+<li><code>ys_pad_lens</code> — Lengths of ys_pad.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def calc_predictor_mask(
+ self,
+ encoder_out: torch.Tensor,
+ encoder_out_lens: torch.Tensor,
+ ys_pad: torch.Tensor = None,
+ ys_pad_lens: torch.Tensor = None,
+ ):
+ # ys_in_pad, ys_out_pad = add_sos_eos(ys_pad, self.sos, self.eos, self.ignore_id)
+ # ys_in_lens = ys_pad_lens + 1
+ &quot;&quot;&quot;Calc predictor mask.
+
+ Args:
+ encoder_out: Encoder output tensor.
+ encoder_out_lens: Encoder output lengths.
+ ys_pad: TODO.
+ ys_pad_lens: Lengths of ys_pad.
+ &quot;&quot;&quot;
+ ys_out_pad, ys_in_lens = None, None
+
+ encoder_out_mask = sequence_mask(
+ encoder_out_lens,
+ maxlen=encoder_out.size(1),
+ dtype=encoder_out.dtype,
+ device=encoder_out.device,
+ )[:, None, :]
+ mask_chunk_predictor = None
+
+ mask_chunk_predictor = self.encoder.overlap_chunk_cls.get_mask_chunk_predictor(
+ None, device=encoder_out.device, batch_size=encoder_out.size(0)
+ )
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/scama/model.py#L462" target="_blank" class="src-more">View full source →</a></details></div><div class="mblk"><code>.init_beam_search(**kwargs)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/scama/model.py#L543" target="_blank" class="gh-link">L543</a><div class="mdoc"><p>Init beam search.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>**kwargs</code> — Additional keyword arguments.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def init_beam_search(
+ self,
+ **kwargs,
+ ):
+
+ &quot;&quot;&quot;Init beam search.
+
+ Args:
+ **kwargs: Additional keyword arguments.
+ &quot;&quot;&quot;
+ from funasr.models.scama.beam_search import BeamSearchScamaStreaming
+
+ from funasr.models.transformer.scorers.ctc import CTCPrefixScorer
+ from funasr.models.transformer.scorers.length_bonus import LengthBonus
+
+ # 1. Build ASR model
+ scorers = {}
+
+ if self.ctc != None:
+ ctc = CTCPrefixScorer(ctc=self.ctc, eos=self.eos)
+ scorers.update(ctc=ctc)
+ token_list = kwargs.get(&quot;token_list&quot;)
+ scorers.update(
+ decoder=self.decoder,
+ length_bonus=LengthBonus(len(token_list)),
+ )
+
+ # 3. Build ngram model
+ # ngram is not supported now
+ ngram = None
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/scama/model.py#L543" target="_blank" class="src-more">View full source →</a></details></div><div class="mblk"><code>.generate_chunk(speech, speech_lengths, key, tokenizer, frontend, **kwargs)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/scama/model.py#L599" target="_blank" class="gh-link">L599</a><div class="mdoc"><p>Generate chunk.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>speech</code> — Speech audio tensor, shape (batch, time).</li>
+<li><code>speech_lengths</code> — Length of each speech sample.</li>
+<li><code>key</code> — Sample identifiers.</li>
+<li><code>tokenizer</code> — Tokenizer instance for text encoding/decoding.</li>
+<li><code>frontend</code> — Audio frontend for feature extraction.</li>
+<li><code>**kwargs</code> — Additional keyword arguments.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def generate_chunk(
+ self,
+ speech,
+ speech_lengths=None,
+ key: list = None,
+ tokenizer=None,
+ frontend=None,
+ **kwargs,
+ ):
+ &quot;&quot;&quot;Generate chunk.
+
+ Args:
+ speech: Speech audio tensor, shape (batch, time).
+ speech_lengths: Length of each speech sample.
+ key: Sample identifiers.
+ tokenizer: Tokenizer instance for text encoding/decoding.
+ frontend: Audio frontend for feature extraction.
+ **kwargs: Additional keyword arguments.
+ &quot;&quot;&quot;
+ cache = kwargs.get(&quot;cache&quot;, {})
+ speech = speech.to(device=kwargs[&quot;device&quot;])
+ speech_lengths = speech_lengths.to(device=kwargs[&quot;device&quot;])
+
+ # Encoder
+ encoder_out, encoder_out_lens = self.encode_chunk(
+ speech, speech_lengths, cache=cache, is_final=kwargs.get(&quot;is_final&quot;, False)
+ )
+ if isinstance(encoder_out, tuple):
+ encoder_out = encoder_out[0]
+ if &quot;running_hyps&quot; not in cache:
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/scama/model.py#L599" target="_blank" class="src-more">View full source →</a></details></div><div class="mblk"><code>.init_cache(cache, **kwargs)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/scama/model.py#L694" target="_blank" class="gh-link">L694</a><div class="mdoc"><p>Init cache.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>cache</code> — State cache dict for streaming inference.</li>
+<li><code>**kwargs</code> — Additional keyword arguments.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def init_cache(self, cache: dict = None, **kwargs):
+ &quot;&quot;&quot;Init cache.
+
+ Args:
+ cache: State cache dict for streaming inference.
+ **kwargs: Additional keyword arguments.
+ &quot;&quot;&quot;
+ if cache is None:
+ cache = {}
+ device = kwargs.get(&quot;device&quot;, &quot;cuda&quot;)
+
+ chunk_size = kwargs.get(&quot;chunk_size&quot;, [0, 10, 5])
+ encoder_chunk_look_back = kwargs.get(&quot;encoder_chunk_look_back&quot;, 0)
+ decoder_chunk_look_back = kwargs.get(&quot;decoder_chunk_look_back&quot;, 0)
+ batch_size = 1
+
+ enc_output_size = kwargs[&quot;encoder_conf&quot;][&quot;output_size&quot;]
+ feats_dims = kwargs[&quot;frontend_conf&quot;][&quot;n_mels&quot;] * kwargs[&quot;frontend_conf&quot;][&quot;lfr_m&quot;]
+
+ cache_encoder = {
+ &quot;start_idx&quot;: 0,
+ &quot;cif_hidden&quot;: torch.zeros((batch_size, 1, enc_output_size)).to(device=device),
+ &quot;cif_alphas&quot;: torch.zeros((batch_size, 1)).to(device=device),
+ &quot;chunk_size&quot;: chunk_size,
+ &quot;encoder_chunk_look_back&quot;: encoder_chunk_look_back,
+ &quot;last_chunk&quot;: False,
+ &quot;opt&quot;: None,
+ &quot;feats&quot;: torch.zeros((batch_size, chunk_size[0] + chunk_size[2], feats_dims)).to(
+ device=device
+ ),
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/scama/model.py#L694" target="_blank" class="src-more">View full source →</a></details></div><div class="mblk"><code>.inference(data_in, data_lengths, key, tokenizer, frontend, cache, **kwargs)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/scama/model.py#L741" target="_blank" class="gh-link">L741</a><div class="mdoc"><p>Run inference on input data.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>data_in</code> — Input data (audio samples, file paths, or text).</li>
+<li><code>data_lengths</code> — Lengths of each input sample in the batch.</li>
+<li><code>key</code> — Sample identifiers.</li>
+<li><code>tokenizer</code> — Tokenizer instance for text encoding/decoding.</li>
+<li><code>frontend</code> — Audio frontend for feature extraction.</li>
+<li><code>cache</code> — State cache dict for streaming inference.</li>
+<li><code>**kwargs</code> — Additional keyword arguments.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def inference(
+ self,
+ data_in,
+ data_lengths=None,
+ key: list = None,
+ tokenizer=None,
+ frontend=None,
+ cache: dict = None,
+ **kwargs,
+ ):
+ &quot;&quot;&quot;Run inference on input data.
+
+ Args:
+ data_in: Input data (audio samples, file paths, or text).
+ data_lengths: Lengths of each input sample in the batch.
+ key: Sample identifiers.
+ tokenizer: Tokenizer instance for text encoding/decoding.
+ frontend: Audio frontend for feature extraction.
+ cache: State cache dict for streaming inference.
+ **kwargs: Additional keyword arguments.
+ &quot;&quot;&quot;
+ if cache is None:
+ cache = {}
+
+ # init beamsearch
+ is_use_ctc = kwargs.get(&quot;decoding_ctc_weight&quot;, 0.0) &gt; 0.00001 and self.ctc != None
+ is_use_lm = (
+ kwargs.get(&quot;lm_weight&quot;, 0.0) &gt; 0.00001 and kwargs.get(&quot;lm_file&quot;, None) is not None
+ )
+
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/scama/model.py#L741" target="_blank" class="src-more">View full source →</a></details></div></div>
+<div class="api-detail" id="e210"><span class="dtag badge-m">method</span><h2><code>SCAMA.forward(speech, speech_lengths, text, text_lengths, **kwargs)</code></h2><div class="dmod">funasr.models.scama.SCAMA · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/scama/model.py#L200" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Encoder + Decoder + Calc loss</p>
+<h4>Args:</h4>
+<ul>
+<li><code>speech</code> — (Batch, Length, ...)</li>
+<li><code>speech_lengths</code> — (Batch, )</li>
+<li><code>text</code> — (Batch, Length)</li>
+<li><code>text_lengths</code> — (Batch,)</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def forward(
+ self,
+ speech: torch.Tensor,
+ speech_lengths: torch.Tensor,
+ text: torch.Tensor,
+ text_lengths: torch.Tensor,
+ **kwargs,
+ ) -&gt; Tuple[torch.Tensor, Dict[str, torch.Tensor], torch.Tensor]:
+ &quot;&quot;&quot;Encoder + Decoder + Calc loss
+ Args:
+ speech: (Batch, Length, ...)
+ speech_lengths: (Batch, )
+ text: (Batch, Length)
+ text_lengths: (Batch,)
+ &quot;&quot;&quot;
+
+ decoding_ind = kwargs.get(&quot;decoding_ind&quot;)
+ if len(text_lengths.size()) &gt; 1:
+ text_lengths = text_lengths[:, 0]
+ if len(speech_lengths.size()) &gt; 1:
+ speech_lengths = speech_lengths[:, 0]
+
+ batch_size = speech.shape[0]
+
+ # Encoder
+ ind = self.encoder.overlap_chunk_cls.random_choice(self.training, decoding_ind)
+ encoder_out, encoder_out_lens = self.encode(speech, speech_lengths, ind=ind)
+
+ loss_ctc, cer_ctc = None, None
+ loss_pre = None
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/scama/model.py#L200" target="_blank" class="src-more">View full source on GitHub →</a></details></div>
+<div class="api-detail" id="e211"><span class="dtag badge-m">method</span><h2><code>SCAMA.encode(speech, speech_lengths, **kwargs)</code></h2><div class="dmod">funasr.models.scama.SCAMA · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/scama/model.py#L277" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Encoder. Note that this method is used by asr_inference.py</p>
+<h4>Args:</h4>
+<ul>
+<li><code>speech</code> — (Batch, Length, ...)</li>
+<li><code>speech_lengths</code> — (Batch, )</li>
+<li><code>ind</code> — int</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def encode(
+ self,
+ speech: torch.Tensor,
+ speech_lengths: torch.Tensor,
+ **kwargs,
+ ) -&gt; Tuple[torch.Tensor, torch.Tensor]:
+ &quot;&quot;&quot;Encoder. Note that this method is used by asr_inference.py
+ Args:
+ speech: (Batch, Length, ...)
+ speech_lengths: (Batch, )
+ ind: int
+ &quot;&quot;&quot;
+ with autocast(False):
+
+ # Data augmentation
+ if self.specaug is not None and self.training:
+ speech, speech_lengths = self.specaug(speech, speech_lengths)
+
+ # Normalization for feature: e.g. Global-CMVN, Utterance-CMVN
+ if self.normalize is not None:
+ speech, speech_lengths = self.normalize(speech, speech_lengths)
+
+ # Forward encoder
+ encoder_out, encoder_out_lens, _ = self.encoder(speech, speech_lengths)
+ if isinstance(encoder_out, tuple):
+ encoder_out = encoder_out[0]
+
+ return encoder_out, encoder_out_lens
+</code></pre></details></div>
+<div class="api-detail" id="e212"><span class="dtag badge-m">method</span><h2><code>SCAMA.encode_chunk(speech, speech_lengths, cache, **kwargs)</code></h2><div class="dmod">funasr.models.scama.SCAMA · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/scama/model.py#L306" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Encode chunk.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>speech</code> — Speech audio tensor, shape (batch, time).</li>
+<li><code>speech_lengths</code> — Length of each speech sample.</li>
+<li><code>cache</code> — State cache dict for streaming inference.</li>
+<li><code>**kwargs</code> — Additional keyword arguments.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def encode_chunk(
+ self,
+ speech: torch.Tensor,
+ speech_lengths: torch.Tensor,
+ cache: dict = None,
+ **kwargs,
+ ) -&gt; Tuple[torch.Tensor, torch.Tensor]:
+ &quot;&quot;&quot;Encode chunk.
+
+ Args:
+ speech: Speech audio tensor, shape (batch, time).
+ speech_lengths: Length of each speech sample.
+ cache: State cache dict for streaming inference.
+ **kwargs: Additional keyword arguments.
+ &quot;&quot;&quot;
+ if cache is None:
+ cache = {}
+ &quot;&quot;&quot;Frontend + Encoder. Note that this method is used by asr_inference.py
+ Args:
+ speech: (Batch, Length, ...)
+ speech_lengths: (Batch, )
+ ind: int
+ &quot;&quot;&quot;
+ with autocast(False):
+
+ # Data augmentation
+ if self.specaug is not None and self.training:
+ speech, speech_lengths = self.specaug(speech, speech_lengths)
+
+ # Normalization for feature: e.g. Global-CMVN, Utterance-CMVN
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/scama/model.py#L306" target="_blank" class="src-more">View full source on GitHub →</a></details></div>
+<div class="api-detail" id="e213"><span class="dtag badge-m">method</span><h2><code>SCAMA.calc_predictor_chunk(encoder_out, encoder_out_lens, cache, **kwargs)</code></h2><div class="dmod">funasr.models.scama.SCAMA · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/scama/model.py#L348" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Calc predictor chunk.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>encoder_out</code> — Encoder output tensor.</li>
+<li><code>encoder_out_lens</code> — Encoder output lengths.</li>
+<li><code>cache</code> — State cache dict for streaming inference.</li>
+<li><code>**kwargs</code> — Additional keyword arguments.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def calc_predictor_chunk(self, encoder_out, encoder_out_lens, cache=None, **kwargs):
+ &quot;&quot;&quot;Calc predictor chunk.
+
+ Args:
+ encoder_out: Encoder output tensor.
+ encoder_out_lens: Encoder output lengths.
+ cache: State cache dict for streaming inference.
+ **kwargs: Additional keyword arguments.
+ &quot;&quot;&quot;
+ is_final = kwargs.get(&quot;is_final&quot;, False)
+
+ return self.predictor.forward_chunk(encoder_out, cache[&quot;encoder&quot;], is_final=is_final)
+</code></pre></details></div>
+<div class="api-detail" id="e214"><span class="dtag badge-m">method</span><h2><code>SCAMA.calc_predictor_mask(encoder_out, encoder_out_lens, ys_pad, ys_pad_lens)</code></h2><div class="dmod">funasr.models.scama.SCAMA · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/scama/model.py#L462" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Calc predictor mask.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>encoder_out</code> — Encoder output tensor.</li>
+<li><code>encoder_out_lens</code> — Encoder output lengths.</li>
+<li><code>ys_pad</code> — TODO.</li>
+<li><code>ys_pad_lens</code> — Lengths of ys_pad.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def calc_predictor_mask(
+ self,
+ encoder_out: torch.Tensor,
+ encoder_out_lens: torch.Tensor,
+ ys_pad: torch.Tensor = None,
+ ys_pad_lens: torch.Tensor = None,
+ ):
+ # ys_in_pad, ys_out_pad = add_sos_eos(ys_pad, self.sos, self.eos, self.ignore_id)
+ # ys_in_lens = ys_pad_lens + 1
+ &quot;&quot;&quot;Calc predictor mask.
+
+ Args:
+ encoder_out: Encoder output tensor.
+ encoder_out_lens: Encoder output lengths.
+ ys_pad: TODO.
+ ys_pad_lens: Lengths of ys_pad.
+ &quot;&quot;&quot;
+ ys_out_pad, ys_in_lens = None, None
+
+ encoder_out_mask = sequence_mask(
+ encoder_out_lens,
+ maxlen=encoder_out.size(1),
+ dtype=encoder_out.dtype,
+ device=encoder_out.device,
+ )[:, None, :]
+ mask_chunk_predictor = None
+
+ mask_chunk_predictor = self.encoder.overlap_chunk_cls.get_mask_chunk_predictor(
+ None, device=encoder_out.device, batch_size=encoder_out.size(0)
+ )
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/scama/model.py#L462" target="_blank" class="src-more">View full source on GitHub →</a></details></div>
+<div class="api-detail" id="e215"><span class="dtag badge-m">method</span><h2><code>SCAMA.init_beam_search(**kwargs)</code></h2><div class="dmod">funasr.models.scama.SCAMA · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/scama/model.py#L543" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Init beam search.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>**kwargs</code> — Additional keyword arguments.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def init_beam_search(
+ self,
+ **kwargs,
+ ):
+
+ &quot;&quot;&quot;Init beam search.
+
+ Args:
+ **kwargs: Additional keyword arguments.
+ &quot;&quot;&quot;
+ from funasr.models.scama.beam_search import BeamSearchScamaStreaming
+
+ from funasr.models.transformer.scorers.ctc import CTCPrefixScorer
+ from funasr.models.transformer.scorers.length_bonus import LengthBonus
+
+ # 1. Build ASR model
+ scorers = {}
+
+ if self.ctc != None:
+ ctc = CTCPrefixScorer(ctc=self.ctc, eos=self.eos)
+ scorers.update(ctc=ctc)
+ token_list = kwargs.get(&quot;token_list&quot;)
+ scorers.update(
+ decoder=self.decoder,
+ length_bonus=LengthBonus(len(token_list)),
+ )
+
+ # 3. Build ngram model
+ # ngram is not supported now
+ ngram = None
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/scama/model.py#L543" target="_blank" class="src-more">View full source on GitHub →</a></details></div>
+<div class="api-detail" id="e216"><span class="dtag badge-m">method</span><h2><code>SCAMA.generate_chunk(speech, speech_lengths, key, tokenizer, frontend, **kwargs)</code></h2><div class="dmod">funasr.models.scama.SCAMA · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/scama/model.py#L599" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Generate chunk.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>speech</code> — Speech audio tensor, shape (batch, time).</li>
+<li><code>speech_lengths</code> — Length of each speech sample.</li>
+<li><code>key</code> — Sample identifiers.</li>
+<li><code>tokenizer</code> — Tokenizer instance for text encoding/decoding.</li>
+<li><code>frontend</code> — Audio frontend for feature extraction.</li>
+<li><code>**kwargs</code> — Additional keyword arguments.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def generate_chunk(
+ self,
+ speech,
+ speech_lengths=None,
+ key: list = None,
+ tokenizer=None,
+ frontend=None,
+ **kwargs,
+ ):
+ &quot;&quot;&quot;Generate chunk.
+
+ Args:
+ speech: Speech audio tensor, shape (batch, time).
+ speech_lengths: Length of each speech sample.
+ key: Sample identifiers.
+ tokenizer: Tokenizer instance for text encoding/decoding.
+ frontend: Audio frontend for feature extraction.
+ **kwargs: Additional keyword arguments.
+ &quot;&quot;&quot;
+ cache = kwargs.get(&quot;cache&quot;, {})
+ speech = speech.to(device=kwargs[&quot;device&quot;])
+ speech_lengths = speech_lengths.to(device=kwargs[&quot;device&quot;])
+
+ # Encoder
+ encoder_out, encoder_out_lens = self.encode_chunk(
+ speech, speech_lengths, cache=cache, is_final=kwargs.get(&quot;is_final&quot;, False)
+ )
+ if isinstance(encoder_out, tuple):
+ encoder_out = encoder_out[0]
+ if &quot;running_hyps&quot; not in cache:
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/scama/model.py#L599" target="_blank" class="src-more">View full source on GitHub →</a></details></div>
+<div class="api-detail" id="e217"><span class="dtag badge-m">method</span><h2><code>SCAMA.init_cache(cache, **kwargs)</code></h2><div class="dmod">funasr.models.scama.SCAMA · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/scama/model.py#L694" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Init cache.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>cache</code> — State cache dict for streaming inference.</li>
+<li><code>**kwargs</code> — Additional keyword arguments.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def init_cache(self, cache: dict = None, **kwargs):
+ &quot;&quot;&quot;Init cache.
+
+ Args:
+ cache: State cache dict for streaming inference.
+ **kwargs: Additional keyword arguments.
+ &quot;&quot;&quot;
+ if cache is None:
+ cache = {}
+ device = kwargs.get(&quot;device&quot;, &quot;cuda&quot;)
+
+ chunk_size = kwargs.get(&quot;chunk_size&quot;, [0, 10, 5])
+ encoder_chunk_look_back = kwargs.get(&quot;encoder_chunk_look_back&quot;, 0)
+ decoder_chunk_look_back = kwargs.get(&quot;decoder_chunk_look_back&quot;, 0)
+ batch_size = 1
+
+ enc_output_size = kwargs[&quot;encoder_conf&quot;][&quot;output_size&quot;]
+ feats_dims = kwargs[&quot;frontend_conf&quot;][&quot;n_mels&quot;] * kwargs[&quot;frontend_conf&quot;][&quot;lfr_m&quot;]
+
+ cache_encoder = {
+ &quot;start_idx&quot;: 0,
+ &quot;cif_hidden&quot;: torch.zeros((batch_size, 1, enc_output_size)).to(device=device),
+ &quot;cif_alphas&quot;: torch.zeros((batch_size, 1)).to(device=device),
+ &quot;chunk_size&quot;: chunk_size,
+ &quot;encoder_chunk_look_back&quot;: encoder_chunk_look_back,
+ &quot;last_chunk&quot;: False,
+ &quot;opt&quot;: None,
+ &quot;feats&quot;: torch.zeros((batch_size, chunk_size[0] + chunk_size[2], feats_dims)).to(
+ device=device
+ ),
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/scama/model.py#L694" target="_blank" class="src-more">View full source on GitHub →</a></details></div>
+<div class="api-detail" id="e218"><span class="dtag badge-m">method</span><h2><code>SCAMA.inference(data_in, data_lengths, key, tokenizer, frontend, cache, **kwargs)</code></h2><div class="dmod">funasr.models.scama.SCAMA · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/scama/model.py#L741" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Run inference on input data.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>data_in</code> — Input data (audio samples, file paths, or text).</li>
+<li><code>data_lengths</code> — Lengths of each input sample in the batch.</li>
+<li><code>key</code> — Sample identifiers.</li>
+<li><code>tokenizer</code> — Tokenizer instance for text encoding/decoding.</li>
+<li><code>frontend</code> — Audio frontend for feature extraction.</li>
+<li><code>cache</code> — State cache dict for streaming inference.</li>
+<li><code>**kwargs</code> — Additional keyword arguments.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def inference(
+ self,
+ data_in,
+ data_lengths=None,
+ key: list = None,
+ tokenizer=None,
+ frontend=None,
+ cache: dict = None,
+ **kwargs,
+ ):
+ &quot;&quot;&quot;Run inference on input data.
+
+ Args:
+ data_in: Input data (audio samples, file paths, or text).
+ data_lengths: Lengths of each input sample in the batch.
+ key: Sample identifiers.
+ tokenizer: Tokenizer instance for text encoding/decoding.
+ frontend: Audio frontend for feature extraction.
+ cache: State cache dict for streaming inference.
+ **kwargs: Additional keyword arguments.
+ &quot;&quot;&quot;
+ if cache is None:
+ cache = {}
+
+ # init beamsearch
+ is_use_ctc = kwargs.get(&quot;decoding_ctc_weight&quot;, 0.0) &gt; 0.00001 and self.ctc != None
+ is_use_lm = (
+ kwargs.get(&quot;lm_weight&quot;, 0.0) &gt; 0.00001 and kwargs.get(&quot;lm_file&quot;, None) is not None
+ )
+
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/scama/model.py#L741" target="_blank" class="src-more">View full source on GitHub →</a></details></div>
+<div class="api-detail" id="e219"><span class="dtag badge-c">class</span><h2>SeacoParaformer</h2><div class="dmod">funasr.models.seaco_paraformer · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/seaco_paraformer/model.py#L50" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><ul>
+<li><code>SeACo</code> — Paraformer: Semantic-Aware Contextual Paraformer.</li>
+</ul>
+<p>The recommended Chinese ASR model. Combines Paraformer&#x27;s non-autoregressive</p>
+<p>architecture with semantic context biasing for hotword recognition.</p>
+<p>Registered as &#x27;paraformer-zh&#x27; alias.</p>
+<ul>
+<li><code>Output</code> — {&quot;key&quot;: str, &quot;text&quot;: str, &quot;timestamp&quot;: [[start_ms, end_ms], ...]}</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code>class SeacoParaformer(BiCifParaformer, Paraformer):
+ &quot;&quot;&quot;SeACo-Paraformer: Semantic-Aware Contextual Paraformer.
+
+ The recommended Chinese ASR model. Combines Paraformer&#x27;s non-autoregressive
+ architecture with semantic context biasing for hotword recognition.
+
+ Registered as &#x27;paraformer-zh&#x27; alias.
+ Output: {&quot;key&quot;: str, &quot;text&quot;: str, &quot;timestamp&quot;: [[start_ms, end_ms], ...]}
+ &quot;&quot;&quot;
+
+ def __init__(
+ self,
+ *args,
+ **kwargs,
+ ):
+ &quot;&quot;&quot;Initialize SeacoParaformer.
+
+ Args:
+ *args: Variable positional arguments.
+ **kwargs: Additional keyword arguments.
+ &quot;&quot;&quot;
+ super().__init__(*args, **kwargs)
+
+ self.inner_dim = kwargs.get(&quot;inner_dim&quot;, 256)
+ self.bias_encoder_type = kwargs.get(&quot;bias_encoder_type&quot;, &quot;lstm&quot;)
+ bias_encoder_dropout_rate = kwargs.get(&quot;bias_encoder_dropout_rate&quot;, 0.0)
+ bias_encoder_bid = kwargs.get(&quot;bias_encoder_bid&quot;, False)
+ seaco_lsm_weight = kwargs.get(&quot;seaco_lsm_weight&quot;, 0.0)
+ seaco_length_normalized_loss = kwargs.get(&quot;seaco_length_normalized_loss&quot;, True)
+
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/seaco_paraformer/model.py#L50" target="_blank" class="src-more">View full source on GitHub →</a></details><h3 class="mt">Methods</h3><div class="mblk"><code>.forward(speech, speech_lengths, text, text_lengths, **kwargs)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/seaco_paraformer/model.py#L122" target="_blank" class="gh-link">L122</a><div class="mdoc"><p>Frontend + Encoder + Decoder + Calc loss</p>
+<h4>Args:</h4>
+<ul>
+<li><code>speech</code> — (Batch, Length, ...)</li>
+<li><code>speech_lengths</code> — (Batch, )</li>
+<li><code>text</code> — (Batch, Length)</li>
+<li><code>text_lengths</code> — (Batch,)</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def forward(
+ self,
+ speech: torch.Tensor,
+ speech_lengths: torch.Tensor,
+ text: torch.Tensor,
+ text_lengths: torch.Tensor,
+ **kwargs,
+ ) -&gt; Tuple[torch.Tensor, Dict[str, torch.Tensor], torch.Tensor]:
+ &quot;&quot;&quot;Frontend + Encoder + Decoder + Calc loss
+
+ Args:
+ speech: (Batch, Length, ...)
+ speech_lengths: (Batch, )
+ text: (Batch, Length)
+ text_lengths: (Batch,)
+ &quot;&quot;&quot;
+ if len(text_lengths.size()) &gt; 1:
+ text_lengths = text_lengths[:, 0]
+ if len(speech_lengths.size()) &gt; 1:
+ speech_lengths = speech_lengths[:, 0]
+ # Check that batch_size is unified
+ assert (
+ speech.shape[0] == speech_lengths.shape[0] == text.shape[0] == text_lengths.shape[0]
+ ), (speech.shape, speech_lengths.shape, text.shape, text_lengths.shape)
+
+ hotword_pad = kwargs.get(&quot;hotword_pad&quot;)
+ hotword_lengths = kwargs.get(&quot;hotword_lengths&quot;)
+ seaco_label_pad = kwargs.get(&quot;seaco_label_pad&quot;)
+ if len(hotword_lengths.size()) &gt; 1:
+ hotword_lengths = hotword_lengths[:, 0]
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/seaco_paraformer/model.py#L122" target="_blank" class="src-more">View full source →</a></details></div><div class="mblk"><code>.calc_predictor(encoder_out, encoder_out_lens)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/seaco_paraformer/model.py#L202" target="_blank" class="gh-link">L202</a><div class="mdoc"><p>Calc predictor.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>encoder_out</code> — Encoder output tensor.</li>
+<li><code>encoder_out_lens</code> — Encoder output lengths.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def calc_predictor(self, encoder_out, encoder_out_lens):
+ &quot;&quot;&quot;Calc predictor.
+
+ Args:
+ encoder_out: Encoder output tensor.
+ encoder_out_lens: Encoder output lengths.
+ &quot;&quot;&quot;
+ encoder_out_mask = (
+ ~make_pad_mask(encoder_out_lens, maxlen=encoder_out.size(1))[:, None, :]
+ ).to(encoder_out.device)
+ predictor_outs = self.predictor(
+ encoder_out, None, encoder_out_mask, ignore_id=self.ignore_id
+ )
+ return predictor_outs[:4]
+</code></pre></details></div><div class="mblk"><code>.inference(data_in, data_lengths, key, tokenizer, frontend, **kwargs)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/seaco_paraformer/model.py#L422" target="_blank" class="gh-link">L422</a><div class="mdoc"><p>Run inference on input data.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>data_in</code> — Input data (audio samples, file paths, or text).</li>
+<li><code>data_lengths</code> — Lengths of each input sample in the batch.</li>
+<li><code>key</code> — Sample identifiers.</li>
+<li><code>tokenizer</code> — Tokenizer instance for text encoding/decoding.</li>
+<li><code>frontend</code> — Audio frontend for feature extraction.</li>
+<li><code>**kwargs</code> — Additional keyword arguments.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def inference(
+ self,
+ data_in,
+ data_lengths=None,
+ key: list = None,
+ tokenizer=None,
+ frontend=None,
+ **kwargs,
+ ):
+
+ # init beamsearch
+ &quot;&quot;&quot;Run inference on input data.
+
+ Args:
+ data_in: Input data (audio samples, file paths, or text).
+ data_lengths: Lengths of each input sample in the batch.
+ key: Sample identifiers.
+ tokenizer: Tokenizer instance for text encoding/decoding.
+ frontend: Audio frontend for feature extraction.
+ **kwargs: Additional keyword arguments.
+ &quot;&quot;&quot;
+ is_use_ctc = kwargs.get(&quot;decoding_ctc_weight&quot;, 0.0) &gt; 0.00001 and self.ctc != None
+ is_use_lm = (
+ kwargs.get(&quot;lm_weight&quot;, 0.0) &gt; 0.00001 and kwargs.get(&quot;lm_file&quot;, None) is not None
+ )
+ if self.beam_search is None and (is_use_lm or is_use_ctc):
+ logging.info(&quot;enable beam_search&quot;)
+ self.init_beam_search(**kwargs)
+ self.nbest = kwargs.get(&quot;nbest&quot;, 1)
+ meta_data = {}
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/seaco_paraformer/model.py#L422" target="_blank" class="src-more">View full source →</a></details></div><div class="mblk"><code>.generate_hotwords_list(hotword_list_or_file, tokenizer, frontend)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/seaco_paraformer/model.py#L583" target="_blank" class="gh-link">L583</a><div class="mdoc"><p>Generate hotwords list.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>hotword_list_or_file</code> — TODO.</li>
+<li><code>tokenizer</code> — Tokenizer instance for text encoding/decoding.</li>
+<li><code>frontend</code> — Audio frontend for feature extraction.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def generate_hotwords_list(self, hotword_list_or_file, tokenizer=None, frontend=None):
+
+ &quot;&quot;&quot;Generate hotwords list.
+
+ Args:
+ hotword_list_or_file: TODO.
+ tokenizer: Tokenizer instance for text encoding/decoding.
+ frontend: Audio frontend for feature extraction.
+ &quot;&quot;&quot;
+ def seg_tokenize(txt, seg_dict):
+ &quot;&quot;&quot;Seg tokenize.
+
+ Args:
+ txt: TODO.
+ seg_dict: TODO.
+ &quot;&quot;&quot;
+ pattern = re.compile(r&quot;^[\u4E00-\u9FA50-9]+$&quot;)
+ out_txt = &quot;&quot;
+ for word in txt:
+ word = word.lower()
+ if word in seg_dict:
+ out_txt += seg_dict[word] + &quot; &quot;
+ else:
+ if pattern.match(word):
+ for char in word:
+ if char in seg_dict:
+ out_txt += seg_dict[char] + &quot; &quot;
+ else:
+ out_txt += &quot;&lt;unk&gt;&quot; + &quot; &quot;
+ else:
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/seaco_paraformer/model.py#L583" target="_blank" class="src-more">View full source →</a></details></div><div class="mblk"><code>.export(**kwargs)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/seaco_paraformer/model.py#L692" target="_blank" class="gh-link">L692</a><div class="mdoc"><p>Export.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>**kwargs</code> — Additional keyword arguments.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def export(
+ self,
+ **kwargs,
+ ):
+ &quot;&quot;&quot;Export.
+
+ Args:
+ **kwargs: Additional keyword arguments.
+ &quot;&quot;&quot;
+ if &quot;max_seq_len&quot; not in kwargs:
+ kwargs[&quot;max_seq_len&quot;] = 512
+ from .export_meta import export_rebuild_model
+
+ models = export_rebuild_model(model=self, **kwargs)
+ return models
+</code></pre></details></div></div>
+<div class="api-detail" id="e220"><span class="dtag badge-m">method</span><h2><code>SeacoParaformer.forward(speech, speech_lengths, text, text_lengths, **kwargs)</code></h2><div class="dmod">funasr.models.seaco_paraformer.SeacoParaformer · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/seaco_paraformer/model.py#L122" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Frontend + Encoder + Decoder + Calc loss</p>
+<h4>Args:</h4>
+<ul>
+<li><code>speech</code> — (Batch, Length, ...)</li>
+<li><code>speech_lengths</code> — (Batch, )</li>
+<li><code>text</code> — (Batch, Length)</li>
+<li><code>text_lengths</code> — (Batch,)</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def forward(
+ self,
+ speech: torch.Tensor,
+ speech_lengths: torch.Tensor,
+ text: torch.Tensor,
+ text_lengths: torch.Tensor,
+ **kwargs,
+ ) -&gt; Tuple[torch.Tensor, Dict[str, torch.Tensor], torch.Tensor]:
+ &quot;&quot;&quot;Frontend + Encoder + Decoder + Calc loss
+
+ Args:
+ speech: (Batch, Length, ...)
+ speech_lengths: (Batch, )
+ text: (Batch, Length)
+ text_lengths: (Batch,)
+ &quot;&quot;&quot;
+ if len(text_lengths.size()) &gt; 1:
+ text_lengths = text_lengths[:, 0]
+ if len(speech_lengths.size()) &gt; 1:
+ speech_lengths = speech_lengths[:, 0]
+ # Check that batch_size is unified
+ assert (
+ speech.shape[0] == speech_lengths.shape[0] == text.shape[0] == text_lengths.shape[0]
+ ), (speech.shape, speech_lengths.shape, text.shape, text_lengths.shape)
+
+ hotword_pad = kwargs.get(&quot;hotword_pad&quot;)
+ hotword_lengths = kwargs.get(&quot;hotword_lengths&quot;)
+ seaco_label_pad = kwargs.get(&quot;seaco_label_pad&quot;)
+ if len(hotword_lengths.size()) &gt; 1:
+ hotword_lengths = hotword_lengths[:, 0]
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/seaco_paraformer/model.py#L122" target="_blank" class="src-more">View full source on GitHub →</a></details></div>
+<div class="api-detail" id="e221"><span class="dtag badge-m">method</span><h2><code>SeacoParaformer.calc_predictor(encoder_out, encoder_out_lens)</code></h2><div class="dmod">funasr.models.seaco_paraformer.SeacoParaformer · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/seaco_paraformer/model.py#L202" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Calc predictor.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>encoder_out</code> — Encoder output tensor.</li>
+<li><code>encoder_out_lens</code> — Encoder output lengths.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def calc_predictor(self, encoder_out, encoder_out_lens):
+ &quot;&quot;&quot;Calc predictor.
+
+ Args:
+ encoder_out: Encoder output tensor.
+ encoder_out_lens: Encoder output lengths.
+ &quot;&quot;&quot;
+ encoder_out_mask = (
+ ~make_pad_mask(encoder_out_lens, maxlen=encoder_out.size(1))[:, None, :]
+ ).to(encoder_out.device)
+ predictor_outs = self.predictor(
+ encoder_out, None, encoder_out_mask, ignore_id=self.ignore_id
+ )
+ return predictor_outs[:4]
+</code></pre></details></div>
+<div class="api-detail" id="e222"><span class="dtag badge-m">method</span><h2><code>SeacoParaformer.inference(data_in, data_lengths, key, tokenizer, frontend, **kwargs)</code></h2><div class="dmod">funasr.models.seaco_paraformer.SeacoParaformer · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/seaco_paraformer/model.py#L422" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Run inference on input data.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>data_in</code> — Input data (audio samples, file paths, or text).</li>
+<li><code>data_lengths</code> — Lengths of each input sample in the batch.</li>
+<li><code>key</code> — Sample identifiers.</li>
+<li><code>tokenizer</code> — Tokenizer instance for text encoding/decoding.</li>
+<li><code>frontend</code> — Audio frontend for feature extraction.</li>
+<li><code>**kwargs</code> — Additional keyword arguments.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def inference(
+ self,
+ data_in,
+ data_lengths=None,
+ key: list = None,
+ tokenizer=None,
+ frontend=None,
+ **kwargs,
+ ):
+
+ # init beamsearch
+ &quot;&quot;&quot;Run inference on input data.
+
+ Args:
+ data_in: Input data (audio samples, file paths, or text).
+ data_lengths: Lengths of each input sample in the batch.
+ key: Sample identifiers.
+ tokenizer: Tokenizer instance for text encoding/decoding.
+ frontend: Audio frontend for feature extraction.
+ **kwargs: Additional keyword arguments.
+ &quot;&quot;&quot;
+ is_use_ctc = kwargs.get(&quot;decoding_ctc_weight&quot;, 0.0) &gt; 0.00001 and self.ctc != None
+ is_use_lm = (
+ kwargs.get(&quot;lm_weight&quot;, 0.0) &gt; 0.00001 and kwargs.get(&quot;lm_file&quot;, None) is not None
+ )
+ if self.beam_search is None and (is_use_lm or is_use_ctc):
+ logging.info(&quot;enable beam_search&quot;)
+ self.init_beam_search(**kwargs)
+ self.nbest = kwargs.get(&quot;nbest&quot;, 1)
+ meta_data = {}
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/seaco_paraformer/model.py#L422" target="_blank" class="src-more">View full source on GitHub →</a></details></div>
+<div class="api-detail" id="e223"><span class="dtag badge-m">method</span><h2><code>SeacoParaformer.generate_hotwords_list(hotword_list_or_file, tokenizer, frontend)</code></h2><div class="dmod">funasr.models.seaco_paraformer.SeacoParaformer · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/seaco_paraformer/model.py#L583" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Generate hotwords list.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>hotword_list_or_file</code> — TODO.</li>
+<li><code>tokenizer</code> — Tokenizer instance for text encoding/decoding.</li>
+<li><code>frontend</code> — Audio frontend for feature extraction.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def generate_hotwords_list(self, hotword_list_or_file, tokenizer=None, frontend=None):
+
+ &quot;&quot;&quot;Generate hotwords list.
+
+ Args:
+ hotword_list_or_file: TODO.
+ tokenizer: Tokenizer instance for text encoding/decoding.
+ frontend: Audio frontend for feature extraction.
+ &quot;&quot;&quot;
+ def seg_tokenize(txt, seg_dict):
+ &quot;&quot;&quot;Seg tokenize.
+
+ Args:
+ txt: TODO.
+ seg_dict: TODO.
+ &quot;&quot;&quot;
+ pattern = re.compile(r&quot;^[\u4E00-\u9FA50-9]+$&quot;)
+ out_txt = &quot;&quot;
+ for word in txt:
+ word = word.lower()
+ if word in seg_dict:
+ out_txt += seg_dict[word] + &quot; &quot;
+ else:
+ if pattern.match(word):
+ for char in word:
+ if char in seg_dict:
+ out_txt += seg_dict[char] + &quot; &quot;
+ else:
+ out_txt += &quot;&lt;unk&gt;&quot; + &quot; &quot;
+ else:
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/seaco_paraformer/model.py#L583" target="_blank" class="src-more">View full source on GitHub →</a></details></div>
+<div class="api-detail" id="e224"><span class="dtag badge-m">method</span><h2><code>SeacoParaformer.export(**kwargs)</code></h2><div class="dmod">funasr.models.seaco_paraformer.SeacoParaformer · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/seaco_paraformer/model.py#L692" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Export.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>**kwargs</code> — Additional keyword arguments.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def export(
+ self,
+ **kwargs,
+ ):
+ &quot;&quot;&quot;Export.
+
+ Args:
+ **kwargs: Additional keyword arguments.
+ &quot;&quot;&quot;
+ if &quot;max_seq_len&quot; not in kwargs:
+ kwargs[&quot;max_seq_len&quot;] = 512
+ from .export_meta import export_rebuild_model
+
+ models = export_rebuild_model(model=self, **kwargs)
+ return models
+</code></pre></details></div>
+<div class="api-detail" id="e225"><span class="dtag badge-f">function</span><h2><code>load_seg_dict(seg_dict_file)</code></h2><div class="dmod">funasr.models.seaco_paraformer · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/seaco_paraformer/model.py#L710" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Load seg dict.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>seg_dict_file</code> — TODO.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code>def load_seg_dict(seg_dict_file):
+ &quot;&quot;&quot;Load seg dict.
+
+ Args:
+ seg_dict_file: TODO.
+ &quot;&quot;&quot;
+ seg_dict = {}
+ assert isinstance(seg_dict_file, str)
+ with open(seg_dict_file, &quot;r&quot;, encoding=&quot;utf8&quot;) as f:
+ lines = f.readlines()
+ for line in lines:
+ s = line.strip().split()
+ key = s[0]
+ value = s[1:]
+ seg_dict[key] = &quot; &quot;.join(value)
+ return seg_dict</code></pre></details></div>
+<div class="api-detail" id="e226"><span class="dtag badge-f">function</span><h2><code>sequence_mask(lengths, maxlen, dtype, device)</code></h2><div class="dmod">funasr.models.sense_voice · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/sense_voice/model.py#L326" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Sequence mask.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>lengths</code> — TODO.</li>
+<li><code>maxlen</code> — TODO.</li>
+<li><code>dtype</code> — TODO.</li>
+<li><code>device</code> — Target device (&quot;cuda:0&quot;, &quot;cpu&quot;, etc.).</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code>def sequence_mask(lengths, maxlen=None, dtype=torch.float32, device=None):
+ &quot;&quot;&quot;Sequence mask.
+
+ Args:
+ lengths: TODO.
+ maxlen: TODO.
+ dtype: TODO.
+ device: Target device (&quot;cuda:0&quot;, &quot;cpu&quot;, etc.).
+ &quot;&quot;&quot;
+ if maxlen is None:
+ maxlen = lengths.max()
+ row_vector = torch.arange(0, maxlen, 1).to(lengths.device)
+ matrix = torch.unsqueeze(lengths, dim=-1)
+ mask = row_vector &lt; matrix
+ mask = mask.detach()
+
+ return mask.type(dtype).to(device) if device is not None else mask.type(dtype)
+</code></pre></details></div>
+<div class="api-detail" id="e227"><span class="dtag badge-c">class</span><h2>SenseVoiceEncoderSmall</h2><div class="dmod">funasr.models.sense_voice · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/sense_voice/model.py#L489" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><ul>
+<li><code>Author</code> — Speech Lab of DAMO Academy, Alibaba Group</li>
+<li><code>SCAMA</code> — Streaming chunk-aware multihead attention for online end-to-end speech recognition</li>
+</ul>
+<p>https://arxiv.org/abs/2006.01713</p>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code>class SenseVoiceEncoderSmall(nn.Module):
+ &quot;&quot;&quot;
+ Author: Speech Lab of DAMO Academy, Alibaba Group
+ SCAMA: Streaming chunk-aware multihead attention for online end-to-end speech recognition
+ https://arxiv.org/abs/2006.01713
+ &quot;&quot;&quot;
+
+ def __init__(
+ self,
+ input_size: int,
+ output_size: int = 256,
+ attention_heads: int = 4,
+ linear_units: int = 2048,
+ num_blocks: int = 6,
+ tp_blocks: int = 0,
+ dropout_rate: float = 0.1,
+ positional_dropout_rate: float = 0.1,
+ attention_dropout_rate: float = 0.0,
+ stochastic_depth_rate: float = 0.0,
+ input_layer: Optional[str] = &quot;conv2d&quot;,
+ pos_enc_class=SinusoidalPositionEncoder,
+ normalize_before: bool = True,
+ concat_after: bool = False,
+ positionwise_layer_type: str = &quot;linear&quot;,
+ positionwise_conv_kernel_size: int = 1,
+ padding_idx: int = -1,
+ kernel_size: int = 11,
+ sanm_shfit: int = 0,
+ selfattention_layer_type: str = &quot;sanm&quot;,
+ **kwargs,
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/sense_voice/model.py#L489" target="_blank" class="src-more">View full source on GitHub →</a></details><h3 class="mt">Methods</h3><div class="mblk"><code>.output_size()</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/sense_voice/model.py#L619" target="_blank" class="gh-link">L619</a><div class="mdoc"><p>Output size.</p>
+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def output_size(self) -&gt; int:
+ &quot;&quot;&quot;Output size.&quot;&quot;&quot;
+ return self._output_size
+</code></pre></details></div><div class="mblk"><code>.forward(xs_pad, ilens)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/sense_voice/model.py#L623" target="_blank" class="gh-link">L623</a><div class="mdoc"><p>Embed positions in tensor.</p>
+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def forward(
+ self,
+ xs_pad: torch.Tensor,
+ ilens: torch.Tensor,
+ ):
+ &quot;&quot;&quot;Embed positions in tensor.&quot;&quot;&quot;
+ maxlen = xs_pad.shape[1]
+ masks = sequence_mask(ilens, maxlen=maxlen, device=ilens.device)[:, None, :]
+
+ xs_pad *= self.output_size() ** 0.5
+
+ xs_pad = self.embed(xs_pad)
+
+ # forward encoder1
+ for layer_idx, encoder_layer in enumerate(self.encoders0):
+ encoder_outs = encoder_layer(xs_pad, masks)
+ xs_pad, masks = encoder_outs[0], encoder_outs[1]
+
+ for layer_idx, encoder_layer in enumerate(self.encoders):
+ encoder_outs = encoder_layer(xs_pad, masks)
+ xs_pad, masks = encoder_outs[0], encoder_outs[1]
+
+ xs_pad = self.after_norm(xs_pad)
+
+ # forward encoder2
+ olens = masks.squeeze(1).sum(1).int()
+
+ for layer_idx, encoder_layer in enumerate(self.tp_encoders):
+ encoder_outs = encoder_layer(xs_pad, masks)
+ xs_pad, masks = encoder_outs[0], encoder_outs[1]
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/sense_voice/model.py#L623" target="_blank" class="src-more">View full source →</a></details></div></div>
+<div class="api-detail" id="e228"><span class="dtag badge-m">method</span><h2><code>SenseVoiceEncoderSmall.output_size()</code></h2><div class="dmod">funasr.models.sense_voice.SenseVoiceEncoderSmall · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/sense_voice/model.py#L619" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Output size.</p>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def output_size(self) -&gt; int:
+ &quot;&quot;&quot;Output size.&quot;&quot;&quot;
+ return self._output_size
+</code></pre></details></div>
+<div class="api-detail" id="e229"><span class="dtag badge-m">method</span><h2><code>SenseVoiceEncoderSmall.forward(xs_pad, ilens)</code></h2><div class="dmod">funasr.models.sense_voice.SenseVoiceEncoderSmall · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/sense_voice/model.py#L623" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Embed positions in tensor.</p>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def forward(
+ self,
+ xs_pad: torch.Tensor,
+ ilens: torch.Tensor,
+ ):
+ &quot;&quot;&quot;Embed positions in tensor.&quot;&quot;&quot;
+ maxlen = xs_pad.shape[1]
+ masks = sequence_mask(ilens, maxlen=maxlen, device=ilens.device)[:, None, :]
+
+ xs_pad *= self.output_size() ** 0.5
+
+ xs_pad = self.embed(xs_pad)
+
+ # forward encoder1
+ for layer_idx, encoder_layer in enumerate(self.encoders0):
+ encoder_outs = encoder_layer(xs_pad, masks)
+ xs_pad, masks = encoder_outs[0], encoder_outs[1]
+
+ for layer_idx, encoder_layer in enumerate(self.encoders):
+ encoder_outs = encoder_layer(xs_pad, masks)
+ xs_pad, masks = encoder_outs[0], encoder_outs[1]
+
+ xs_pad = self.after_norm(xs_pad)
+
+ # forward encoder2
+ olens = masks.squeeze(1).sum(1).int()
+
+ for layer_idx, encoder_layer in enumerate(self.tp_encoders):
+ encoder_outs = encoder_layer(xs_pad, masks)
+ xs_pad, masks = encoder_outs[0], encoder_outs[1]
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/sense_voice/model.py#L623" target="_blank" class="src-more">View full source on GitHub →</a></details></div>
+<div class="api-detail" id="e230"><span class="dtag badge-c">class</span><h2>SenseVoiceSmall</h2><div class="dmod">funasr.models.sense_voice · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/sense_voice/model.py#L659" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><ul>
+<li><code>CTC</code> — attention hybrid Encoder-Decoder model</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code>class SenseVoiceSmall(nn.Module):
+ &quot;&quot;&quot;CTC-attention hybrid Encoder-Decoder model&quot;&quot;&quot;
+
+ def __init__(
+ self,
+ specaug: str = None,
+ specaug_conf: dict = None,
+ normalize: str = None,
+ normalize_conf: dict = None,
+ encoder: str = None,
+ encoder_conf: dict = None,
+ ctc_conf: dict = None,
+ input_size: int = 80,
+ vocab_size: int = -1,
+ ignore_id: int = -1,
+ blank_id: int = 0,
+ sos: int = 1,
+ eos: int = 2,
+ length_normalized_loss: bool = False,
+ **kwargs,
+ ):
+
+ &quot;&quot;&quot;Initialize SenseVoiceSmall.
+
+ Args:
+ specaug: TODO.
+ specaug_conf: Configuration dict for specaug.
+ normalize: TODO.
+ normalize_conf: Configuration dict for normalize.
+ encoder: TODO.
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/sense_voice/model.py#L659" target="_blank" class="src-more">View full source on GitHub →</a></details><h3 class="mt">Methods</h3><div class="mblk"><code>.from_pretrained(model, **kwargs)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/sense_voice/model.py#L754" target="_blank" class="gh-link">L754</a><div class="mdoc"><p>From pretrained.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>model</code> — Model instance or model name.</li>
+<li><code>**kwargs</code> — Additional keyword arguments.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def from_pretrained(model: str = None, **kwargs):
+ &quot;&quot;&quot;From pretrained.
+
+ Args:
+ model: Model instance or model name.
+ **kwargs: Additional keyword arguments.
+ &quot;&quot;&quot;
+ from funasr import AutoModel
+
+ model, kwargs = AutoModel.build_model(model=model, trust_remote_code=True, **kwargs)
+
+ return model, kwargs
+</code></pre></details></div><div class="mblk"><code>.forward(speech, speech_lengths, text, text_lengths, **kwargs)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/sense_voice/model.py#L767" target="_blank" class="gh-link">L767</a><div class="mdoc"><p>Encoder + Decoder + Calc loss</p>
+<h4>Args:</h4>
+<ul>
+<li><code>speech</code> — (Batch, Length, ...)</li>
+<li><code>speech_lengths</code> — (Batch, )</li>
+<li><code>text</code> — (Batch, Length)</li>
+<li><code>text_lengths</code> — (Batch,)</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def forward(
+ self,
+ speech: torch.Tensor,
+ speech_lengths: torch.Tensor,
+ text: torch.Tensor,
+ text_lengths: torch.Tensor,
+ **kwargs,
+ ):
+ &quot;&quot;&quot;Encoder + Decoder + Calc loss
+ Args:
+ speech: (Batch, Length, ...)
+ speech_lengths: (Batch, )
+ text: (Batch, Length)
+ text_lengths: (Batch,)
+ &quot;&quot;&quot;
+ # import pdb;
+ # pdb.set_trace()
+ if len(text_lengths.size()) &gt; 1:
+ text_lengths = text_lengths[:, 0]
+ if len(speech_lengths.size()) &gt; 1:
+ speech_lengths = speech_lengths[:, 0]
+
+ batch_size = speech.shape[0]
+
+ # 1. Encoder
+ encoder_out, encoder_out_lens = self.encode(speech, speech_lengths, text)
+
+ loss_ctc, cer_ctc = None, None
+ loss_rich, acc_rich = None, None
+ stats = dict()
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/sense_voice/model.py#L767" target="_blank" class="src-more">View full source →</a></details></div><div class="mblk"><code>.encode(speech, speech_lengths, text, **kwargs)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/sense_voice/model.py#L817" target="_blank" class="gh-link">L817</a><div class="mdoc"><p>Frontend + Encoder. Note that this method is used by asr_inference.py</p>
+<h4>Args:</h4>
+<ul>
+<li><code>speech</code> — (Batch, Length, ...)</li>
+<li><code>speech_lengths</code> — (Batch, )</li>
+<li><code>ind</code> — int</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def encode(
+ self,
+ speech: torch.Tensor,
+ speech_lengths: torch.Tensor,
+ text: torch.Tensor,
+ **kwargs,
+ ):
+ &quot;&quot;&quot;Frontend + Encoder. Note that this method is used by asr_inference.py
+ Args:
+ speech: (Batch, Length, ...)
+ speech_lengths: (Batch, )
+ ind: int
+ &quot;&quot;&quot;
+
+ # Data augmentation
+ if self.specaug is not None and self.training:
+ speech, speech_lengths = self.specaug(speech, speech_lengths)
+
+ # Normalization for feature: e.g. Global-CMVN, Utterance-CMVN
+ if self.normalize is not None:
+ speech, speech_lengths = self.normalize(speech, speech_lengths)
+
+ lids = torch.LongTensor(
+ [
+ [
+ (
+ self.lid_int_dict[int(lid)]
+ if torch.rand(1) &gt; 0.2 and int(lid) in self.lid_int_dict
+ else 0
+ )
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/sense_voice/model.py#L817" target="_blank" class="src-more">View full source →</a></details></div><div class="mblk"><code>.inference(data_in, data_lengths, key, tokenizer, frontend, **kwargs)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/sense_voice/model.py#L918" target="_blank" class="gh-link">L918</a><div class="mdoc"><p>Run inference on input data.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>data_in</code> — Input data (audio samples, file paths, or text).</li>
+<li><code>data_lengths</code> — Lengths of each input sample in the batch.</li>
+<li><code>key</code> — Sample identifiers.</li>
+<li><code>tokenizer</code> — Tokenizer instance for text encoding/decoding.</li>
+<li><code>frontend</code> — Audio frontend for feature extraction.</li>
+<li><code>**kwargs</code> — Additional keyword arguments.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def inference(
+ self,
+ data_in,
+ data_lengths=None,
+ key: list = [&quot;wav_file_tmp_name&quot;],
+ tokenizer=None,
+ frontend=None,
+ **kwargs,
+ ):
+
+ &quot;&quot;&quot;Run inference on input data.
+
+ Args:
+ data_in: Input data (audio samples, file paths, or text).
+ data_lengths: Lengths of each input sample in the batch.
+ key: Sample identifiers.
+ tokenizer: Tokenizer instance for text encoding/decoding.
+ frontend: Audio frontend for feature extraction.
+ **kwargs: Additional keyword arguments.
+ &quot;&quot;&quot;
+ meta_data = {}
+ if (
+ isinstance(data_in, torch.Tensor) and kwargs.get(&quot;data_type&quot;, &quot;sound&quot;) == &quot;fbank&quot;
+ ): # fbank
+ speech, speech_lengths = data_in, data_lengths
+ if len(speech.shape) &lt; 3:
+ speech = speech[None, :, :]
+ if speech_lengths is None:
+ speech_lengths = speech.shape[1]
+ else:
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/sense_voice/model.py#L918" target="_blank" class="src-more">View full source →</a></details></div><div class="mblk"><code>.post(timestamp)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/sense_voice/model.py#L1082" target="_blank" class="gh-link">L1082</a><div class="mdoc"><p>Post.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>timestamp</code> — TODO.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def post(self, timestamp):
+ &quot;&quot;&quot;Post.
+
+ Args:
+ timestamp: TODO.
+ &quot;&quot;&quot;
+ timestamp_new = []
+ words_new = []
+ prev_word = None
+ for i, t in enumerate(timestamp):
+ word, start, end = t
+ start = int(start * 1000)
+ end = int(end * 1000)
+ if word == &quot;▁&quot;:
+ continue
+ if i == 0:
+ # timestamp_new.append([word, start, end])
+ timestamp_new.append([start, end])
+ words_new.append(word)
+ elif word.startswith(&quot;▁&quot;):
+ word = word[1:]
+ timestamp_new.append([start, end])
+ words_new.append(word)
+ elif prev_word is not None and prev_word.isalpha() and prev_word.isascii() and word.isalpha() and word.isascii():
+ word = prev_word + word
+ timestamp_new[-1][1] = end
+ words_new[-1] = word
+ else:
+ # timestamp_new[-1][0] += word
+ timestamp_new.append([start, end])
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/sense_voice/model.py#L1082" target="_blank" class="src-more">View full source →</a></details></div><div class="mblk"><code>.export(**kwargs)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/sense_voice/model.py#L1116" target="_blank" class="gh-link">L1116</a><div class="mdoc"><p>Export.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>**kwargs</code> — Additional keyword arguments.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def export(self, **kwargs):
+ &quot;&quot;&quot;Export.
+
+ Args:
+ **kwargs: Additional keyword arguments.
+ &quot;&quot;&quot;
+ from .export_meta import export_rebuild_model
+
+ if &quot;max_seq_len&quot; not in kwargs:
+ kwargs[&quot;max_seq_len&quot;] = 512
+ models = export_rebuild_model(model=self, **kwargs)
+ return models
+
+ return results, meta_data
+</code></pre></details></div></div>
+<div class="api-detail" id="e231"><span class="dtag badge-m">method</span><h2><code>SenseVoiceSmall.from_pretrained(model, **kwargs)</code></h2><div class="dmod">funasr.models.sense_voice.SenseVoiceSmall · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/sense_voice/model.py#L754" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>From pretrained.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>model</code> — Model instance or model name.</li>
+<li><code>**kwargs</code> — Additional keyword arguments.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def from_pretrained(model: str = None, **kwargs):
+ &quot;&quot;&quot;From pretrained.
+
+ Args:
+ model: Model instance or model name.
+ **kwargs: Additional keyword arguments.
+ &quot;&quot;&quot;
+ from funasr import AutoModel
+
+ model, kwargs = AutoModel.build_model(model=model, trust_remote_code=True, **kwargs)
+
+ return model, kwargs
+</code></pre></details></div>
+<div class="api-detail" id="e232"><span class="dtag badge-m">method</span><h2><code>SenseVoiceSmall.forward(speech, speech_lengths, text, text_lengths, **kwargs)</code></h2><div class="dmod">funasr.models.sense_voice.SenseVoiceSmall · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/sense_voice/model.py#L767" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Encoder + Decoder + Calc loss</p>
+<h4>Args:</h4>
+<ul>
+<li><code>speech</code> — (Batch, Length, ...)</li>
+<li><code>speech_lengths</code> — (Batch, )</li>
+<li><code>text</code> — (Batch, Length)</li>
+<li><code>text_lengths</code> — (Batch,)</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def forward(
+ self,
+ speech: torch.Tensor,
+ speech_lengths: torch.Tensor,
+ text: torch.Tensor,
+ text_lengths: torch.Tensor,
+ **kwargs,
+ ):
+ &quot;&quot;&quot;Encoder + Decoder + Calc loss
+ Args:
+ speech: (Batch, Length, ...)
+ speech_lengths: (Batch, )
+ text: (Batch, Length)
+ text_lengths: (Batch,)
+ &quot;&quot;&quot;
+ # import pdb;
+ # pdb.set_trace()
+ if len(text_lengths.size()) &gt; 1:
+ text_lengths = text_lengths[:, 0]
+ if len(speech_lengths.size()) &gt; 1:
+ speech_lengths = speech_lengths[:, 0]
+
+ batch_size = speech.shape[0]
+
+ # 1. Encoder
+ encoder_out, encoder_out_lens = self.encode(speech, speech_lengths, text)
+
+ loss_ctc, cer_ctc = None, None
+ loss_rich, acc_rich = None, None
+ stats = dict()
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/sense_voice/model.py#L767" target="_blank" class="src-more">View full source on GitHub →</a></details></div>
+<div class="api-detail" id="e233"><span class="dtag badge-m">method</span><h2><code>SenseVoiceSmall.encode(speech, speech_lengths, text, **kwargs)</code></h2><div class="dmod">funasr.models.sense_voice.SenseVoiceSmall · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/sense_voice/model.py#L817" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Frontend + Encoder. Note that this method is used by asr_inference.py</p>
+<h4>Args:</h4>
+<ul>
+<li><code>speech</code> — (Batch, Length, ...)</li>
+<li><code>speech_lengths</code> — (Batch, )</li>
+<li><code>ind</code> — int</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def encode(
+ self,
+ speech: torch.Tensor,
+ speech_lengths: torch.Tensor,
+ text: torch.Tensor,
+ **kwargs,
+ ):
+ &quot;&quot;&quot;Frontend + Encoder. Note that this method is used by asr_inference.py
+ Args:
+ speech: (Batch, Length, ...)
+ speech_lengths: (Batch, )
+ ind: int
+ &quot;&quot;&quot;
+
+ # Data augmentation
+ if self.specaug is not None and self.training:
+ speech, speech_lengths = self.specaug(speech, speech_lengths)
+
+ # Normalization for feature: e.g. Global-CMVN, Utterance-CMVN
+ if self.normalize is not None:
+ speech, speech_lengths = self.normalize(speech, speech_lengths)
+
+ lids = torch.LongTensor(
+ [
+ [
+ (
+ self.lid_int_dict[int(lid)]
+ if torch.rand(1) &gt; 0.2 and int(lid) in self.lid_int_dict
+ else 0
+ )
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/sense_voice/model.py#L817" target="_blank" class="src-more">View full source on GitHub →</a></details></div>
+<div class="api-detail" id="e234"><span class="dtag badge-m">method</span><h2><code>SenseVoiceSmall.inference(data_in, data_lengths, key, tokenizer, frontend, **kwargs)</code></h2><div class="dmod">funasr.models.sense_voice.SenseVoiceSmall · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/sense_voice/model.py#L918" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Run inference on input data.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>data_in</code> — Input data (audio samples, file paths, or text).</li>
+<li><code>data_lengths</code> — Lengths of each input sample in the batch.</li>
+<li><code>key</code> — Sample identifiers.</li>
+<li><code>tokenizer</code> — Tokenizer instance for text encoding/decoding.</li>
+<li><code>frontend</code> — Audio frontend for feature extraction.</li>
+<li><code>**kwargs</code> — Additional keyword arguments.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def inference(
+ self,
+ data_in,
+ data_lengths=None,
+ key: list = [&quot;wav_file_tmp_name&quot;],
+ tokenizer=None,
+ frontend=None,
+ **kwargs,
+ ):
+
+ &quot;&quot;&quot;Run inference on input data.
+
+ Args:
+ data_in: Input data (audio samples, file paths, or text).
+ data_lengths: Lengths of each input sample in the batch.
+ key: Sample identifiers.
+ tokenizer: Tokenizer instance for text encoding/decoding.
+ frontend: Audio frontend for feature extraction.
+ **kwargs: Additional keyword arguments.
+ &quot;&quot;&quot;
+ meta_data = {}
+ if (
+ isinstance(data_in, torch.Tensor) and kwargs.get(&quot;data_type&quot;, &quot;sound&quot;) == &quot;fbank&quot;
+ ): # fbank
+ speech, speech_lengths = data_in, data_lengths
+ if len(speech.shape) &lt; 3:
+ speech = speech[None, :, :]
+ if speech_lengths is None:
+ speech_lengths = speech.shape[1]
+ else:
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/sense_voice/model.py#L918" target="_blank" class="src-more">View full source on GitHub →</a></details></div>
+<div class="api-detail" id="e235"><span class="dtag badge-m">method</span><h2><code>SenseVoiceSmall.post(timestamp)</code></h2><div class="dmod">funasr.models.sense_voice.SenseVoiceSmall · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/sense_voice/model.py#L1082" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Post.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>timestamp</code> — TODO.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def post(self, timestamp):
+ &quot;&quot;&quot;Post.
+
+ Args:
+ timestamp: TODO.
+ &quot;&quot;&quot;
+ timestamp_new = []
+ words_new = []
+ prev_word = None
+ for i, t in enumerate(timestamp):
+ word, start, end = t
+ start = int(start * 1000)
+ end = int(end * 1000)
+ if word == &quot;▁&quot;:
+ continue
+ if i == 0:
+ # timestamp_new.append([word, start, end])
+ timestamp_new.append([start, end])
+ words_new.append(word)
+ elif word.startswith(&quot;▁&quot;):
+ word = word[1:]
+ timestamp_new.append([start, end])
+ words_new.append(word)
+ elif prev_word is not None and prev_word.isalpha() and prev_word.isascii() and word.isalpha() and word.isascii():
+ word = prev_word + word
+ timestamp_new[-1][1] = end
+ words_new[-1] = word
+ else:
+ # timestamp_new[-1][0] += word
+ timestamp_new.append([start, end])
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/sense_voice/model.py#L1082" target="_blank" class="src-more">View full source on GitHub →</a></details></div>
+<div class="api-detail" id="e236"><span class="dtag badge-m">method</span><h2><code>SenseVoiceSmall.export(**kwargs)</code></h2><div class="dmod">funasr.models.sense_voice.SenseVoiceSmall · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/sense_voice/model.py#L1116" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Export.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>**kwargs</code> — Additional keyword arguments.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def export(self, **kwargs):
+ &quot;&quot;&quot;Export.
+
+ Args:
+ **kwargs: Additional keyword arguments.
+ &quot;&quot;&quot;
+ from .export_meta import export_rebuild_model
+
+ if &quot;max_seq_len&quot; not in kwargs:
+ kwargs[&quot;max_seq_len&quot;] = 512
+ models = export_rebuild_model(model=self, **kwargs)
+ return models
+
+ return results, meta_data
+</code></pre></details></div>
+<div class="api-detail" id="e237"><span class="dtag badge-c">class</span><h2>Transducer</h2><div class="dmod">funasr.models.transducer · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/transducer/model.py#L40" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><ul>
+<li><code>Transducer (RNN-T)</code> — Streaming ASR using encoder-predictor-joint architecture.</li>
+</ul>
+<p>Combines encoder (audio frames), predictor (text history), and joint network.</p>
+<p>Supports beam search decoding. Suitable for low-latency streaming applications.</p>
+<ul>
+<li><code>Output</code> — {&quot;key&quot;: str, &quot;text&quot;: str}</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code>class Transducer(torch.nn.Module):
+ &quot;&quot;&quot;Transducer (RNN-T): Streaming ASR using encoder-predictor-joint architecture.
+
+ Combines encoder (audio frames), predictor (text history), and joint network.
+ Supports beam search decoding. Suitable for low-latency streaming applications.
+
+ Output: {&quot;key&quot;: str, &quot;text&quot;: str}
+ &quot;&quot;&quot;
+
+ def __init__(
+ self,
+ frontend: Optional[str] = None,
+ frontend_conf: Optional[Dict] = None,
+ specaug: Optional[str] = None,
+ specaug_conf: Optional[Dict] = None,
+ normalize: str = None,
+ normalize_conf: Optional[Dict] = None,
+ encoder: str = None,
+ encoder_conf: Optional[Dict] = None,
+ decoder: str = None,
+ decoder_conf: Optional[Dict] = None,
+ joint_network: str = None,
+ joint_network_conf: Optional[Dict] = None,
+ transducer_weight: float = 1.0,
+ fastemit_lambda: float = 0.0,
+ auxiliary_ctc_weight: float = 0.0,
+ auxiliary_ctc_dropout_rate: float = 0.0,
+ auxiliary_lm_loss_weight: float = 0.0,
+ auxiliary_lm_loss_smoothing: float = 0.0,
+ input_size: int = 80,
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/transducer/model.py#L40" target="_blank" class="src-more">View full source on GitHub →</a></details><h3 class="mt">Methods</h3><div class="mblk"><code>.forward(speech, speech_lengths, text, text_lengths, **kwargs)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/transducer/model.py#L190" target="_blank" class="gh-link">L190</a><div class="mdoc"><p>Encoder + Decoder + Calc loss</p>
+<h4>Args:</h4>
+<ul>
+<li><code>speech</code> — (Batch, Length, ...)</li>
+<li><code>speech_lengths</code> — (Batch, )</li>
+<li><code>text</code> — (Batch, Length)</li>
+<li><code>text_lengths</code> — (Batch,)</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def forward(
+ self,
+ speech: torch.Tensor,
+ speech_lengths: torch.Tensor,
+ text: torch.Tensor,
+ text_lengths: torch.Tensor,
+ **kwargs,
+ ) -&gt; Tuple[torch.Tensor, Dict[str, torch.Tensor], torch.Tensor]:
+ &quot;&quot;&quot;Encoder + Decoder + Calc loss
+ Args:
+ speech: (Batch, Length, ...)
+ speech_lengths: (Batch, )
+ text: (Batch, Length)
+ text_lengths: (Batch,)
+ &quot;&quot;&quot;
+ if len(text_lengths.size()) &gt; 1:
+ text_lengths = text_lengths[:, 0]
+ if len(speech_lengths.size()) &gt; 1:
+ speech_lengths = speech_lengths[:, 0]
+
+ batch_size = speech.shape[0]
+ # 1. Encoder
+ encoder_out, encoder_out_lens = self.encode(speech, speech_lengths)
+ if (
+ hasattr(self.encoder, &quot;overlap_chunk_cls&quot;)
+ and self.encoder.overlap_chunk_cls is not None
+ ):
+ encoder_out, encoder_out_lens = self.encoder.overlap_chunk_cls.remove_chunk(
+ encoder_out, encoder_out_lens, chunk_outs=None
+ )
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/transducer/model.py#L190" target="_blank" class="src-more">View full source →</a></details></div><div class="mblk"><code>.encode(speech, speech_lengths, **kwargs)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/transducer/model.py#L276" target="_blank" class="gh-link">L276</a><div class="mdoc"><p>Frontend + Encoder. Note that this method is used by asr_inference.py</p>
+<h4>Args:</h4>
+<ul>
+<li><code>speech</code> — (Batch, Length, ...)</li>
+<li><code>speech_lengths</code> — (Batch, )</li>
+<li><code>ind</code> — int</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def encode(
+ self,
+ speech: torch.Tensor,
+ speech_lengths: torch.Tensor,
+ **kwargs,
+ ) -&gt; Tuple[torch.Tensor, torch.Tensor]:
+ &quot;&quot;&quot;Frontend + Encoder. Note that this method is used by asr_inference.py
+ Args:
+ speech: (Batch, Length, ...)
+ speech_lengths: (Batch, )
+ ind: int
+ &quot;&quot;&quot;
+ with autocast(False):
+
+ # Data augmentation
+ if self.specaug is not None and self.training:
+ speech, speech_lengths = self.specaug(speech, speech_lengths)
+
+ # Normalization for feature: e.g. Global-CMVN, Utterance-CMVN
+ if self.normalize is not None:
+ speech, speech_lengths = self.normalize(speech, speech_lengths)
+
+ # Forward encoder
+ # feats: (Batch, Length, Dim)
+ # -&gt; encoder_out: (Batch, Length2, Dim2)
+ encoder_out, encoder_out_lens, _ = self.encoder(speech, speech_lengths)
+ intermediate_outs = None
+ if isinstance(encoder_out, tuple):
+ intermediate_outs = encoder_out[1]
+ encoder_out = encoder_out[0]
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/transducer/model.py#L276" target="_blank" class="src-more">View full source →</a></details></div><div class="mblk"><code>.init_beam_search(**kwargs)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/transducer/model.py#L455" target="_blank" class="gh-link">L455</a><div class="mdoc"><p>Init beam search.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>**kwargs</code> — Additional keyword arguments.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def init_beam_search(
+ self,
+ **kwargs,
+ ):
+
+ # 1. Build ASR model
+ &quot;&quot;&quot;Init beam search.
+
+ Args:
+ **kwargs: Additional keyword arguments.
+ &quot;&quot;&quot;
+ scorers = {}
+
+ if self.ctc != None:
+ ctc = CTCPrefixScorer(ctc=self.ctc, eos=self.eos)
+ scorers.update(ctc=ctc)
+ token_list = kwargs.get(&quot;token_list&quot;)
+ scorers.update(
+ length_bonus=LengthBonus(len(token_list)),
+ )
+
+ # 3. Build ngram model
+ # ngram is not supported now
+ ngram = None
+ scorers[&quot;ngram&quot;] = ngram
+
+ beam_search = BeamSearchTransducer(
+ self.decoder,
+ self.joint_network,
+ kwargs.get(&quot;beam_size&quot;, 2),
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/transducer/model.py#L455" target="_blank" class="src-more">View full source →</a></details></div><div class="mblk"><code>.inference(data_in, data_lengths, key, tokenizer, **kwargs)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/transducer/model.py#L493" target="_blank" class="gh-link">L493</a><div class="mdoc"><p>Run inference on input data.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>data_in</code> — Input data (audio samples, file paths, or text).</li>
+<li><code>data_lengths</code> — Lengths of each input sample in the batch.</li>
+<li><code>key</code> — Sample identifiers.</li>
+<li><code>tokenizer</code> — Tokenizer instance for text encoding/decoding.</li>
+<li><code>**kwargs</code> — Additional keyword arguments.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def inference(
+ self,
+ data_in: list,
+ data_lengths: list = None,
+ key: list = None,
+ tokenizer=None,
+ **kwargs,
+ ):
+
+ &quot;&quot;&quot;Run inference on input data.
+
+ Args:
+ data_in: Input data (audio samples, file paths, or text).
+ data_lengths: Lengths of each input sample in the batch.
+ key: Sample identifiers.
+ tokenizer: Tokenizer instance for text encoding/decoding.
+ **kwargs: Additional keyword arguments.
+ &quot;&quot;&quot;
+ if kwargs.get(&quot;batch_size&quot;, 1) &gt; 1:
+ raise NotImplementedError(&quot;batch decoding is not implemented&quot;)
+
+ # init beamsearch
+ is_use_ctc = kwargs.get(&quot;decoding_ctc_weight&quot;, 0.0) &gt; 0.00001 and self.ctc != None
+ is_use_lm = (
+ kwargs.get(&quot;lm_weight&quot;, 0.0) &gt; 0.00001 and kwargs.get(&quot;lm_file&quot;, None) is not None
+ )
+ # if self.beam_search is None and (is_use_lm or is_use_ctc):
+ logging.info(&quot;enable beam_search&quot;)
+ self.init_beam_search(**kwargs)
+ self.nbest = kwargs.get(&quot;nbest&quot;, 1)
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/transducer/model.py#L493" target="_blank" class="src-more">View full source →</a></details></div></div>
+<div class="api-detail" id="e238"><span class="dtag badge-m">method</span><h2><code>Transducer.forward(speech, speech_lengths, text, text_lengths, **kwargs)</code></h2><div class="dmod">funasr.models.transducer.Transducer · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/transducer/model.py#L190" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Encoder + Decoder + Calc loss</p>
+<h4>Args:</h4>
+<ul>
+<li><code>speech</code> — (Batch, Length, ...)</li>
+<li><code>speech_lengths</code> — (Batch, )</li>
+<li><code>text</code> — (Batch, Length)</li>
+<li><code>text_lengths</code> — (Batch,)</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def forward(
+ self,
+ speech: torch.Tensor,
+ speech_lengths: torch.Tensor,
+ text: torch.Tensor,
+ text_lengths: torch.Tensor,
+ **kwargs,
+ ) -&gt; Tuple[torch.Tensor, Dict[str, torch.Tensor], torch.Tensor]:
+ &quot;&quot;&quot;Encoder + Decoder + Calc loss
+ Args:
+ speech: (Batch, Length, ...)
+ speech_lengths: (Batch, )
+ text: (Batch, Length)
+ text_lengths: (Batch,)
+ &quot;&quot;&quot;
+ if len(text_lengths.size()) &gt; 1:
+ text_lengths = text_lengths[:, 0]
+ if len(speech_lengths.size()) &gt; 1:
+ speech_lengths = speech_lengths[:, 0]
+
+ batch_size = speech.shape[0]
+ # 1. Encoder
+ encoder_out, encoder_out_lens = self.encode(speech, speech_lengths)
+ if (
+ hasattr(self.encoder, &quot;overlap_chunk_cls&quot;)
+ and self.encoder.overlap_chunk_cls is not None
+ ):
+ encoder_out, encoder_out_lens = self.encoder.overlap_chunk_cls.remove_chunk(
+ encoder_out, encoder_out_lens, chunk_outs=None
+ )
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/transducer/model.py#L190" target="_blank" class="src-more">View full source on GitHub →</a></details></div>
+<div class="api-detail" id="e239"><span class="dtag badge-m">method</span><h2><code>Transducer.encode(speech, speech_lengths, **kwargs)</code></h2><div class="dmod">funasr.models.transducer.Transducer · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/transducer/model.py#L276" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Frontend + Encoder. Note that this method is used by asr_inference.py</p>
+<h4>Args:</h4>
+<ul>
+<li><code>speech</code> — (Batch, Length, ...)</li>
+<li><code>speech_lengths</code> — (Batch, )</li>
+<li><code>ind</code> — int</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def encode(
+ self,
+ speech: torch.Tensor,
+ speech_lengths: torch.Tensor,
+ **kwargs,
+ ) -&gt; Tuple[torch.Tensor, torch.Tensor]:
+ &quot;&quot;&quot;Frontend + Encoder. Note that this method is used by asr_inference.py
+ Args:
+ speech: (Batch, Length, ...)
+ speech_lengths: (Batch, )
+ ind: int
+ &quot;&quot;&quot;
+ with autocast(False):
+
+ # Data augmentation
+ if self.specaug is not None and self.training:
+ speech, speech_lengths = self.specaug(speech, speech_lengths)
+
+ # Normalization for feature: e.g. Global-CMVN, Utterance-CMVN
+ if self.normalize is not None:
+ speech, speech_lengths = self.normalize(speech, speech_lengths)
+
+ # Forward encoder
+ # feats: (Batch, Length, Dim)
+ # -&gt; encoder_out: (Batch, Length2, Dim2)
+ encoder_out, encoder_out_lens, _ = self.encoder(speech, speech_lengths)
+ intermediate_outs = None
+ if isinstance(encoder_out, tuple):
+ intermediate_outs = encoder_out[1]
+ encoder_out = encoder_out[0]
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/transducer/model.py#L276" target="_blank" class="src-more">View full source on GitHub →</a></details></div>
+<div class="api-detail" id="e240"><span class="dtag badge-m">method</span><h2><code>Transducer.init_beam_search(**kwargs)</code></h2><div class="dmod">funasr.models.transducer.Transducer · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/transducer/model.py#L455" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Init beam search.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>**kwargs</code> — Additional keyword arguments.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def init_beam_search(
+ self,
+ **kwargs,
+ ):
+
+ # 1. Build ASR model
+ &quot;&quot;&quot;Init beam search.
+
+ Args:
+ **kwargs: Additional keyword arguments.
+ &quot;&quot;&quot;
+ scorers = {}
+
+ if self.ctc != None:
+ ctc = CTCPrefixScorer(ctc=self.ctc, eos=self.eos)
+ scorers.update(ctc=ctc)
+ token_list = kwargs.get(&quot;token_list&quot;)
+ scorers.update(
+ length_bonus=LengthBonus(len(token_list)),
+ )
+
+ # 3. Build ngram model
+ # ngram is not supported now
+ ngram = None
+ scorers[&quot;ngram&quot;] = ngram
+
+ beam_search = BeamSearchTransducer(
+ self.decoder,
+ self.joint_network,
+ kwargs.get(&quot;beam_size&quot;, 2),
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/transducer/model.py#L455" target="_blank" class="src-more">View full source on GitHub →</a></details></div>
+<div class="api-detail" id="e241"><span class="dtag badge-m">method</span><h2><code>Transducer.inference(data_in, data_lengths, key, tokenizer, **kwargs)</code></h2><div class="dmod">funasr.models.transducer.Transducer · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/transducer/model.py#L493" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Run inference on input data.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>data_in</code> — Input data (audio samples, file paths, or text).</li>
+<li><code>data_lengths</code> — Lengths of each input sample in the batch.</li>
+<li><code>key</code> — Sample identifiers.</li>
+<li><code>tokenizer</code> — Tokenizer instance for text encoding/decoding.</li>
+<li><code>**kwargs</code> — Additional keyword arguments.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def inference(
+ self,
+ data_in: list,
+ data_lengths: list = None,
+ key: list = None,
+ tokenizer=None,
+ **kwargs,
+ ):
+
+ &quot;&quot;&quot;Run inference on input data.
+
+ Args:
+ data_in: Input data (audio samples, file paths, or text).
+ data_lengths: Lengths of each input sample in the batch.
+ key: Sample identifiers.
+ tokenizer: Tokenizer instance for text encoding/decoding.
+ **kwargs: Additional keyword arguments.
+ &quot;&quot;&quot;
+ if kwargs.get(&quot;batch_size&quot;, 1) &gt; 1:
+ raise NotImplementedError(&quot;batch decoding is not implemented&quot;)
+
+ # init beamsearch
+ is_use_ctc = kwargs.get(&quot;decoding_ctc_weight&quot;, 0.0) &gt; 0.00001 and self.ctc != None
+ is_use_lm = (
+ kwargs.get(&quot;lm_weight&quot;, 0.0) &gt; 0.00001 and kwargs.get(&quot;lm_file&quot;, None) is not None
+ )
+ # if self.beam_search is None and (is_use_lm or is_use_ctc):
+ logging.info(&quot;enable beam_search&quot;)
+ self.init_beam_search(**kwargs)
+ self.nbest = kwargs.get(&quot;nbest&quot;, 1)
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/transducer/model.py#L493" target="_blank" class="src-more">View full source on GitHub →</a></details></div>
+<div class="api-detail" id="e242"><span class="dtag badge-c">class</span><h2>Transformer</h2><div class="dmod">funasr.models.transformer · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/transformer/model.py#L23" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><ul>
+<li><code>Transformer</code> — Base encoder-decoder ASR model.</li>
+</ul>
+<p>Standard CTC-attention hybrid architecture with:</p>
+<ul>
+<li>Encoder (self-attention + position encoding)</li>
+<li>CTC branch for auxiliary loss</li>
+<li>Attention decoder for sequence generation</li>
+<li>Beam search with LM fusion</li>
+</ul>
+<p>Base class for Conformer, Branchformer, etc.</p>
+<ul>
+<li><code>Output</code> — {&quot;key&quot;: str, &quot;text&quot;: str}</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code>class Transformer(nn.Module):
+ &quot;&quot;&quot;Transformer: Base encoder-decoder ASR model.
+
+ Standard CTC-attention hybrid architecture with:
+ - Encoder (self-attention + position encoding)
+ - CTC branch for auxiliary loss
+ - Attention decoder for sequence generation
+ - Beam search with LM fusion
+
+ Base class for Conformer, Branchformer, etc.
+ Output: {&quot;key&quot;: str, &quot;text&quot;: str}
+ &quot;&quot;&quot;
+
+ def __init__(
+ self,
+ specaug: str = None,
+ specaug_conf: dict = None,
+ normalize: str = None,
+ normalize_conf: dict = None,
+ encoder: str = None,
+ encoder_conf: dict = None,
+ decoder: str = None,
+ decoder_conf: dict = None,
+ ctc: str = None,
+ ctc_conf: dict = None,
+ ctc_weight: float = 0.5,
+ interctc_weight: float = 0.0,
+ input_size: int = 80,
+ vocab_size: int = -1,
+ ignore_id: int = -1,
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/transformer/model.py#L23" target="_blank" class="src-more">View full source on GitHub →</a></details><h3 class="mt">Methods</h3><div class="mblk"><code>.forward(speech, speech_lengths, text, text_lengths, **kwargs)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/transformer/model.py#L173" target="_blank" class="gh-link">L173</a><div class="mdoc"><p>Encoder + Decoder + Calc loss</p>
+<h4>Args:</h4>
+<ul>
+<li><code>speech</code> — (Batch, Length, ...)</li>
+<li><code>speech_lengths</code> — (Batch, )</li>
+<li><code>text</code> — (Batch, Length)</li>
+<li><code>text_lengths</code> — (Batch,)</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def forward(
+ self,
+ speech: torch.Tensor,
+ speech_lengths: torch.Tensor,
+ text: torch.Tensor,
+ text_lengths: torch.Tensor,
+ **kwargs,
+ ) -&gt; Tuple[torch.Tensor, Dict[str, torch.Tensor], torch.Tensor]:
+ &quot;&quot;&quot;Encoder + Decoder + Calc loss
+ Args:
+ speech: (Batch, Length, ...)
+ speech_lengths: (Batch, )
+ text: (Batch, Length)
+ text_lengths: (Batch,)
+ &quot;&quot;&quot;
+ if len(text_lengths.size()) &gt; 1:
+ text_lengths = text_lengths[:, 0]
+ if len(speech_lengths.size()) &gt; 1:
+ speech_lengths = speech_lengths[:, 0]
+
+ batch_size = speech.shape[0]
+
+ # 1. Encoder
+ encoder_out, encoder_out_lens = self.encode(speech, speech_lengths)
+ intermediate_outs = None
+ if isinstance(encoder_out, tuple):
+ intermediate_outs = encoder_out[1]
+ encoder_out = encoder_out[0]
+
+ loss_att, acc_att, cer_att, wer_att = None, None, None, None
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/transformer/model.py#L173" target="_blank" class="src-more">View full source →</a></details></div><div class="mblk"><code>.encode(speech, speech_lengths, **kwargs)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/transformer/model.py#L266" target="_blank" class="gh-link">L266</a><div class="mdoc"><p>Frontend + Encoder. Note that this method is used by asr_inference.py</p>
+<h4>Args:</h4>
+<ul>
+<li><code>speech</code> — (Batch, Length, ...)</li>
+<li><code>speech_lengths</code> — (Batch, )</li>
+<li><code>ind</code> — int</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def encode(
+ self,
+ speech: torch.Tensor,
+ speech_lengths: torch.Tensor,
+ **kwargs,
+ ) -&gt; Tuple[torch.Tensor, torch.Tensor]:
+ &quot;&quot;&quot;Frontend + Encoder. Note that this method is used by asr_inference.py
+ Args:
+ speech: (Batch, Length, ...)
+ speech_lengths: (Batch, )
+ ind: int
+ &quot;&quot;&quot;
+ with autocast(False):
+
+ # Data augmentation
+ if self.specaug is not None and self.training:
+ speech, speech_lengths = self.specaug(speech, speech_lengths)
+
+ # Normalization for feature: e.g. Global-CMVN, Utterance-CMVN
+ if self.normalize is not None:
+ speech, speech_lengths = self.normalize(speech, speech_lengths)
+
+ # Forward encoder
+ # feats: (Batch, Length, Dim)
+ # -&gt; encoder_out: (Batch, Length2, Dim2)
+ if self.encoder.interctc_use_conditioning:
+ encoder_out, encoder_out_lens, _ = self.encoder(speech, speech_lengths, ctc=self.ctc)
+ else:
+ encoder_out, encoder_out_lens, _ = self.encoder(speech, speech_lengths)
+ intermediate_outs = None
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/transformer/model.py#L266" target="_blank" class="src-more">View full source →</a></details></div><div class="mblk"><code>.init_beam_search(**kwargs)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/transformer/model.py#L368" target="_blank" class="gh-link">L368</a><div class="mdoc"><p>Init beam search.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>**kwargs</code> — Additional keyword arguments.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def init_beam_search(
+ self,
+ **kwargs,
+ ):
+ &quot;&quot;&quot;Init beam search.
+
+ Args:
+ **kwargs: Additional keyword arguments.
+ &quot;&quot;&quot;
+ from funasr.models.transformer.search import BeamSearch
+ from funasr.models.transformer.scorers.ctc import CTCPrefixScorer
+ from funasr.models.transformer.scorers.length_bonus import LengthBonus
+
+ # 1. Build ASR model
+ scorers = {}
+
+ if self.ctc != None:
+ ctc = CTCPrefixScorer(ctc=self.ctc, eos=self.eos)
+ scorers.update(ctc=ctc)
+ token_list = kwargs.get(&quot;token_list&quot;)
+ scorers.update(
+ decoder=self.decoder,
+ length_bonus=LengthBonus(len(token_list)),
+ )
+
+ # 3. Build ngram model
+ # ngram is not supported now
+ ngram = None
+ scorers[&quot;ngram&quot;] = ngram
+
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/transformer/model.py#L368" target="_blank" class="src-more">View full source →</a></details></div><div class="mblk"><code>.inference(data_in, data_lengths, key, tokenizer, frontend, **kwargs)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/transformer/model.py#L418" target="_blank" class="gh-link">L418</a><div class="mdoc"><p>Run inference on input data.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>data_in</code> — Input data (audio samples, file paths, or text).</li>
+<li><code>data_lengths</code> — Lengths of each input sample in the batch.</li>
+<li><code>key</code> — Sample identifiers.</li>
+<li><code>tokenizer</code> — Tokenizer instance for text encoding/decoding.</li>
+<li><code>frontend</code> — Audio frontend for feature extraction.</li>
+<li><code>**kwargs</code> — Additional keyword arguments.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def inference(
+ self,
+ data_in,
+ data_lengths=None,
+ key: list = None,
+ tokenizer=None,
+ frontend=None,
+ **kwargs,
+ ):
+
+ &quot;&quot;&quot;Run inference on input data.
+
+ Args:
+ data_in: Input data (audio samples, file paths, or text).
+ data_lengths: Lengths of each input sample in the batch.
+ key: Sample identifiers.
+ tokenizer: Tokenizer instance for text encoding/decoding.
+ frontend: Audio frontend for feature extraction.
+ **kwargs: Additional keyword arguments.
+ &quot;&quot;&quot;
+ if kwargs.get(&quot;batch_size&quot;, 1) &gt; 1:
+ raise NotImplementedError(&quot;batch decoding is not implemented&quot;)
+
+ # init beamsearch
+ if self.beam_search is None:
+ logging.info(&quot;enable beam_search&quot;)
+ self.init_beam_search(**kwargs)
+ self.nbest = kwargs.get(&quot;nbest&quot;, 1)
+
+ meta_data = {}
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/transformer/model.py#L418" target="_blank" class="src-more">View full source →</a></details></div></div>
+<div class="api-detail" id="e243"><span class="dtag badge-m">method</span><h2><code>Transformer.forward(speech, speech_lengths, text, text_lengths, **kwargs)</code></h2><div class="dmod">funasr.models.transformer.Transformer · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/transformer/model.py#L173" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Encoder + Decoder + Calc loss</p>
+<h4>Args:</h4>
+<ul>
+<li><code>speech</code> — (Batch, Length, ...)</li>
+<li><code>speech_lengths</code> — (Batch, )</li>
+<li><code>text</code> — (Batch, Length)</li>
+<li><code>text_lengths</code> — (Batch,)</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def forward(
+ self,
+ speech: torch.Tensor,
+ speech_lengths: torch.Tensor,
+ text: torch.Tensor,
+ text_lengths: torch.Tensor,
+ **kwargs,
+ ) -&gt; Tuple[torch.Tensor, Dict[str, torch.Tensor], torch.Tensor]:
+ &quot;&quot;&quot;Encoder + Decoder + Calc loss
+ Args:
+ speech: (Batch, Length, ...)
+ speech_lengths: (Batch, )
+ text: (Batch, Length)
+ text_lengths: (Batch,)
+ &quot;&quot;&quot;
+ if len(text_lengths.size()) &gt; 1:
+ text_lengths = text_lengths[:, 0]
+ if len(speech_lengths.size()) &gt; 1:
+ speech_lengths = speech_lengths[:, 0]
+
+ batch_size = speech.shape[0]
+
+ # 1. Encoder
+ encoder_out, encoder_out_lens = self.encode(speech, speech_lengths)
+ intermediate_outs = None
+ if isinstance(encoder_out, tuple):
+ intermediate_outs = encoder_out[1]
+ encoder_out = encoder_out[0]
+
+ loss_att, acc_att, cer_att, wer_att = None, None, None, None
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/transformer/model.py#L173" target="_blank" class="src-more">View full source on GitHub →</a></details></div>
+<div class="api-detail" id="e244"><span class="dtag badge-m">method</span><h2><code>Transformer.encode(speech, speech_lengths, **kwargs)</code></h2><div class="dmod">funasr.models.transformer.Transformer · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/transformer/model.py#L266" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Frontend + Encoder. Note that this method is used by asr_inference.py</p>
+<h4>Args:</h4>
+<ul>
+<li><code>speech</code> — (Batch, Length, ...)</li>
+<li><code>speech_lengths</code> — (Batch, )</li>
+<li><code>ind</code> — int</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def encode(
+ self,
+ speech: torch.Tensor,
+ speech_lengths: torch.Tensor,
+ **kwargs,
+ ) -&gt; Tuple[torch.Tensor, torch.Tensor]:
+ &quot;&quot;&quot;Frontend + Encoder. Note that this method is used by asr_inference.py
+ Args:
+ speech: (Batch, Length, ...)
+ speech_lengths: (Batch, )
+ ind: int
+ &quot;&quot;&quot;
+ with autocast(False):
+
+ # Data augmentation
+ if self.specaug is not None and self.training:
+ speech, speech_lengths = self.specaug(speech, speech_lengths)
+
+ # Normalization for feature: e.g. Global-CMVN, Utterance-CMVN
+ if self.normalize is not None:
+ speech, speech_lengths = self.normalize(speech, speech_lengths)
+
+ # Forward encoder
+ # feats: (Batch, Length, Dim)
+ # -&gt; encoder_out: (Batch, Length2, Dim2)
+ if self.encoder.interctc_use_conditioning:
+ encoder_out, encoder_out_lens, _ = self.encoder(speech, speech_lengths, ctc=self.ctc)
+ else:
+ encoder_out, encoder_out_lens, _ = self.encoder(speech, speech_lengths)
+ intermediate_outs = None
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/transformer/model.py#L266" target="_blank" class="src-more">View full source on GitHub →</a></details></div>
+<div class="api-detail" id="e245"><span class="dtag badge-m">method</span><h2><code>Transformer.init_beam_search(**kwargs)</code></h2><div class="dmod">funasr.models.transformer.Transformer · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/transformer/model.py#L368" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Init beam search.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>**kwargs</code> — Additional keyword arguments.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def init_beam_search(
+ self,
+ **kwargs,
+ ):
+ &quot;&quot;&quot;Init beam search.
+
+ Args:
+ **kwargs: Additional keyword arguments.
+ &quot;&quot;&quot;
+ from funasr.models.transformer.search import BeamSearch
+ from funasr.models.transformer.scorers.ctc import CTCPrefixScorer
+ from funasr.models.transformer.scorers.length_bonus import LengthBonus
+
+ # 1. Build ASR model
+ scorers = {}
+
+ if self.ctc != None:
+ ctc = CTCPrefixScorer(ctc=self.ctc, eos=self.eos)
+ scorers.update(ctc=ctc)
+ token_list = kwargs.get(&quot;token_list&quot;)
+ scorers.update(
+ decoder=self.decoder,
+ length_bonus=LengthBonus(len(token_list)),
+ )
+
+ # 3. Build ngram model
+ # ngram is not supported now
+ ngram = None
+ scorers[&quot;ngram&quot;] = ngram
+
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/transformer/model.py#L368" target="_blank" class="src-more">View full source on GitHub →</a></details></div>
+<div class="api-detail" id="e246"><span class="dtag badge-m">method</span><h2><code>Transformer.inference(data_in, data_lengths, key, tokenizer, frontend, **kwargs)</code></h2><div class="dmod">funasr.models.transformer.Transformer · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/transformer/model.py#L418" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Run inference on input data.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>data_in</code> — Input data (audio samples, file paths, or text).</li>
+<li><code>data_lengths</code> — Lengths of each input sample in the batch.</li>
+<li><code>key</code> — Sample identifiers.</li>
+<li><code>tokenizer</code> — Tokenizer instance for text encoding/decoding.</li>
+<li><code>frontend</code> — Audio frontend for feature extraction.</li>
+<li><code>**kwargs</code> — Additional keyword arguments.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def inference(
+ self,
+ data_in,
+ data_lengths=None,
+ key: list = None,
+ tokenizer=None,
+ frontend=None,
+ **kwargs,
+ ):
+
+ &quot;&quot;&quot;Run inference on input data.
+
+ Args:
+ data_in: Input data (audio samples, file paths, or text).
+ data_lengths: Lengths of each input sample in the batch.
+ key: Sample identifiers.
+ tokenizer: Tokenizer instance for text encoding/decoding.
+ frontend: Audio frontend for feature extraction.
+ **kwargs: Additional keyword arguments.
+ &quot;&quot;&quot;
+ if kwargs.get(&quot;batch_size&quot;, 1) &gt; 1:
+ raise NotImplementedError(&quot;batch decoding is not implemented&quot;)
+
+ # init beamsearch
+ if self.beam_search is None:
+ logging.info(&quot;enable beam_search&quot;)
+ self.init_beam_search(**kwargs)
+ self.nbest = kwargs.get(&quot;nbest&quot;, 1)
+
+ meta_data = {}
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/transformer/model.py#L418" target="_blank" class="src-more">View full source on GitHub →</a></details></div>
+<div class="api-detail" id="e247"><span class="dtag badge-c">class</span><h2>UniASR</h2><div class="dmod">funasr.models.uniasr · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/uniasr/model.py#L27" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><ul>
+<li><code>UniASR</code> — Unified Streaming and Non-streaming ASR model.</li>
+</ul>
+<p>Single model that supports both streaming (online) and non-streaming</p>
+<p>(offline) decoding through dynamic masking in the encoder.</p>
+<p>Inherits Paraformer pipeline.</p>
+<ul>
+<li><code>Output</code> — {&quot;key&quot;: str, &quot;text&quot;: str}</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code>class UniASR(torch.nn.Module):
+ &quot;&quot;&quot;UniASR: Unified Streaming and Non-streaming ASR model.
+
+ Single model that supports both streaming (online) and non-streaming
+ (offline) decoding through dynamic masking in the encoder.
+
+ Inherits Paraformer pipeline.
+ Output: {&quot;key&quot;: str, &quot;text&quot;: str}
+ &quot;&quot;&quot;
+
+ def __init__(
+ self,
+ specaug: str = None,
+ specaug_conf: dict = None,
+ normalize: str = None,
+ normalize_conf: dict = None,
+ encoder: str = None,
+ encoder_conf: dict = None,
+ encoder2: str = None,
+ encoder2_conf: dict = None,
+ decoder: str = None,
+ decoder_conf: dict = None,
+ decoder2: str = None,
+ decoder2_conf: dict = None,
+ predictor: str = None,
+ predictor_conf: dict = None,
+ predictor_bias: int = 0,
+ predictor_weight: float = 0.0,
+ predictor2: str = None,
+ predictor2_conf: dict = None,
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/uniasr/model.py#L27" target="_blank" class="src-more">View full source on GitHub →</a></details><h3 class="mt">Methods</h3><div class="mblk"><code>.forward(speech, speech_lengths, text, text_lengths, **kwargs)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/uniasr/model.py#L235" target="_blank" class="gh-link">L235</a><div class="mdoc"><p>Frontend + Encoder + Decoder + Calc loss</p>
+<h4>Args:</h4>
+<ul>
+<li><code>speech</code> — (Batch, Length, ...)</li>
+<li><code>speech_lengths</code> — (Batch, )</li>
+<li><code>text</code> — (Batch, Length)</li>
+<li><code>text_lengths</code> — (Batch,)</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def forward(
+ self,
+ speech: torch.Tensor,
+ speech_lengths: torch.Tensor,
+ text: torch.Tensor,
+ text_lengths: torch.Tensor,
+ **kwargs,
+ ) -&gt; Tuple[torch.Tensor, Dict[str, torch.Tensor], torch.Tensor]:
+ &quot;&quot;&quot;Frontend + Encoder + Decoder + Calc loss
+ Args:
+ speech: (Batch, Length, ...)
+ speech_lengths: (Batch, )
+ text: (Batch, Length)
+ text_lengths: (Batch,)
+ &quot;&quot;&quot;
+ decoding_ind = kwargs.get(&quot;decoding_ind&quot;, None)
+ if len(text_lengths.size()) &gt; 1:
+ text_lengths = text_lengths[:, 0]
+ if len(speech_lengths.size()) &gt; 1:
+ speech_lengths = speech_lengths[:, 0]
+
+ batch_size = speech.shape[0]
+
+ ind = self.encoder.overlap_chunk_cls.random_choice(self.training, decoding_ind)
+ # 1. Encoder
+ if self.enable_maas_finetune:
+ with torch.no_grad():
+ speech_raw, encoder_out, encoder_out_lens = self.encode(
+ speech, speech_lengths, ind=ind
+ )
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/uniasr/model.py#L235" target="_blank" class="src-more">View full source →</a></details></div><div class="mblk"><code>.collect_feats(speech, speech_lengths, text, text_lengths)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/uniasr/model.py#L354" target="_blank" class="gh-link">L354</a><div class="mdoc"><p>Collect feats.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>speech</code> — Speech audio tensor, shape (batch, time).</li>
+<li><code>speech_lengths</code> — Length of each speech sample.</li>
+<li><code>text</code> — Text tensor or string input.</li>
+<li><code>text_lengths</code> — Length of each text sample.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def collect_feats(
+ self,
+ speech: torch.Tensor,
+ speech_lengths: torch.Tensor,
+ text: torch.Tensor,
+ text_lengths: torch.Tensor,
+ ) -&gt; Dict[str, torch.Tensor]:
+ &quot;&quot;&quot;Collect feats.
+
+ Args:
+ speech: Speech audio tensor, shape (batch, time).
+ speech_lengths: Length of each speech sample.
+ text: Text tensor or string input.
+ text_lengths: Length of each text sample.
+ &quot;&quot;&quot;
+ if self.extract_feats_in_collect_stats:
+ feats, feats_lengths = self._extract_feats(speech, speech_lengths)
+ else:
+ # Generate dummy stats if extract_feats_in_collect_stats is False
+ logging.warning(
+ &quot;Generating dummy stats for feats and feats_lengths, &quot;
+ &quot;because encoder_conf.extract_feats_in_collect_stats is &quot;
+ f&quot;{self.extract_feats_in_collect_stats}&quot;
+ )
+ feats, feats_lengths = speech, speech_lengths
+ return {&quot;feats&quot;: feats, &quot;feats_lengths&quot;: feats_lengths}
+</code></pre></details></div><div class="mblk"><code>.encode(speech, speech_lengths, **kwargs)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/uniasr/model.py#L381" target="_blank" class="gh-link">L381</a><div class="mdoc"><p>Frontend + Encoder. Note that this method is used by asr_inference.py</p>
+<h4>Args:</h4>
+<ul>
+<li><code>speech</code> — (Batch, Length, ...)</li>
+<li><code>speech_lengths</code> — (Batch, )</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def encode(
+ self,
+ speech: torch.Tensor,
+ speech_lengths: torch.Tensor,
+ **kwargs,
+ ):
+ &quot;&quot;&quot;Frontend + Encoder. Note that this method is used by asr_inference.py
+ Args:
+ speech: (Batch, Length, ...)
+ speech_lengths: (Batch, )
+ &quot;&quot;&quot;
+ ind = kwargs.get(&quot;ind&quot;, 0)
+ with autocast(False):
+ # Data augmentation
+ if self.specaug is not None and self.training:
+ speech, speech_lengths = self.specaug(speech, speech_lengths)
+
+ # Normalization for feature: e.g. Global-CMVN, Utterance-CMVN
+ if self.normalize is not None:
+ speech, speech_lengths = self.normalize(speech, speech_lengths)
+
+ speech_raw = speech.clone().to(speech.device)
+
+ # 4. Forward encoder
+ encoder_out, encoder_out_lens, _ = self.encoder(speech, speech_lengths, ind=ind)
+ if isinstance(encoder_out, tuple):
+ encoder_out = encoder_out[0]
+
+ return speech_raw, encoder_out, encoder_out_lens
+</code></pre></details></div><div class="mblk"><code>.encode2(encoder_out, encoder_out_lens, speech, speech_lengths, **kwargs)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/uniasr/model.py#L411" target="_blank" class="gh-link">L411</a><div class="mdoc"><p>Frontend + Encoder. Note that this method is used by asr_inference.py</p>
+<h4>Args:</h4>
+<ul>
+<li><code>speech</code> — (Batch, Length, ...)</li>
+<li><code>speech_lengths</code> — (Batch, )</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def encode2(
+ self,
+ encoder_out: torch.Tensor,
+ encoder_out_lens: torch.Tensor,
+ speech: torch.Tensor,
+ speech_lengths: torch.Tensor,
+ **kwargs,
+ ):
+ &quot;&quot;&quot;Frontend + Encoder. Note that this method is used by asr_inference.py
+ Args:
+ speech: (Batch, Length, ...)
+ speech_lengths: (Batch, )
+ &quot;&quot;&quot;
+
+ ind = kwargs.get(&quot;ind&quot;, 0)
+ encoder_out_rm, encoder_out_lens_rm = self.encoder.overlap_chunk_cls.remove_chunk(
+ encoder_out,
+ encoder_out_lens,
+ chunk_outs=None,
+ )
+ # residual_input
+ encoder_out = torch.cat((speech, encoder_out_rm), dim=-1)
+ encoder_out_lens = encoder_out_lens_rm
+ if self.stride_conv is not None:
+ speech, speech_lengths = self.stride_conv(encoder_out, encoder_out_lens)
+ if not self.encoder1_encoder2_joint_training:
+ speech = speech.detach()
+ speech_lengths = speech_lengths.detach()
+ # 4. Forward encoder
+ # feats: (Batch, Length, Dim)
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/uniasr/model.py#L411" target="_blank" class="src-more">View full source →</a></details></div><div class="mblk"><code>.nll(encoder_out, encoder_out_lens, ys_pad, ys_pad_lens)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/uniasr/model.py#L449" target="_blank" class="gh-link">L449</a><div class="mdoc"><p>Compute negative log likelihood(nll) from transformer-decoder</p>
+<p>Normally, this function is called in batchify_nll.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>encoder_out</code> — (Batch, Length, Dim)</li>
+<li><code>encoder_out_lens</code> — (Batch,)</li>
+<li><code>ys_pad</code> — (Batch, Length)</li>
+<li><code>ys_pad_lens</code> — (Batch,)</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def nll(
+ self,
+ encoder_out: torch.Tensor,
+ encoder_out_lens: torch.Tensor,
+ ys_pad: torch.Tensor,
+ ys_pad_lens: torch.Tensor,
+ ) -&gt; torch.Tensor:
+ &quot;&quot;&quot;Compute negative log likelihood(nll) from transformer-decoder
+ Normally, this function is called in batchify_nll.
+ Args:
+ encoder_out: (Batch, Length, Dim)
+ encoder_out_lens: (Batch,)
+ ys_pad: (Batch, Length)
+ ys_pad_lens: (Batch,)
+ &quot;&quot;&quot;
+ ys_in_pad, ys_out_pad = add_sos_eos(ys_pad, self.sos, self.eos, self.ignore_id)
+ ys_in_lens = ys_pad_lens + 1
+
+ # 1. Forward decoder
+ decoder_out, _ = self.decoder(
+ encoder_out, encoder_out_lens, ys_in_pad, ys_in_lens
+ ) # [batch, seqlen, dim]
+ batch_size = decoder_out.size(0)
+ decoder_num_class = decoder_out.size(2)
+ # nll: negative log-likelihood
+ nll = torch.nn.functional.cross_entropy(
+ decoder_out.view(-1, decoder_num_class),
+ ys_out_pad.view(-1),
+ ignore_index=self.ignore_id,
+ reduction=&quot;none&quot;,
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/uniasr/model.py#L449" target="_blank" class="src-more">View full source →</a></details></div><div class="mblk"><code>.batchify_nll(encoder_out, encoder_out_lens, ys_pad, ys_pad_lens, batch_size)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/uniasr/model.py#L485" target="_blank" class="gh-link">L485</a><div class="mdoc"><p>Compute negative log likelihood(nll) from transformer-decoder</p>
+<p>To avoid OOM, this fuction seperate the input into batches.</p>
+<p>Then call nll for each batch and combine and return results.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>encoder_out</code> — (Batch, Length, Dim)</li>
+<li><code>encoder_out_lens</code> — (Batch,)</li>
+<li><code>ys_pad</code> — (Batch, Length)</li>
+<li><code>ys_pad_lens</code> — (Batch,)</li>
+<li><code>batch_size</code> — int, samples each batch contain when computing nll,</li>
+<li class="sub">you may change this to avoid OOM or increase</li>
+<li class="sub">GPU memory usage</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def batchify_nll(
+ self,
+ encoder_out: torch.Tensor,
+ encoder_out_lens: torch.Tensor,
+ ys_pad: torch.Tensor,
+ ys_pad_lens: torch.Tensor,
+ batch_size: int = 100,
+ ):
+ &quot;&quot;&quot;Compute negative log likelihood(nll) from transformer-decoder
+ To avoid OOM, this fuction seperate the input into batches.
+ Then call nll for each batch and combine and return results.
+ Args:
+ encoder_out: (Batch, Length, Dim)
+ encoder_out_lens: (Batch,)
+ ys_pad: (Batch, Length)
+ ys_pad_lens: (Batch,)
+ batch_size: int, samples each batch contain when computing nll,
+ you may change this to avoid OOM or increase
+ GPU memory usage
+ &quot;&quot;&quot;
+ total_num = encoder_out.size(0)
+ if total_num &lt;= batch_size:
+ nll = self.nll(encoder_out, encoder_out_lens, ys_pad, ys_pad_lens)
+ else:
+ nll = []
+ start_idx = 0
+ while True:
+ end_idx = min(start_idx + batch_size, total_num)
+ batch_encoder_out = encoder_out[start_idx:end_idx, :, :]
+ batch_encoder_out_lens = encoder_out_lens[start_idx:end_idx]
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/uniasr/model.py#L485" target="_blank" class="src-more">View full source →</a></details></div><div class="mblk"><code>.calc_predictor_mask(encoder_out, encoder_out_lens, ys_pad, ys_pad_lens)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/uniasr/model.py#L787" target="_blank" class="gh-link">L787</a><div class="mdoc"><p>Calc predictor mask.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>encoder_out</code> — Encoder output tensor.</li>
+<li><code>encoder_out_lens</code> — Encoder output lengths.</li>
+<li><code>ys_pad</code> — TODO.</li>
+<li><code>ys_pad_lens</code> — Lengths of ys_pad.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def calc_predictor_mask(
+ self,
+ encoder_out: torch.Tensor,
+ encoder_out_lens: torch.Tensor,
+ ys_pad: torch.Tensor = None,
+ ys_pad_lens: torch.Tensor = None,
+ ):
+ # ys_in_pad, ys_out_pad = add_sos_eos(ys_pad, self.sos, self.eos, self.ignore_id)
+ # ys_in_lens = ys_pad_lens + 1
+ &quot;&quot;&quot;Calc predictor mask.
+
+ Args:
+ encoder_out: Encoder output tensor.
+ encoder_out_lens: Encoder output lengths.
+ ys_pad: TODO.
+ ys_pad_lens: Lengths of ys_pad.
+ &quot;&quot;&quot;
+ ys_out_pad, ys_in_lens = None, None
+
+ encoder_out_mask = sequence_mask(
+ encoder_out_lens,
+ maxlen=encoder_out.size(1),
+ dtype=encoder_out.dtype,
+ device=encoder_out.device,
+ )[:, None, :]
+ mask_chunk_predictor = None
+ if self.encoder.overlap_chunk_cls is not None:
+ mask_chunk_predictor = self.encoder.overlap_chunk_cls.get_mask_chunk_predictor(
+ None, device=encoder_out.device, batch_size=encoder_out.size(0)
+ )
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/uniasr/model.py#L787" target="_blank" class="src-more">View full source →</a></details></div><div class="mblk"><code>.calc_predictor_mask2(encoder_out, encoder_out_lens, ys_pad, ys_pad_lens)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/uniasr/model.py#L877" target="_blank" class="gh-link">L877</a><div class="mdoc"><p>Calc predictor mask2.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>encoder_out</code> — Encoder output tensor.</li>
+<li><code>encoder_out_lens</code> — Encoder output lengths.</li>
+<li><code>ys_pad</code> — TODO.</li>
+<li><code>ys_pad_lens</code> — Lengths of ys_pad.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def calc_predictor_mask2(
+ self,
+ encoder_out: torch.Tensor,
+ encoder_out_lens: torch.Tensor,
+ ys_pad: torch.Tensor = None,
+ ys_pad_lens: torch.Tensor = None,
+ ):
+ # ys_in_pad, ys_out_pad = add_sos_eos(ys_pad, self.sos, self.eos, self.ignore_id)
+ # ys_in_lens = ys_pad_lens + 1
+ &quot;&quot;&quot;Calc predictor mask2.
+
+ Args:
+ encoder_out: Encoder output tensor.
+ encoder_out_lens: Encoder output lengths.
+ ys_pad: TODO.
+ ys_pad_lens: Lengths of ys_pad.
+ &quot;&quot;&quot;
+ ys_out_pad, ys_in_lens = None, None
+
+ encoder_out_mask = sequence_mask(
+ encoder_out_lens,
+ maxlen=encoder_out.size(1),
+ dtype=encoder_out.dtype,
+ device=encoder_out.device,
+ )[:, None, :]
+ mask_chunk_predictor = None
+ if self.encoder2.overlap_chunk_cls is not None:
+ mask_chunk_predictor = self.encoder2.overlap_chunk_cls.get_mask_chunk_predictor(
+ None, device=encoder_out.device, batch_size=encoder_out.size(0)
+ )
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/uniasr/model.py#L877" target="_blank" class="src-more">View full source →</a></details></div><div class="mblk"><code>.init_beam_search(**kwargs)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/uniasr/model.py#L967" target="_blank" class="gh-link">L967</a><div class="mdoc"><p>Init beam search.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>**kwargs</code> — Additional keyword arguments.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def init_beam_search(
+ self,
+ **kwargs,
+ ):
+ &quot;&quot;&quot;Init beam search.
+
+ Args:
+ **kwargs: Additional keyword arguments.
+ &quot;&quot;&quot;
+ from funasr.models.uniasr.beam_search import BeamSearchScama
+ from funasr.models.transformer.scorers.ctc import CTCPrefixScorer
+ from funasr.models.transformer.scorers.length_bonus import LengthBonus
+
+ decoding_mode = kwargs.get(&quot;decoding_mode&quot;, &quot;model1&quot;)
+ if decoding_mode == &quot;model1&quot;:
+ decoder = self.decoder
+ else:
+ decoder = self.decoder2
+ # 1. Build ASR model
+ scorers = {}
+
+ if self.ctc != None:
+ ctc = CTCPrefixScorer(ctc=self.ctc, eos=self.eos)
+ scorers.update(ctc=ctc)
+ token_list = kwargs.get(&quot;token_list&quot;)
+ scorers.update(
+ decoder=decoder,
+ length_bonus=LengthBonus(len(token_list)),
+ )
+
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/uniasr/model.py#L967" target="_blank" class="src-more">View full source →</a></details></div><div class="mblk"><code>.inference(data_in, data_lengths, key, tokenizer, frontend, **kwargs)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/uniasr/model.py#L1022" target="_blank" class="gh-link">L1022</a><div class="mdoc"><p>Run inference on input data.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>data_in</code> — Input data (audio samples, file paths, or text).</li>
+<li><code>data_lengths</code> — Lengths of each input sample in the batch.</li>
+<li><code>key</code> — Sample identifiers.</li>
+<li><code>tokenizer</code> — Tokenizer instance for text encoding/decoding.</li>
+<li><code>frontend</code> — Audio frontend for feature extraction.</li>
+<li><code>**kwargs</code> — Additional keyword arguments.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def inference(
+ self,
+ data_in,
+ data_lengths=None,
+ key: list = None,
+ tokenizer=None,
+ frontend=None,
+ **kwargs,
+ ):
+
+ &quot;&quot;&quot;Run inference on input data.
+
+ Args:
+ data_in: Input data (audio samples, file paths, or text).
+ data_lengths: Lengths of each input sample in the batch.
+ key: Sample identifiers.
+ tokenizer: Tokenizer instance for text encoding/decoding.
+ frontend: Audio frontend for feature extraction.
+ **kwargs: Additional keyword arguments.
+ &quot;&quot;&quot;
+ decoding_model = kwargs.get(&quot;decoding_model&quot;, &quot;normal&quot;)
+ token_num_relax = kwargs.get(&quot;token_num_relax&quot;, 5)
+ if decoding_model == &quot;fast&quot;:
+ decoding_ind = 0
+ decoding_mode = &quot;model1&quot;
+ elif decoding_model == &quot;offline&quot;:
+ decoding_ind = 1
+ decoding_mode = &quot;model2&quot;
+ else:
+ decoding_ind = 0
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/uniasr/model.py#L1022" target="_blank" class="src-more">View full source →</a></details></div></div>
+<div class="api-detail" id="e248"><span class="dtag badge-m">method</span><h2><code>UniASR.forward(speech, speech_lengths, text, text_lengths, **kwargs)</code></h2><div class="dmod">funasr.models.uniasr.UniASR · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/uniasr/model.py#L235" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Frontend + Encoder + Decoder + Calc loss</p>
+<h4>Args:</h4>
+<ul>
+<li><code>speech</code> — (Batch, Length, ...)</li>
+<li><code>speech_lengths</code> — (Batch, )</li>
+<li><code>text</code> — (Batch, Length)</li>
+<li><code>text_lengths</code> — (Batch,)</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def forward(
+ self,
+ speech: torch.Tensor,
+ speech_lengths: torch.Tensor,
+ text: torch.Tensor,
+ text_lengths: torch.Tensor,
+ **kwargs,
+ ) -&gt; Tuple[torch.Tensor, Dict[str, torch.Tensor], torch.Tensor]:
+ &quot;&quot;&quot;Frontend + Encoder + Decoder + Calc loss
+ Args:
+ speech: (Batch, Length, ...)
+ speech_lengths: (Batch, )
+ text: (Batch, Length)
+ text_lengths: (Batch,)
+ &quot;&quot;&quot;
+ decoding_ind = kwargs.get(&quot;decoding_ind&quot;, None)
+ if len(text_lengths.size()) &gt; 1:
+ text_lengths = text_lengths[:, 0]
+ if len(speech_lengths.size()) &gt; 1:
+ speech_lengths = speech_lengths[:, 0]
+
+ batch_size = speech.shape[0]
+
+ ind = self.encoder.overlap_chunk_cls.random_choice(self.training, decoding_ind)
+ # 1. Encoder
+ if self.enable_maas_finetune:
+ with torch.no_grad():
+ speech_raw, encoder_out, encoder_out_lens = self.encode(
+ speech, speech_lengths, ind=ind
+ )
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/uniasr/model.py#L235" target="_blank" class="src-more">View full source on GitHub →</a></details></div>
+<div class="api-detail" id="e249"><span class="dtag badge-m">method</span><h2><code>UniASR.collect_feats(speech, speech_lengths, text, text_lengths)</code></h2><div class="dmod">funasr.models.uniasr.UniASR · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/uniasr/model.py#L354" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Collect feats.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>speech</code> — Speech audio tensor, shape (batch, time).</li>
+<li><code>speech_lengths</code> — Length of each speech sample.</li>
+<li><code>text</code> — Text tensor or string input.</li>
+<li><code>text_lengths</code> — Length of each text sample.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def collect_feats(
+ self,
+ speech: torch.Tensor,
+ speech_lengths: torch.Tensor,
+ text: torch.Tensor,
+ text_lengths: torch.Tensor,
+ ) -&gt; Dict[str, torch.Tensor]:
+ &quot;&quot;&quot;Collect feats.
+
+ Args:
+ speech: Speech audio tensor, shape (batch, time).
+ speech_lengths: Length of each speech sample.
+ text: Text tensor or string input.
+ text_lengths: Length of each text sample.
+ &quot;&quot;&quot;
+ if self.extract_feats_in_collect_stats:
+ feats, feats_lengths = self._extract_feats(speech, speech_lengths)
+ else:
+ # Generate dummy stats if extract_feats_in_collect_stats is False
+ logging.warning(
+ &quot;Generating dummy stats for feats and feats_lengths, &quot;
+ &quot;because encoder_conf.extract_feats_in_collect_stats is &quot;
+ f&quot;{self.extract_feats_in_collect_stats}&quot;
+ )
+ feats, feats_lengths = speech, speech_lengths
+ return {&quot;feats&quot;: feats, &quot;feats_lengths&quot;: feats_lengths}
+</code></pre></details></div>
+<div class="api-detail" id="e250"><span class="dtag badge-m">method</span><h2><code>UniASR.encode(speech, speech_lengths, **kwargs)</code></h2><div class="dmod">funasr.models.uniasr.UniASR · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/uniasr/model.py#L381" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Frontend + Encoder. Note that this method is used by asr_inference.py</p>
+<h4>Args:</h4>
+<ul>
+<li><code>speech</code> — (Batch, Length, ...)</li>
+<li><code>speech_lengths</code> — (Batch, )</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def encode(
+ self,
+ speech: torch.Tensor,
+ speech_lengths: torch.Tensor,
+ **kwargs,
+ ):
+ &quot;&quot;&quot;Frontend + Encoder. Note that this method is used by asr_inference.py
+ Args:
+ speech: (Batch, Length, ...)
+ speech_lengths: (Batch, )
+ &quot;&quot;&quot;
+ ind = kwargs.get(&quot;ind&quot;, 0)
+ with autocast(False):
+ # Data augmentation
+ if self.specaug is not None and self.training:
+ speech, speech_lengths = self.specaug(speech, speech_lengths)
+
+ # Normalization for feature: e.g. Global-CMVN, Utterance-CMVN
+ if self.normalize is not None:
+ speech, speech_lengths = self.normalize(speech, speech_lengths)
+
+ speech_raw = speech.clone().to(speech.device)
+
+ # 4. Forward encoder
+ encoder_out, encoder_out_lens, _ = self.encoder(speech, speech_lengths, ind=ind)
+ if isinstance(encoder_out, tuple):
+ encoder_out = encoder_out[0]
+
+ return speech_raw, encoder_out, encoder_out_lens
+</code></pre></details></div>
+<div class="api-detail" id="e251"><span class="dtag badge-m">method</span><h2><code>UniASR.encode2(encoder_out, encoder_out_lens, speech, speech_lengths, **kwargs)</code></h2><div class="dmod">funasr.models.uniasr.UniASR · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/uniasr/model.py#L411" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Frontend + Encoder. Note that this method is used by asr_inference.py</p>
+<h4>Args:</h4>
+<ul>
+<li><code>speech</code> — (Batch, Length, ...)</li>
+<li><code>speech_lengths</code> — (Batch, )</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def encode2(
+ self,
+ encoder_out: torch.Tensor,
+ encoder_out_lens: torch.Tensor,
+ speech: torch.Tensor,
+ speech_lengths: torch.Tensor,
+ **kwargs,
+ ):
+ &quot;&quot;&quot;Frontend + Encoder. Note that this method is used by asr_inference.py
+ Args:
+ speech: (Batch, Length, ...)
+ speech_lengths: (Batch, )
+ &quot;&quot;&quot;
+
+ ind = kwargs.get(&quot;ind&quot;, 0)
+ encoder_out_rm, encoder_out_lens_rm = self.encoder.overlap_chunk_cls.remove_chunk(
+ encoder_out,
+ encoder_out_lens,
+ chunk_outs=None,
+ )
+ # residual_input
+ encoder_out = torch.cat((speech, encoder_out_rm), dim=-1)
+ encoder_out_lens = encoder_out_lens_rm
+ if self.stride_conv is not None:
+ speech, speech_lengths = self.stride_conv(encoder_out, encoder_out_lens)
+ if not self.encoder1_encoder2_joint_training:
+ speech = speech.detach()
+ speech_lengths = speech_lengths.detach()
+ # 4. Forward encoder
+ # feats: (Batch, Length, Dim)
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/uniasr/model.py#L411" target="_blank" class="src-more">View full source on GitHub →</a></details></div>
+<div class="api-detail" id="e252"><span class="dtag badge-m">method</span><h2><code>UniASR.nll(encoder_out, encoder_out_lens, ys_pad, ys_pad_lens)</code></h2><div class="dmod">funasr.models.uniasr.UniASR · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/uniasr/model.py#L449" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Compute negative log likelihood(nll) from transformer-decoder</p>
+<p>Normally, this function is called in batchify_nll.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>encoder_out</code> — (Batch, Length, Dim)</li>
+<li><code>encoder_out_lens</code> — (Batch,)</li>
+<li><code>ys_pad</code> — (Batch, Length)</li>
+<li><code>ys_pad_lens</code> — (Batch,)</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def nll(
+ self,
+ encoder_out: torch.Tensor,
+ encoder_out_lens: torch.Tensor,
+ ys_pad: torch.Tensor,
+ ys_pad_lens: torch.Tensor,
+ ) -&gt; torch.Tensor:
+ &quot;&quot;&quot;Compute negative log likelihood(nll) from transformer-decoder
+ Normally, this function is called in batchify_nll.
+ Args:
+ encoder_out: (Batch, Length, Dim)
+ encoder_out_lens: (Batch,)
+ ys_pad: (Batch, Length)
+ ys_pad_lens: (Batch,)
+ &quot;&quot;&quot;
+ ys_in_pad, ys_out_pad = add_sos_eos(ys_pad, self.sos, self.eos, self.ignore_id)
+ ys_in_lens = ys_pad_lens + 1
+
+ # 1. Forward decoder
+ decoder_out, _ = self.decoder(
+ encoder_out, encoder_out_lens, ys_in_pad, ys_in_lens
+ ) # [batch, seqlen, dim]
+ batch_size = decoder_out.size(0)
+ decoder_num_class = decoder_out.size(2)
+ # nll: negative log-likelihood
+ nll = torch.nn.functional.cross_entropy(
+ decoder_out.view(-1, decoder_num_class),
+ ys_out_pad.view(-1),
+ ignore_index=self.ignore_id,
+ reduction=&quot;none&quot;,
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/uniasr/model.py#L449" target="_blank" class="src-more">View full source on GitHub →</a></details></div>
+<div class="api-detail" id="e253"><span class="dtag badge-m">method</span><h2><code>UniASR.batchify_nll(encoder_out, encoder_out_lens, ys_pad, ys_pad_lens, batch_size)</code></h2><div class="dmod">funasr.models.uniasr.UniASR · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/uniasr/model.py#L485" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Compute negative log likelihood(nll) from transformer-decoder</p>
+<p>To avoid OOM, this fuction seperate the input into batches.</p>
+<p>Then call nll for each batch and combine and return results.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>encoder_out</code> — (Batch, Length, Dim)</li>
+<li><code>encoder_out_lens</code> — (Batch,)</li>
+<li><code>ys_pad</code> — (Batch, Length)</li>
+<li><code>ys_pad_lens</code> — (Batch,)</li>
+<li><code>batch_size</code> — int, samples each batch contain when computing nll,</li>
+<li class="sub">you may change this to avoid OOM or increase</li>
+<li class="sub">GPU memory usage</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def batchify_nll(
+ self,
+ encoder_out: torch.Tensor,
+ encoder_out_lens: torch.Tensor,
+ ys_pad: torch.Tensor,
+ ys_pad_lens: torch.Tensor,
+ batch_size: int = 100,
+ ):
+ &quot;&quot;&quot;Compute negative log likelihood(nll) from transformer-decoder
+ To avoid OOM, this fuction seperate the input into batches.
+ Then call nll for each batch and combine and return results.
+ Args:
+ encoder_out: (Batch, Length, Dim)
+ encoder_out_lens: (Batch,)
+ ys_pad: (Batch, Length)
+ ys_pad_lens: (Batch,)
+ batch_size: int, samples each batch contain when computing nll,
+ you may change this to avoid OOM or increase
+ GPU memory usage
+ &quot;&quot;&quot;
+ total_num = encoder_out.size(0)
+ if total_num &lt;= batch_size:
+ nll = self.nll(encoder_out, encoder_out_lens, ys_pad, ys_pad_lens)
+ else:
+ nll = []
+ start_idx = 0
+ while True:
+ end_idx = min(start_idx + batch_size, total_num)
+ batch_encoder_out = encoder_out[start_idx:end_idx, :, :]
+ batch_encoder_out_lens = encoder_out_lens[start_idx:end_idx]
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/uniasr/model.py#L485" target="_blank" class="src-more">View full source on GitHub →</a></details></div>
+<div class="api-detail" id="e254"><span class="dtag badge-m">method</span><h2><code>UniASR.calc_predictor_mask(encoder_out, encoder_out_lens, ys_pad, ys_pad_lens)</code></h2><div class="dmod">funasr.models.uniasr.UniASR · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/uniasr/model.py#L787" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Calc predictor mask.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>encoder_out</code> — Encoder output tensor.</li>
+<li><code>encoder_out_lens</code> — Encoder output lengths.</li>
+<li><code>ys_pad</code> — TODO.</li>
+<li><code>ys_pad_lens</code> — Lengths of ys_pad.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def calc_predictor_mask(
+ self,
+ encoder_out: torch.Tensor,
+ encoder_out_lens: torch.Tensor,
+ ys_pad: torch.Tensor = None,
+ ys_pad_lens: torch.Tensor = None,
+ ):
+ # ys_in_pad, ys_out_pad = add_sos_eos(ys_pad, self.sos, self.eos, self.ignore_id)
+ # ys_in_lens = ys_pad_lens + 1
+ &quot;&quot;&quot;Calc predictor mask.
+
+ Args:
+ encoder_out: Encoder output tensor.
+ encoder_out_lens: Encoder output lengths.
+ ys_pad: TODO.
+ ys_pad_lens: Lengths of ys_pad.
+ &quot;&quot;&quot;
+ ys_out_pad, ys_in_lens = None, None
+
+ encoder_out_mask = sequence_mask(
+ encoder_out_lens,
+ maxlen=encoder_out.size(1),
+ dtype=encoder_out.dtype,
+ device=encoder_out.device,
+ )[:, None, :]
+ mask_chunk_predictor = None
+ if self.encoder.overlap_chunk_cls is not None:
+ mask_chunk_predictor = self.encoder.overlap_chunk_cls.get_mask_chunk_predictor(
+ None, device=encoder_out.device, batch_size=encoder_out.size(0)
+ )
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/uniasr/model.py#L787" target="_blank" class="src-more">View full source on GitHub →</a></details></div>
+<div class="api-detail" id="e255"><span class="dtag badge-m">method</span><h2><code>UniASR.calc_predictor_mask2(encoder_out, encoder_out_lens, ys_pad, ys_pad_lens)</code></h2><div class="dmod">funasr.models.uniasr.UniASR · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/uniasr/model.py#L877" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Calc predictor mask2.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>encoder_out</code> — Encoder output tensor.</li>
+<li><code>encoder_out_lens</code> — Encoder output lengths.</li>
+<li><code>ys_pad</code> — TODO.</li>
+<li><code>ys_pad_lens</code> — Lengths of ys_pad.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def calc_predictor_mask2(
+ self,
+ encoder_out: torch.Tensor,
+ encoder_out_lens: torch.Tensor,
+ ys_pad: torch.Tensor = None,
+ ys_pad_lens: torch.Tensor = None,
+ ):
+ # ys_in_pad, ys_out_pad = add_sos_eos(ys_pad, self.sos, self.eos, self.ignore_id)
+ # ys_in_lens = ys_pad_lens + 1
+ &quot;&quot;&quot;Calc predictor mask2.
+
+ Args:
+ encoder_out: Encoder output tensor.
+ encoder_out_lens: Encoder output lengths.
+ ys_pad: TODO.
+ ys_pad_lens: Lengths of ys_pad.
+ &quot;&quot;&quot;
+ ys_out_pad, ys_in_lens = None, None
+
+ encoder_out_mask = sequence_mask(
+ encoder_out_lens,
+ maxlen=encoder_out.size(1),
+ dtype=encoder_out.dtype,
+ device=encoder_out.device,
+ )[:, None, :]
+ mask_chunk_predictor = None
+ if self.encoder2.overlap_chunk_cls is not None:
+ mask_chunk_predictor = self.encoder2.overlap_chunk_cls.get_mask_chunk_predictor(
+ None, device=encoder_out.device, batch_size=encoder_out.size(0)
+ )
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/uniasr/model.py#L877" target="_blank" class="src-more">View full source on GitHub →</a></details></div>
+<div class="api-detail" id="e256"><span class="dtag badge-m">method</span><h2><code>UniASR.init_beam_search(**kwargs)</code></h2><div class="dmod">funasr.models.uniasr.UniASR · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/uniasr/model.py#L967" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Init beam search.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>**kwargs</code> — Additional keyword arguments.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def init_beam_search(
+ self,
+ **kwargs,
+ ):
+ &quot;&quot;&quot;Init beam search.
+
+ Args:
+ **kwargs: Additional keyword arguments.
+ &quot;&quot;&quot;
+ from funasr.models.uniasr.beam_search import BeamSearchScama
+ from funasr.models.transformer.scorers.ctc import CTCPrefixScorer
+ from funasr.models.transformer.scorers.length_bonus import LengthBonus
+
+ decoding_mode = kwargs.get(&quot;decoding_mode&quot;, &quot;model1&quot;)
+ if decoding_mode == &quot;model1&quot;:
+ decoder = self.decoder
+ else:
+ decoder = self.decoder2
+ # 1. Build ASR model
+ scorers = {}
+
+ if self.ctc != None:
+ ctc = CTCPrefixScorer(ctc=self.ctc, eos=self.eos)
+ scorers.update(ctc=ctc)
+ token_list = kwargs.get(&quot;token_list&quot;)
+ scorers.update(
+ decoder=decoder,
+ length_bonus=LengthBonus(len(token_list)),
+ )
+
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/uniasr/model.py#L967" target="_blank" class="src-more">View full source on GitHub →</a></details></div>
+<div class="api-detail" id="e257"><span class="dtag badge-m">method</span><h2><code>UniASR.inference(data_in, data_lengths, key, tokenizer, frontend, **kwargs)</code></h2><div class="dmod">funasr.models.uniasr.UniASR · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/uniasr/model.py#L1022" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Run inference on input data.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>data_in</code> — Input data (audio samples, file paths, or text).</li>
+<li><code>data_lengths</code> — Lengths of each input sample in the batch.</li>
+<li><code>key</code> — Sample identifiers.</li>
+<li><code>tokenizer</code> — Tokenizer instance for text encoding/decoding.</li>
+<li><code>frontend</code> — Audio frontend for feature extraction.</li>
+<li><code>**kwargs</code> — Additional keyword arguments.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def inference(
+ self,
+ data_in,
+ data_lengths=None,
+ key: list = None,
+ tokenizer=None,
+ frontend=None,
+ **kwargs,
+ ):
+
+ &quot;&quot;&quot;Run inference on input data.
+
+ Args:
+ data_in: Input data (audio samples, file paths, or text).
+ data_lengths: Lengths of each input sample in the batch.
+ key: Sample identifiers.
+ tokenizer: Tokenizer instance for text encoding/decoding.
+ frontend: Audio frontend for feature extraction.
+ **kwargs: Additional keyword arguments.
+ &quot;&quot;&quot;
+ decoding_model = kwargs.get(&quot;decoding_model&quot;, &quot;normal&quot;)
+ token_num_relax = kwargs.get(&quot;token_num_relax&quot;, 5)
+ if decoding_model == &quot;fast&quot;:
+ decoding_ind = 0
+ decoding_mode = &quot;model1&quot;
+ elif decoding_model == &quot;offline&quot;:
+ decoding_ind = 1
+ decoding_mode = &quot;model2&quot;
+ else:
+ decoding_ind = 0
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/uniasr/model.py#L1022" target="_blank" class="src-more">View full source on GitHub →</a></details></div>
+<div class="api-detail" id="e258"><span class="dtag badge-c">class</span><h2>WhisperWarp</h2><div class="dmod">funasr.models.whisper · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/whisper/model.py#L33" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><ul>
+<li><code>Whisper</code> — OpenAI Whisper model integration.</li>
+</ul>
+<p>Wraps Whisper for multilingual speech recognition and translation</p>
+<p>within FunASR&#x27;s AutoModel interface.</p>
+<ul>
+<li><code>Supports</code> — whisper-tiny through whisper-large-v3-turbo.</li>
+<li><code>Output</code> — {&quot;key&quot;: str, &quot;text&quot;: str}</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code>class WhisperWarp(nn.Module):
+ &quot;&quot;&quot;Whisper: OpenAI Whisper model integration.
+
+ Wraps Whisper for multilingual speech recognition and translation
+ within FunASR&#x27;s AutoModel interface.
+
+ Supports: whisper-tiny through whisper-large-v3-turbo.
+ Output: {&quot;key&quot;: str, &quot;text&quot;: str}
+ &quot;&quot;&quot;
+
+ def __init__(self, *args, **kwargs):
+ &quot;&quot;&quot;Initialize WhisperWarp.
+
+ Args:
+ *args: Variable positional arguments.
+ **kwargs: Additional keyword arguments.
+ &quot;&quot;&quot;
+ super().__init__()
+ hub = kwargs.get(&quot;hub&quot;, &quot;funasr&quot;)
+ if hub == &quot;openai&quot;:
+ model_or_path = kwargs.get(&quot;model_path&quot;, &quot;Whisper-large-v3&quot;)
+ if model_or_path.startswith(&quot;Whisper-&quot;):
+ model_or_path = model_or_path.replace(&quot;Whisper-&quot;, &quot;&quot;)
+ model = whisper.load_model(model_or_path)
+ else:
+ dims = kwargs.get(&quot;dims&quot;, {})
+ dims = whisper.model.ModelDimensions(**dims)
+ model = whisper.model.Whisper(dims=dims)
+
+ self.model = model
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/whisper/model.py#L33" target="_blank" class="src-more">View full source on GitHub →</a></details><h3 class="mt">Methods</h3><div class="mblk"><code>.forward()</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/whisper/model.py#L66" target="_blank" class="gh-link">L66</a><div class="mdoc"><p>Forward pass for training.</p>
+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def forward(
+ self,
+ ):
+ &quot;&quot;&quot;Forward pass for training.&quot;&quot;&quot;
+ pass
+</code></pre></details></div><div class="mblk"><code>.inference(data_in, data_lengths, key, tokenizer, frontend, **kwargs)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/whisper/model.py#L72" target="_blank" class="gh-link">L72</a><div class="mdoc"><p>Run inference on input data.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>data_in</code> — Input data (audio samples, file paths, or text).</li>
+<li><code>data_lengths</code> — Lengths of each input sample in the batch.</li>
+<li><code>key</code> — Sample identifiers.</li>
+<li><code>tokenizer</code> — Tokenizer instance for text encoding/decoding.</li>
+<li><code>frontend</code> — Audio frontend for feature extraction.</li>
+<li><code>**kwargs</code> — Additional keyword arguments.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def inference(
+ self,
+ data_in,
+ data_lengths=None,
+ key: list = None,
+ tokenizer=None,
+ frontend=None,
+ **kwargs,
+ ):
+ &quot;&quot;&quot;Run inference on input data.
+
+ Args:
+ data_in: Input data (audio samples, file paths, or text).
+ data_lengths: Lengths of each input sample in the batch.
+ key: Sample identifiers.
+ tokenizer: Tokenizer instance for text encoding/decoding.
+ frontend: Audio frontend for feature extraction.
+ **kwargs: Additional keyword arguments.
+ &quot;&quot;&quot;
+ if kwargs.get(&quot;batch_size&quot;, 1) &gt; 1:
+ raise NotImplementedError(&quot;batch decoding is not implemented&quot;)
+
+ if frontend is None and not hasattr(self, &quot;frontend&quot;):
+ frontend_class = tables.frontend_classes.get(&quot;WhisperFrontend&quot;)
+ frontend = frontend_class(
+ n_mels=self.model.dims.n_mels, do_pad_trim=kwargs.get(&quot;do_pad_trim&quot;, True)
+ )
+ self.frontend = frontend
+ else:
+ frontend = frontend if frontend is not None else self.frontend
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/whisper/model.py#L72" target="_blank" class="src-more">View full source →</a></details></div></div>
+<div class="api-detail" id="e259"><span class="dtag badge-m">method</span><h2><code>WhisperWarp.forward()</code></h2><div class="dmod">funasr.models.whisper.WhisperWarp · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/whisper/model.py#L66" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Forward pass for training.</p>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def forward(
+ self,
+ ):
+ &quot;&quot;&quot;Forward pass for training.&quot;&quot;&quot;
+ pass
+</code></pre></details></div>
+<div class="api-detail" id="e260"><span class="dtag badge-m">method</span><h2><code>WhisperWarp.inference(data_in, data_lengths, key, tokenizer, frontend, **kwargs)</code></h2><div class="dmod">funasr.models.whisper.WhisperWarp · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/whisper/model.py#L72" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Run inference on input data.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>data_in</code> — Input data (audio samples, file paths, or text).</li>
+<li><code>data_lengths</code> — Lengths of each input sample in the batch.</li>
+<li><code>key</code> — Sample identifiers.</li>
+<li><code>tokenizer</code> — Tokenizer instance for text encoding/decoding.</li>
+<li><code>frontend</code> — Audio frontend for feature extraction.</li>
+<li><code>**kwargs</code> — Additional keyword arguments.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def inference(
+ self,
+ data_in,
+ data_lengths=None,
+ key: list = None,
+ tokenizer=None,
+ frontend=None,
+ **kwargs,
+ ):
+ &quot;&quot;&quot;Run inference on input data.
+
+ Args:
+ data_in: Input data (audio samples, file paths, or text).
+ data_lengths: Lengths of each input sample in the batch.
+ key: Sample identifiers.
+ tokenizer: Tokenizer instance for text encoding/decoding.
+ frontend: Audio frontend for feature extraction.
+ **kwargs: Additional keyword arguments.
+ &quot;&quot;&quot;
+ if kwargs.get(&quot;batch_size&quot;, 1) &gt; 1:
+ raise NotImplementedError(&quot;batch decoding is not implemented&quot;)
+
+ if frontend is None and not hasattr(self, &quot;frontend&quot;):
+ frontend_class = tables.frontend_classes.get(&quot;WhisperFrontend&quot;)
+ frontend = frontend_class(
+ n_mels=self.model.dims.n_mels, do_pad_trim=kwargs.get(&quot;do_pad_trim&quot;, True)
+ )
+ self.frontend = frontend
+ else:
+ frontend = frontend if frontend is not None else self.frontend
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/whisper/model.py#L72" target="_blank" class="src-more">View full source on GitHub →</a></details></div>
+<div class="api-detail" id="e261"><span class="dtag badge-c">class</span><h2>OpenAIWhisperModel</h2><div class="dmod">funasr.models.whisper_lid · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/whisper_lid/model.py#L22" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><ul>
+<li><code>Whisper</code> — LID: Language Identification using OpenAI Whisper encoder.</li>
+</ul>
+<p>Detects spoken language from audio input. Supports 99 languages.</p>
+<p>Uses Whisper encoder features with a classification head.</p>
+<ul>
+<li><code>Output</code> — {&quot;key&quot;: str, &quot;text&quot;: str (language code)}</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code>class OpenAIWhisperModel(nn.Module):
+ &quot;&quot;&quot;Whisper-LID: Language Identification using OpenAI Whisper encoder.
+
+ Detects spoken language from audio input. Supports 99 languages.
+ Uses Whisper encoder features with a classification head.
+
+ Output: {&quot;key&quot;: str, &quot;text&quot;: str (language code)}
+ &quot;&quot;&quot;
+
+ def __init__(
+ self,
+ specaug: str = None,
+ specaug_conf: dict = None,
+ normalize: str = None,
+ normalize_conf: dict = None,
+ encoder: str = None,
+ encoder_conf: dict = None,
+ decoder: str = None,
+ decoder_conf: dict = None,
+ ctc: str = None,
+ ctc_conf: dict = None,
+ ctc_weight: float = 0.5,
+ interctc_weight: float = 0.0,
+ input_size: int = 80,
+ vocab_size: int = -1,
+ ignore_id: int = -1,
+ blank_id: int = 0,
+ sos: int = 1,
+ eos: int = 2,
+ lsm_weight: float = 0.0,
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/whisper_lid/model.py#L22" target="_blank" class="src-more">View full source on GitHub →</a></details><h3 class="mt">Methods</h3><div class="mblk"><code>.forward(speech, speech_lengths, text, text_lengths, **kwargs)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/whisper_lid/model.py#L164" target="_blank" class="gh-link">L164</a><div class="mdoc"><p>Encoder + Decoder + Calc loss</p>
+<h4>Args:</h4>
+<ul>
+<li><code>speech</code> — (Batch, Length, ...)</li>
+<li><code>speech_lengths</code> — (Batch, )</li>
+<li><code>text</code> — (Batch, Length)</li>
+<li><code>text_lengths</code> — (Batch,)</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def forward(
+ self,
+ speech: torch.Tensor,
+ speech_lengths: torch.Tensor,
+ text: torch.Tensor,
+ text_lengths: torch.Tensor,
+ **kwargs,
+ ) -&gt; Tuple[torch.Tensor, Dict[str, torch.Tensor], torch.Tensor]:
+ &quot;&quot;&quot;Encoder + Decoder + Calc loss
+ Args:
+ speech: (Batch, Length, ...)
+ speech_lengths: (Batch, )
+ text: (Batch, Length)
+ text_lengths: (Batch,)
+ &quot;&quot;&quot;
+ if len(text_lengths.size()) &gt; 1:
+ text_lengths = text_lengths[:, 0]
+ if len(speech_lengths.size()) &gt; 1:
+ speech_lengths = speech_lengths[:, 0]
+
+ batch_size = speech.shape[0]
+
+ # 1. Encoder
+ encoder_out, encoder_out_lens = self.encode(speech, speech_lengths)
+ intermediate_outs = None
+ if isinstance(encoder_out, tuple):
+ intermediate_outs = encoder_out[1]
+ encoder_out = encoder_out[0]
+
+ loss_att, acc_att, cer_att, wer_att = None, None, None, None
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/whisper_lid/model.py#L164" target="_blank" class="src-more">View full source →</a></details></div><div class="mblk"><code>.encode(speech, speech_lengths, **kwargs)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/whisper_lid/model.py#L257" target="_blank" class="gh-link">L257</a><div class="mdoc"><p>Frontend + Encoder. Note that this method is used by asr_inference.py</p>
+<h4>Args:</h4>
+<ul>
+<li><code>speech</code> — (Batch, Length, ...)</li>
+<li><code>speech_lengths</code> — (Batch, )</li>
+<li><code>ind</code> — int</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def encode(
+ self,
+ speech: torch.Tensor,
+ speech_lengths: torch.Tensor,
+ **kwargs,
+ ) -&gt; Tuple[torch.Tensor, torch.Tensor]:
+ &quot;&quot;&quot;Frontend + Encoder. Note that this method is used by asr_inference.py
+ Args:
+ speech: (Batch, Length, ...)
+ speech_lengths: (Batch, )
+ ind: int
+ &quot;&quot;&quot;
+ with autocast(False):
+
+ # Data augmentation
+ if self.specaug is not None and self.training:
+ speech, speech_lengths = self.specaug(speech, speech_lengths)
+
+ # Normalization for feature: e.g. Global-CMVN, Utterance-CMVN
+ if self.normalize is not None:
+ speech, speech_lengths = self.normalize(speech, speech_lengths)
+
+ # Forward encoder
+ # feats: (Batch, Length, Dim)
+ # -&gt; encoder_out: (Batch, Length2, Dim2)
+ if self.encoder.interctc_use_conditioning:
+ encoder_out, encoder_out_lens, _ = self.encoder(speech, speech_lengths, ctc=self.ctc)
+ else:
+ encoder_out, encoder_out_lens, _ = self.encoder(speech, speech_lengths)
+ intermediate_outs = None
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/whisper_lid/model.py#L257" target="_blank" class="src-more">View full source →</a></details></div><div class="mblk"><code>.init_beam_search(**kwargs)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/whisper_lid/model.py#L359" target="_blank" class="gh-link">L359</a><div class="mdoc"><p>Init beam search.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>**kwargs</code> — Additional keyword arguments.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def init_beam_search(
+ self,
+ **kwargs,
+ ):
+ &quot;&quot;&quot;Init beam search.
+
+ Args:
+ **kwargs: Additional keyword arguments.
+ &quot;&quot;&quot;
+ from funasr.models.transformer.search import BeamSearch
+ from funasr.models.transformer.scorers.ctc import CTCPrefixScorer
+ from funasr.models.transformer.scorers.length_bonus import LengthBonus
+
+ # 1. Build ASR model
+ scorers = {}
+
+ if self.ctc != None:
+ ctc = CTCPrefixScorer(ctc=self.ctc, eos=self.eos)
+ scorers.update(ctc=ctc)
+ token_list = kwargs.get(&quot;token_list&quot;)
+ scorers.update(
+ decoder=self.decoder,
+ length_bonus=LengthBonus(len(token_list)),
+ )
+
+ # 3. Build ngram model
+ # ngram is not supported now
+ ngram = None
+ scorers[&quot;ngram&quot;] = ngram
+
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/whisper_lid/model.py#L359" target="_blank" class="src-more">View full source →</a></details></div><div class="mblk"><code>.inference(data_in, data_lengths, key, tokenizer, frontend, **kwargs)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/whisper_lid/model.py#L409" target="_blank" class="gh-link">L409</a><div class="mdoc"><p>Run inference on input data.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>data_in</code> — Input data (audio samples, file paths, or text).</li>
+<li><code>data_lengths</code> — Lengths of each input sample in the batch.</li>
+<li><code>key</code> — Sample identifiers.</li>
+<li><code>tokenizer</code> — Tokenizer instance for text encoding/decoding.</li>
+<li><code>frontend</code> — Audio frontend for feature extraction.</li>
+<li><code>**kwargs</code> — Additional keyword arguments.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def inference(
+ self,
+ data_in,
+ data_lengths=None,
+ key: list = None,
+ tokenizer=None,
+ frontend=None,
+ **kwargs,
+ ):
+
+ &quot;&quot;&quot;Run inference on input data.
+
+ Args:
+ data_in: Input data (audio samples, file paths, or text).
+ data_lengths: Lengths of each input sample in the batch.
+ key: Sample identifiers.
+ tokenizer: Tokenizer instance for text encoding/decoding.
+ frontend: Audio frontend for feature extraction.
+ **kwargs: Additional keyword arguments.
+ &quot;&quot;&quot;
+ if kwargs.get(&quot;batch_size&quot;, 1) &gt; 1:
+ raise NotImplementedError(&quot;batch decoding is not implemented&quot;)
+
+ # init beamsearch
+ if self.beam_search is None:
+ logging.info(&quot;enable beam_search&quot;)
+ self.init_beam_search(**kwargs)
+ self.nbest = kwargs.get(&quot;nbest&quot;, 1)
+
+ meta_data = {}
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/whisper_lid/model.py#L409" target="_blank" class="src-more">View full source →</a></details></div></div>
+<div class="api-detail" id="e262"><span class="dtag badge-m">method</span><h2><code>OpenAIWhisperModel.forward(speech, speech_lengths, text, text_lengths, **kwargs)</code></h2><div class="dmod">funasr.models.whisper_lid.OpenAIWhisperModel · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/whisper_lid/model.py#L164" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Encoder + Decoder + Calc loss</p>
+<h4>Args:</h4>
+<ul>
+<li><code>speech</code> — (Batch, Length, ...)</li>
+<li><code>speech_lengths</code> — (Batch, )</li>
+<li><code>text</code> — (Batch, Length)</li>
+<li><code>text_lengths</code> — (Batch,)</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def forward(
+ self,
+ speech: torch.Tensor,
+ speech_lengths: torch.Tensor,
+ text: torch.Tensor,
+ text_lengths: torch.Tensor,
+ **kwargs,
+ ) -&gt; Tuple[torch.Tensor, Dict[str, torch.Tensor], torch.Tensor]:
+ &quot;&quot;&quot;Encoder + Decoder + Calc loss
+ Args:
+ speech: (Batch, Length, ...)
+ speech_lengths: (Batch, )
+ text: (Batch, Length)
+ text_lengths: (Batch,)
+ &quot;&quot;&quot;
+ if len(text_lengths.size()) &gt; 1:
+ text_lengths = text_lengths[:, 0]
+ if len(speech_lengths.size()) &gt; 1:
+ speech_lengths = speech_lengths[:, 0]
+
+ batch_size = speech.shape[0]
+
+ # 1. Encoder
+ encoder_out, encoder_out_lens = self.encode(speech, speech_lengths)
+ intermediate_outs = None
+ if isinstance(encoder_out, tuple):
+ intermediate_outs = encoder_out[1]
+ encoder_out = encoder_out[0]
+
+ loss_att, acc_att, cer_att, wer_att = None, None, None, None
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/whisper_lid/model.py#L164" target="_blank" class="src-more">View full source on GitHub →</a></details></div>
+<div class="api-detail" id="e263"><span class="dtag badge-m">method</span><h2><code>OpenAIWhisperModel.encode(speech, speech_lengths, **kwargs)</code></h2><div class="dmod">funasr.models.whisper_lid.OpenAIWhisperModel · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/whisper_lid/model.py#L257" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Frontend + Encoder. Note that this method is used by asr_inference.py</p>
+<h4>Args:</h4>
+<ul>
+<li><code>speech</code> — (Batch, Length, ...)</li>
+<li><code>speech_lengths</code> — (Batch, )</li>
+<li><code>ind</code> — int</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def encode(
+ self,
+ speech: torch.Tensor,
+ speech_lengths: torch.Tensor,
+ **kwargs,
+ ) -&gt; Tuple[torch.Tensor, torch.Tensor]:
+ &quot;&quot;&quot;Frontend + Encoder. Note that this method is used by asr_inference.py
+ Args:
+ speech: (Batch, Length, ...)
+ speech_lengths: (Batch, )
+ ind: int
+ &quot;&quot;&quot;
+ with autocast(False):
+
+ # Data augmentation
+ if self.specaug is not None and self.training:
+ speech, speech_lengths = self.specaug(speech, speech_lengths)
+
+ # Normalization for feature: e.g. Global-CMVN, Utterance-CMVN
+ if self.normalize is not None:
+ speech, speech_lengths = self.normalize(speech, speech_lengths)
+
+ # Forward encoder
+ # feats: (Batch, Length, Dim)
+ # -&gt; encoder_out: (Batch, Length2, Dim2)
+ if self.encoder.interctc_use_conditioning:
+ encoder_out, encoder_out_lens, _ = self.encoder(speech, speech_lengths, ctc=self.ctc)
+ else:
+ encoder_out, encoder_out_lens, _ = self.encoder(speech, speech_lengths)
+ intermediate_outs = None
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/whisper_lid/model.py#L257" target="_blank" class="src-more">View full source on GitHub →</a></details></div>
+<div class="api-detail" id="e264"><span class="dtag badge-m">method</span><h2><code>OpenAIWhisperModel.init_beam_search(**kwargs)</code></h2><div class="dmod">funasr.models.whisper_lid.OpenAIWhisperModel · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/whisper_lid/model.py#L359" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Init beam search.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>**kwargs</code> — Additional keyword arguments.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def init_beam_search(
+ self,
+ **kwargs,
+ ):
+ &quot;&quot;&quot;Init beam search.
+
+ Args:
+ **kwargs: Additional keyword arguments.
+ &quot;&quot;&quot;
+ from funasr.models.transformer.search import BeamSearch
+ from funasr.models.transformer.scorers.ctc import CTCPrefixScorer
+ from funasr.models.transformer.scorers.length_bonus import LengthBonus
+
+ # 1. Build ASR model
+ scorers = {}
+
+ if self.ctc != None:
+ ctc = CTCPrefixScorer(ctc=self.ctc, eos=self.eos)
+ scorers.update(ctc=ctc)
+ token_list = kwargs.get(&quot;token_list&quot;)
+ scorers.update(
+ decoder=self.decoder,
+ length_bonus=LengthBonus(len(token_list)),
+ )
+
+ # 3. Build ngram model
+ # ngram is not supported now
+ ngram = None
+ scorers[&quot;ngram&quot;] = ngram
+
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/whisper_lid/model.py#L359" target="_blank" class="src-more">View full source on GitHub →</a></details></div>
+<div class="api-detail" id="e265"><span class="dtag badge-m">method</span><h2><code>OpenAIWhisperModel.inference(data_in, data_lengths, key, tokenizer, frontend, **kwargs)</code></h2><div class="dmod">funasr.models.whisper_lid.OpenAIWhisperModel · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/whisper_lid/model.py#L409" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Run inference on input data.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>data_in</code> — Input data (audio samples, file paths, or text).</li>
+<li><code>data_lengths</code> — Lengths of each input sample in the batch.</li>
+<li><code>key</code> — Sample identifiers.</li>
+<li><code>tokenizer</code> — Tokenizer instance for text encoding/decoding.</li>
+<li><code>frontend</code> — Audio frontend for feature extraction.</li>
+<li><code>**kwargs</code> — Additional keyword arguments.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def inference(
+ self,
+ data_in,
+ data_lengths=None,
+ key: list = None,
+ tokenizer=None,
+ frontend=None,
+ **kwargs,
+ ):
+
+ &quot;&quot;&quot;Run inference on input data.
+
+ Args:
+ data_in: Input data (audio samples, file paths, or text).
+ data_lengths: Lengths of each input sample in the batch.
+ key: Sample identifiers.
+ tokenizer: Tokenizer instance for text encoding/decoding.
+ frontend: Audio frontend for feature extraction.
+ **kwargs: Additional keyword arguments.
+ &quot;&quot;&quot;
+ if kwargs.get(&quot;batch_size&quot;, 1) &gt; 1:
+ raise NotImplementedError(&quot;batch decoding is not implemented&quot;)
+
+ # init beamsearch
+ if self.beam_search is None:
+ logging.info(&quot;enable beam_search&quot;)
+ self.init_beam_search(**kwargs)
+ self.nbest = kwargs.get(&quot;nbest&quot;, 1)
+
+ meta_data = {}
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/whisper_lid/model.py#L409" target="_blank" class="src-more">View full source on GitHub →</a></details></div>
+<div class="api-detail" id="e266"><span class="dtag badge-c">class</span><h2>OpenAIWhisperLIDModel</h2><div class="dmod">funasr.models.whisper_lid · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/whisper_lid/model.py#L525" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>WhisperEncoder and EResNet based LID Model</p>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code>class OpenAIWhisperLIDModel(nn.Module):
+ &quot;&quot;&quot;WhisperEncoder and EResNet based LID Model&quot;&quot;&quot;
+
+ def __init__(
+ self,
+ vocab_size: int,
+ specaug: str = None,
+ specaug_conf: dict = None,
+ encoder: str = None,
+ encoder_conf: dict = None,
+ lid_predictor: str = None,
+ lid_predictor_conf: dict = None,
+ proj_dim: int = None,
+ clip_frames: int = None,
+ random_clip: bool = False,
+ **kwargs,
+ ):
+ &quot;&quot;&quot;Initialize OpenAIWhisperLIDModel.
+
+ Args:
+ vocab_size: Size/dimension parameter.
+ specaug: TODO.
+ specaug_conf: Configuration dict for specaug.
+ encoder: TODO.
+ encoder_conf: Configuration dict for encoder.
+ lid_predictor: TODO.
+ lid_predictor_conf: Configuration dict for lid_predictor.
+ proj_dim: Size/dimension parameter.
+ clip_frames: TODO.
+ random_clip: TODO.
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/whisper_lid/model.py#L525" target="_blank" class="src-more">View full source on GitHub →</a></details><h3 class="mt">Methods</h3><div class="mblk"><code>.forward(speech, speech_lengths, lid, lid_lengths)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/whisper_lid/model.py#L587" target="_blank" class="gh-link">L587</a><div class="mdoc"><p>Forward pass for training.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>speech</code> — Speech audio tensor, shape (batch, time).</li>
+<li><code>speech_lengths</code> — Length of each speech sample.</li>
+<li><code>lid</code> — TODO.</li>
+<li><code>lid_lengths</code> — Lengths of lid.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def forward(
+ self,
+ speech: torch.Tensor, # may be padding
+ speech_lengths: torch.Tensor, # actual length
+ lid: torch.Tensor, # lid label, (batch_size, 1)
+ lid_lengths: torch.Tensor,
+ ):
+ &quot;&quot;&quot;Forward pass for training.
+
+ Args:
+ speech: Speech audio tensor, shape (batch, time).
+ speech_lengths: Length of each speech sample.
+ lid: TODO.
+ lid_lengths: Lengths of lid.
+ &quot;&quot;&quot;
+ assert lid.shape[1] == 1
+ batch_size = speech.shape[0]
+ encoder_out, encoder_out_lens = self.encode(speech, speech_lengths)
+
+ # re-generate encoder_out
+ if self.clip_frames is None:
+ reduced_encoder_out = (
+ torch.zeros(batch_size, encoder_out_lens.max(), encoder_out.shape[-1])
+ .to(encoder_out.dtype)
+ .to(encoder_out.device)
+ )
+ for i, enc_length in enumerate(encoder_out_lens):
+ reduced_encoder_out[i, :enc_length] = encoder_out[i, :enc_length]
+ else:
+ reduced_encoder_out = (
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/whisper_lid/model.py#L587" target="_blank" class="src-more">View full source →</a></details></div><div class="mblk"><code>.encode(speech, speech_lengths)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/whisper_lid/model.py#L655" target="_blank" class="gh-link">L655</a><div class="mdoc"><p>Frontend + Encoder. Note that this method is used by asr_inference.py</p>
+<h4>Args:</h4>
+<ul>
+<li><code>speech</code> — (Batch, Length, ...)</li>
+<li><code>speech_lengths</code> — (Batch, )</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def encode(
+ self, speech: torch.Tensor, speech_lengths: torch.Tensor
+ ) -&gt; Tuple[torch.Tensor, torch.Tensor]:
+ &quot;&quot;&quot;Frontend + Encoder. Note that this method is used by asr_inference.py
+ Args:
+ speech: (Batch, Length, ...)
+ speech_lengths: (Batch, )
+ &quot;&quot;&quot;
+ with autocast(False):
+
+ # Data augmentation
+ if self.specaug is not None and self.training:
+ speech = speech.permute(0, 2, 1)
+ # suit for whisper padding
+ padded_speech_lengths = torch.ones_like(speech_lengths) * speech.shape[1]
+ speech, padded_speech_lengths = self.specaug(speech, padded_speech_lengths)
+ speech = speech.permute(0, 2, 1)
+
+ # Normalization for feature: e.g. Global-CMVN, Utterance-CMVN
+ if self.normalize is not None:
+ speech, speech_lengths = self.normalize(speech, speech_lengths)
+
+ # Forward encoder
+ # feats: (Batch, Length, Dim)
+ # -&gt; encoder_out: (Batch, Length2, Dim2)
+ if self.encoder.interctc_use_conditioning:
+ encoder_out, encoder_out_lens, _ = self.encoder(speech, speech_lengths, ctc=self.ctc)
+ else:
+ encoder_out, encoder_out_lens, _ = self.encoder(speech, speech_lengths)
+ intermediate_outs = None
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/whisper_lid/model.py#L655" target="_blank" class="src-more">View full source →</a></details></div><div class="mblk"><code>.inference(data_in, data_lengths, key, tokenizer, frontend, **kwargs)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/whisper_lid/model.py#L694" target="_blank" class="gh-link">L694</a><div class="mdoc"><p>Run inference on input data.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>data_in</code> — Input data (audio samples, file paths, or text).</li>
+<li><code>data_lengths</code> — Lengths of each input sample in the batch.</li>
+<li><code>key</code> — Sample identifiers.</li>
+<li><code>tokenizer</code> — Tokenizer instance for text encoding/decoding.</li>
+<li><code>frontend</code> — Audio frontend for feature extraction.</li>
+<li><code>**kwargs</code> — Additional keyword arguments.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def inference(
+ self,
+ data_in,
+ data_lengths=None,
+ key: list = None,
+ tokenizer=None,
+ frontend=None,
+ **kwargs,
+ ):
+
+ &quot;&quot;&quot;Run inference on input data.
+
+ Args:
+ data_in: Input data (audio samples, file paths, or text).
+ data_lengths: Lengths of each input sample in the batch.
+ key: Sample identifiers.
+ tokenizer: Tokenizer instance for text encoding/decoding.
+ frontend: Audio frontend for feature extraction.
+ **kwargs: Additional keyword arguments.
+ &quot;&quot;&quot;
+ if kwargs.get(&quot;batch_size&quot;, 1) &gt; 1:
+ raise NotImplementedError(&quot;batch decoding is not implemented&quot;)
+
+ meta_data = {}
+ if (
+ isinstance(data_in, torch.Tensor) and kwargs.get(&quot;data_type&quot;, &quot;sound&quot;) == &quot;fbank&quot;
+ ): # fbank
+ speech, speech_lengths = data_in, data_lengths
+ if len(speech.shape) &lt; 3:
+ speech = speech[None, :, :]
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/whisper_lid/model.py#L694" target="_blank" class="src-more">View full source →</a></details></div></div>
+<div class="api-detail" id="e267"><span class="dtag badge-m">method</span><h2><code>OpenAIWhisperLIDModel.forward(speech, speech_lengths, lid, lid_lengths)</code></h2><div class="dmod">funasr.models.whisper_lid.OpenAIWhisperLIDModel · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/whisper_lid/model.py#L587" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Forward pass for training.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>speech</code> — Speech audio tensor, shape (batch, time).</li>
+<li><code>speech_lengths</code> — Length of each speech sample.</li>
+<li><code>lid</code> — TODO.</li>
+<li><code>lid_lengths</code> — Lengths of lid.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def forward(
+ self,
+ speech: torch.Tensor, # may be padding
+ speech_lengths: torch.Tensor, # actual length
+ lid: torch.Tensor, # lid label, (batch_size, 1)
+ lid_lengths: torch.Tensor,
+ ):
+ &quot;&quot;&quot;Forward pass for training.
+
+ Args:
+ speech: Speech audio tensor, shape (batch, time).
+ speech_lengths: Length of each speech sample.
+ lid: TODO.
+ lid_lengths: Lengths of lid.
+ &quot;&quot;&quot;
+ assert lid.shape[1] == 1
+ batch_size = speech.shape[0]
+ encoder_out, encoder_out_lens = self.encode(speech, speech_lengths)
+
+ # re-generate encoder_out
+ if self.clip_frames is None:
+ reduced_encoder_out = (
+ torch.zeros(batch_size, encoder_out_lens.max(), encoder_out.shape[-1])
+ .to(encoder_out.dtype)
+ .to(encoder_out.device)
+ )
+ for i, enc_length in enumerate(encoder_out_lens):
+ reduced_encoder_out[i, :enc_length] = encoder_out[i, :enc_length]
+ else:
+ reduced_encoder_out = (
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/whisper_lid/model.py#L587" target="_blank" class="src-more">View full source on GitHub →</a></details></div>
+<div class="api-detail" id="e268"><span class="dtag badge-m">method</span><h2><code>OpenAIWhisperLIDModel.encode(speech, speech_lengths)</code></h2><div class="dmod">funasr.models.whisper_lid.OpenAIWhisperLIDModel · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/whisper_lid/model.py#L655" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Frontend + Encoder. Note that this method is used by asr_inference.py</p>
+<h4>Args:</h4>
+<ul>
+<li><code>speech</code> — (Batch, Length, ...)</li>
+<li><code>speech_lengths</code> — (Batch, )</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def encode(
+ self, speech: torch.Tensor, speech_lengths: torch.Tensor
+ ) -&gt; Tuple[torch.Tensor, torch.Tensor]:
+ &quot;&quot;&quot;Frontend + Encoder. Note that this method is used by asr_inference.py
+ Args:
+ speech: (Batch, Length, ...)
+ speech_lengths: (Batch, )
+ &quot;&quot;&quot;
+ with autocast(False):
+
+ # Data augmentation
+ if self.specaug is not None and self.training:
+ speech = speech.permute(0, 2, 1)
+ # suit for whisper padding
+ padded_speech_lengths = torch.ones_like(speech_lengths) * speech.shape[1]
+ speech, padded_speech_lengths = self.specaug(speech, padded_speech_lengths)
+ speech = speech.permute(0, 2, 1)
+
+ # Normalization for feature: e.g. Global-CMVN, Utterance-CMVN
+ if self.normalize is not None:
+ speech, speech_lengths = self.normalize(speech, speech_lengths)
+
+ # Forward encoder
+ # feats: (Batch, Length, Dim)
+ # -&gt; encoder_out: (Batch, Length2, Dim2)
+ if self.encoder.interctc_use_conditioning:
+ encoder_out, encoder_out_lens, _ = self.encoder(speech, speech_lengths, ctc=self.ctc)
+ else:
+ encoder_out, encoder_out_lens, _ = self.encoder(speech, speech_lengths)
+ intermediate_outs = None
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/whisper_lid/model.py#L655" target="_blank" class="src-more">View full source on GitHub →</a></details></div>
+<div class="api-detail" id="e269"><span class="dtag badge-m">method</span><h2><code>OpenAIWhisperLIDModel.inference(data_in, data_lengths, key, tokenizer, frontend, **kwargs)</code></h2><div class="dmod">funasr.models.whisper_lid.OpenAIWhisperLIDModel · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/whisper_lid/model.py#L694" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Run inference on input data.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>data_in</code> — Input data (audio samples, file paths, or text).</li>
+<li><code>data_lengths</code> — Lengths of each input sample in the batch.</li>
+<li><code>key</code> — Sample identifiers.</li>
+<li><code>tokenizer</code> — Tokenizer instance for text encoding/decoding.</li>
+<li><code>frontend</code> — Audio frontend for feature extraction.</li>
+<li><code>**kwargs</code> — Additional keyword arguments.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def inference(
+ self,
+ data_in,
+ data_lengths=None,
+ key: list = None,
+ tokenizer=None,
+ frontend=None,
+ **kwargs,
+ ):
+
+ &quot;&quot;&quot;Run inference on input data.
+
+ Args:
+ data_in: Input data (audio samples, file paths, or text).
+ data_lengths: Lengths of each input sample in the batch.
+ key: Sample identifiers.
+ tokenizer: Tokenizer instance for text encoding/decoding.
+ frontend: Audio frontend for feature extraction.
+ **kwargs: Additional keyword arguments.
+ &quot;&quot;&quot;
+ if kwargs.get(&quot;batch_size&quot;, 1) &gt; 1:
+ raise NotImplementedError(&quot;batch decoding is not implemented&quot;)
+
+ meta_data = {}
+ if (
+ isinstance(data_in, torch.Tensor) and kwargs.get(&quot;data_type&quot;, &quot;sound&quot;) == &quot;fbank&quot;
+ ): # fbank
+ speech, speech_lengths = data_in, data_lengths
+ if len(speech.shape) &lt; 3:
+ speech = speech[None, :, :]
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/whisper_lid/model.py#L694" target="_blank" class="src-more">View full source on GitHub →</a></details></div>
+<div class="api-detail" id="e270"><span class="dtag badge-c">class</span><h2>DefaultFrontend</h2><div class="dmod">funasr.frontends.default · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/frontends/default.py#L24" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Conventional frontend structure for ASR.</p>
+<ul>
+<li><code>Stft</code> — &gt; WPE -&gt; MVDR-Beamformer -&gt; Power-spec -&gt; Mel-Fbank -&gt; CMVN</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code>class DefaultFrontend(nn.Module):
+ &quot;&quot;&quot;Conventional frontend structure for ASR.
+ Stft -&gt; WPE -&gt; MVDR-Beamformer -&gt; Power-spec -&gt; Mel-Fbank -&gt; CMVN
+ &quot;&quot;&quot;
+
+ def __init__(
+ self,
+ fs: int = 16000,
+ n_fft: int = 512,
+ win_length: int = None,
+ hop_length: int = 128,
+ window: Optional[str] = &quot;hann&quot;,
+ center: bool = True,
+ normalized: bool = False,
+ onesided: bool = True,
+ n_mels: int = 80,
+ fmin: int = None,
+ fmax: int = None,
+ htk: bool = False,
+ frontend_conf: Optional[dict] = None,
+ apply_stft: bool = True,
+ use_channel: int = None,
+ **kwargs,
+ ):
+ &quot;&quot;&quot;Initialize DefaultFrontend.
+
+ Args:
+ fs: TODO.
+ n_fft: TODO.
+ win_length: TODO.
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/frontends/default.py#L24" target="_blank" class="src-more">View full source on GitHub →</a></details><h3 class="mt">Methods</h3><div class="mblk"><code>.output_size()</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/frontends/default.py#L106" target="_blank" class="gh-link">L106</a><div class="mdoc"><p>Output size.</p>
+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def output_size(self) -&gt; int:
+ &quot;&quot;&quot;Output size.&quot;&quot;&quot;
+ return self.n_mels
+</code></pre></details></div><div class="mblk"><code>.forward(input, input_lengths)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/frontends/default.py#L110" target="_blank" class="gh-link">L110</a><div class="mdoc"><p>Forward pass for training.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>input</code> — Input audio/text data.</li>
+<li><code>input_lengths</code> — Lengths of input.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def forward(
+ self, input: torch.Tensor, input_lengths: Union[torch.Tensor, list]
+ ) -&gt; Tuple[torch.Tensor, torch.Tensor]:
+ &quot;&quot;&quot;Forward pass for training.
+
+ Args:
+ input: Input audio/text data.
+ input_lengths: Lengths of input.
+ &quot;&quot;&quot;
+ if isinstance(input_lengths, list):
+ input_lengths = torch.tensor(input_lengths)
+ if input.dtype == torch.float64:
+ input = input.float()
+ # 1. Domain-conversion: e.g. Stft: time -&gt; time-freq
+ if self.stft is not None:
+ input_stft, feats_lens = self._compute_stft(input, input_lengths)
+ else:
+ input_stft = ComplexTensor(input[..., 0], input[..., 1])
+ feats_lens = input_lengths
+ # 2. [Option] Speech enhancement
+ if self.frontend is not None:
+ assert isinstance(input_stft, ComplexTensor), type(input_stft)
+ # input_stft: (Batch, Length, [Channel], Freq)
+ input_stft, _, mask = self.frontend(input_stft, feats_lens)
+
+ # 3. [Multi channel case]: Select a channel
+ if input_stft.dim() == 4:
+ # h: (B, T, C, F) -&gt; h: (B, T, F)
+ if self.training:
+ if self.use_channel is not None:
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/frontends/default.py#L110" target="_blank" class="src-more">View full source →</a></details></div></div>
+<div class="api-detail" id="e271"><span class="dtag badge-m">method</span><h2><code>DefaultFrontend.output_size()</code></h2><div class="dmod">funasr.frontends.default.DefaultFrontend · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/frontends/default.py#L106" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Output size.</p>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def output_size(self) -&gt; int:
+ &quot;&quot;&quot;Output size.&quot;&quot;&quot;
+ return self.n_mels
+</code></pre></details></div>
+<div class="api-detail" id="e272"><span class="dtag badge-m">method</span><h2><code>DefaultFrontend.forward(input, input_lengths)</code></h2><div class="dmod">funasr.frontends.default.DefaultFrontend · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/frontends/default.py#L110" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Forward pass for training.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>input</code> — Input audio/text data.</li>
+<li><code>input_lengths</code> — Lengths of input.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def forward(
+ self, input: torch.Tensor, input_lengths: Union[torch.Tensor, list]
+ ) -&gt; Tuple[torch.Tensor, torch.Tensor]:
+ &quot;&quot;&quot;Forward pass for training.
+
+ Args:
+ input: Input audio/text data.
+ input_lengths: Lengths of input.
+ &quot;&quot;&quot;
+ if isinstance(input_lengths, list):
+ input_lengths = torch.tensor(input_lengths)
+ if input.dtype == torch.float64:
+ input = input.float()
+ # 1. Domain-conversion: e.g. Stft: time -&gt; time-freq
+ if self.stft is not None:
+ input_stft, feats_lens = self._compute_stft(input, input_lengths)
+ else:
+ input_stft = ComplexTensor(input[..., 0], input[..., 1])
+ feats_lens = input_lengths
+ # 2. [Option] Speech enhancement
+ if self.frontend is not None:
+ assert isinstance(input_stft, ComplexTensor), type(input_stft)
+ # input_stft: (Batch, Length, [Channel], Freq)
+ input_stft, _, mask = self.frontend(input_stft, feats_lens)
+
+ # 3. [Multi channel case]: Select a channel
+ if input_stft.dim() == 4:
+ # h: (B, T, C, F) -&gt; h: (B, T, F)
+ if self.training:
+ if self.use_channel is not None:
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/frontends/default.py#L110" target="_blank" class="src-more">View full source on GitHub →</a></details></div>
+<div class="api-detail" id="e273"><span class="dtag badge-c">class</span><h2>MultiChannelFrontend</h2><div class="dmod">funasr.frontends.default · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/frontends/default.py#L179" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Conventional frontend structure for ASR.</p>
+<ul>
+<li><code>Stft</code> — &gt; WPE -&gt; MVDR-Beamformer -&gt; Power-spec -&gt; Mel-Fbank -&gt; CMVN</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code>class MultiChannelFrontend(nn.Module):
+ &quot;&quot;&quot;Conventional frontend structure for ASR.
+ Stft -&gt; WPE -&gt; MVDR-Beamformer -&gt; Power-spec -&gt; Mel-Fbank -&gt; CMVN
+ &quot;&quot;&quot;
+
+ def __init__(
+ self,
+ fs: int = 16000,
+ n_fft: int = 512,
+ win_length: int = None,
+ hop_length: int = None,
+ frame_length: int = None,
+ frame_shift: int = None,
+ window: Optional[str] = &quot;hann&quot;,
+ center: bool = True,
+ normalized: bool = False,
+ onesided: bool = True,
+ n_mels: int = 80,
+ fmin: int = None,
+ fmax: int = None,
+ htk: bool = False,
+ frontend_conf: Optional[dict] = None,
+ apply_stft: bool = True,
+ use_channel: int = None,
+ lfr_m: int = 1,
+ lfr_n: int = 1,
+ cmvn_file: str = None,
+ mc: bool = True,
+ ):
+ &quot;&quot;&quot;Initialize MultiChannelFrontend.
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/frontends/default.py#L179" target="_blank" class="src-more">View full source on GitHub →</a></details><h3 class="mt">Methods</h3><div class="mblk"><code>.output_size()</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/frontends/default.py#L290" target="_blank" class="gh-link">L290</a><div class="mdoc"><p>Output size.</p>
+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def output_size(self) -&gt; int:
+ &quot;&quot;&quot;Output size.&quot;&quot;&quot;
+ return self.n_mels
+</code></pre></details></div><div class="mblk"><code>.forward(input, input_lengths)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/frontends/default.py#L294" target="_blank" class="gh-link">L294</a><div class="mdoc"><p>Forward pass for training.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>input</code> — Input audio/text data.</li>
+<li><code>input_lengths</code> — Lengths of input.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def forward(
+ self, input: torch.Tensor, input_lengths: torch.Tensor
+ ) -&gt; Tuple[torch.Tensor, torch.Tensor]:
+ # 1. Domain-conversion: e.g. Stft: time -&gt; time-freq
+ &quot;&quot;&quot;Forward pass for training.
+
+ Args:
+ input: Input audio/text data.
+ input_lengths: Lengths of input.
+ &quot;&quot;&quot;
+ if self.stft is not None:
+ input_stft, feats_lens = self._compute_stft(input, input_lengths)
+ else:
+ input_stft = ComplexTensor(input[..., 0], input[..., 1])
+ feats_lens = input_lengths
+ # 2. [Option] Speech enhancement
+ if self.frontend is not None:
+ assert isinstance(input_stft, ComplexTensor), type(input_stft)
+ # input_stft: (Batch, Length, [Channel], Freq)
+ input_stft, _, mask = self.frontend(input_stft, feats_lens)
+
+ # 3. [Multi channel case]: Select a channel(sa_asr)
+ if input_stft.dim() == 4 and not self.mc:
+ # h: (B, T, C, F) -&gt; h: (B, T, F)
+ if self.training:
+ if self.use_channel is not None:
+ input_stft = input_stft[:, :, self.use_channel, :]
+
+ else:
+ # Select 1ch randomly
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/frontends/default.py#L294" target="_blank" class="src-more">View full source →</a></details></div></div>
+<div class="api-detail" id="e274"><span class="dtag badge-m">method</span><h2><code>MultiChannelFrontend.output_size()</code></h2><div class="dmod">funasr.frontends.default.MultiChannelFrontend · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/frontends/default.py#L290" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Output size.</p>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def output_size(self) -&gt; int:
+ &quot;&quot;&quot;Output size.&quot;&quot;&quot;
+ return self.n_mels
+</code></pre></details></div>
+<div class="api-detail" id="e275"><span class="dtag badge-m">method</span><h2><code>MultiChannelFrontend.forward(input, input_lengths)</code></h2><div class="dmod">funasr.frontends.default.MultiChannelFrontend · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/frontends/default.py#L294" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Forward pass for training.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>input</code> — Input audio/text data.</li>
+<li><code>input_lengths</code> — Lengths of input.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def forward(
+ self, input: torch.Tensor, input_lengths: torch.Tensor
+ ) -&gt; Tuple[torch.Tensor, torch.Tensor]:
+ # 1. Domain-conversion: e.g. Stft: time -&gt; time-freq
+ &quot;&quot;&quot;Forward pass for training.
+
+ Args:
+ input: Input audio/text data.
+ input_lengths: Lengths of input.
+ &quot;&quot;&quot;
+ if self.stft is not None:
+ input_stft, feats_lens = self._compute_stft(input, input_lengths)
+ else:
+ input_stft = ComplexTensor(input[..., 0], input[..., 1])
+ feats_lens = input_lengths
+ # 2. [Option] Speech enhancement
+ if self.frontend is not None:
+ assert isinstance(input_stft, ComplexTensor), type(input_stft)
+ # input_stft: (Batch, Length, [Channel], Freq)
+ input_stft, _, mask = self.frontend(input_stft, feats_lens)
+
+ # 3. [Multi channel case]: Select a channel(sa_asr)
+ if input_stft.dim() == 4 and not self.mc:
+ # h: (B, T, C, F) -&gt; h: (B, T, F)
+ if self.training:
+ if self.use_channel is not None:
+ input_stft = input_stft[:, :, self.use_channel, :]
+
+ else:
+ # Select 1ch randomly
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/frontends/default.py#L294" target="_blank" class="src-more">View full source on GitHub →</a></details></div>
+<div class="api-detail" id="e276"><span class="dtag badge-f">function</span><h2><code>transform(Y, dtype)</code></h2><div class="dmod">funasr.frontends.eend_ola_feature · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/frontends/eend_ola_feature.py#L10" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Transform.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>Y</code> — TODO.</li>
+<li><code>dtype</code> — TODO.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code>def transform(Y, dtype=np.float32):
+ &quot;&quot;&quot;Transform.
+
+ Args:
+ Y: TODO.
+ dtype: TODO.
+ &quot;&quot;&quot;
+ Y = np.abs(Y)
+ n_fft = 2 * (Y.shape[1] - 1)
+ sr = 8000
+ n_mels = 23
+ mel_basis = librosa.filters.mel(sr, n_fft, n_mels)
+ Y = np.dot(Y**2, mel_basis.T)
+ Y = np.log10(np.maximum(Y, 1e-10))
+ mean = np.mean(Y, axis=0)
+ Y = Y - mean
+ return Y.astype(dtype)
+</code></pre></details></div>
+<div class="api-detail" id="e277"><span class="dtag badge-f">function</span><h2><code>subsample(Y, T, subsampling)</code></h2><div class="dmod">funasr.frontends.eend_ola_feature · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/frontends/eend_ola_feature.py#L29" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Subsample.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>Y</code> — TODO.</li>
+<li><code>T</code> — TODO.</li>
+<li><code>subsampling</code> — TODO.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code>def subsample(Y, T, subsampling=1):
+ &quot;&quot;&quot;Subsample.
+
+ Args:
+ Y: TODO.
+ T: TODO.
+ subsampling: TODO.
+ &quot;&quot;&quot;
+ Y_ss = Y[::subsampling]
+ T_ss = T[::subsampling]
+ return Y_ss, T_ss
+</code></pre></details></div>
+<div class="api-detail" id="e278"><span class="dtag badge-f">function</span><h2><code>splice(Y, context_size)</code></h2><div class="dmod">funasr.frontends.eend_ola_feature · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/frontends/eend_ola_feature.py#L42" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Splice.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>Y</code> — TODO.</li>
+<li><code>context_size</code> — Size/dimension parameter.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code>def splice(Y, context_size=0):
+ &quot;&quot;&quot;Splice.
+
+ Args:
+ Y: TODO.
+ context_size: Size/dimension parameter.
+ &quot;&quot;&quot;
+ Y_pad = np.pad(Y, [(context_size, context_size), (0, 0)], &quot;constant&quot;)
+ Y_spliced = np.lib.stride_tricks.as_strided(
+ np.ascontiguousarray(Y_pad),
+ (Y.shape[0], Y.shape[1] * (2 * context_size + 1)),
+ (Y.itemsize * Y.shape[1], Y.itemsize),
+ writeable=False,
+ )
+ return Y_spliced
+</code></pre></details></div>
+<div class="api-detail" id="e279"><span class="dtag badge-f">function</span><h2><code>stft(data, frame_size, frame_shift)</code></h2><div class="dmod">funasr.frontends.eend_ola_feature · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/frontends/eend_ola_feature.py#L59" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Stft.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>data</code> — TODO.</li>
+<li><code>frame_size</code> — Size/dimension parameter.</li>
+<li><code>frame_shift</code> — TODO.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code>def stft(data, frame_size=1024, frame_shift=256):
+ &quot;&quot;&quot;Stft.
+
+ Args:
+ data: TODO.
+ frame_size: Size/dimension parameter.
+ frame_shift: TODO.
+ &quot;&quot;&quot;
+ fft_size = 1 &lt;&lt; (frame_size - 1).bit_length()
+ if len(data) % frame_shift == 0:
+ return librosa.stft(data, n_fft=fft_size, win_length=frame_size, hop_length=frame_shift).T[
+ :-1
+ ]
+ else:
+ return librosa.stft(data, n_fft=fft_size, win_length=frame_size, hop_length=frame_shift).T
+</code></pre></details></div>
+<div class="api-detail" id="e280"><span class="dtag badge-c">class</span><h2>FusedFrontends</h2><div class="dmod">funasr.frontends.fused · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/frontends/fused.py#L9" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p class="muted">No documentation yet.</p></div><details class="src-block"><summary>📄 Source code</summary><pre><code>class FusedFrontends(nn.Module):
+ def __init__(self, frontends=None, align_method=&quot;linear_projection&quot;, proj_dim=100, fs=16000):
+
+ &quot;&quot;&quot;Initialize FusedFrontends.
+
+ Args:
+ frontends: TODO.
+ align_method: TODO.
+ proj_dim: Size/dimension parameter.
+ fs: TODO.
+ &quot;&quot;&quot;
+ super().__init__()
+ self.align_method = align_method # fusing method : linear_projection only for now
+ self.proj_dim = proj_dim # dim of the projection done on each frontend
+ self.frontends = [] # list of the frontends to combine
+
+ for i, frontend in enumerate(frontends):
+ frontend_type = frontend[&quot;frontend_type&quot;]
+ if frontend_type == &quot;default&quot;:
+ n_mels, fs, n_fft, win_length, hop_length = (
+ frontend.get(&quot;n_mels&quot;, 80),
+ fs,
+ frontend.get(&quot;n_fft&quot;, 512),
+ frontend.get(&quot;win_length&quot;),
+ frontend.get(&quot;hop_length&quot;, 128),
+ )
+ window, center, normalized, onesided = (
+ frontend.get(&quot;window&quot;, &quot;hann&quot;),
+ frontend.get(&quot;center&quot;, True),
+ frontend.get(&quot;normalized&quot;, False),
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/frontends/fused.py#L9" target="_blank" class="src-more">View full source on GitHub →</a></details><h3 class="mt">Methods</h3><div class="mblk"><code>.output_size()</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/frontends/fused.py#L106" target="_blank" class="gh-link">L106</a><div class="mdoc"><p>Output size.</p>
+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def output_size(self) -&gt; int:
+ &quot;&quot;&quot;Output size.&quot;&quot;&quot;
+ return len(self.frontends) * self.proj_dim
+</code></pre></details></div><div class="mblk"><code>.forward(input, input_lengths)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/frontends/fused.py#L110" target="_blank" class="gh-link">L110</a><div class="mdoc"><p>Forward pass for training.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>input</code> — Input audio/text data.</li>
+<li><code>input_lengths</code> — Lengths of input.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def forward(
+ self, input: torch.Tensor, input_lengths: torch.Tensor
+ ) -&gt; Tuple[torch.Tensor, torch.Tensor]:
+
+ # step 0 : get all frontends features
+ &quot;&quot;&quot;Forward pass for training.
+
+ Args:
+ input: Input audio/text data.
+ input_lengths: Lengths of input.
+ &quot;&quot;&quot;
+ self.feats = []
+ for frontend in self.frontends:
+ with torch.no_grad():
+ input_feats, feats_lens = frontend.forward(input, input_lengths)
+ self.feats.append([input_feats, feats_lens])
+
+ if self.align_method == &quot;linear_projection&quot;: # TODO(Dan): to add other align methods
+
+ # first step : projections
+ self.feats_proj = []
+ for i, frontend in enumerate(self.frontends):
+ input_feats = self.feats[i][0]
+ self.feats_proj.append(self.projection_layers[i](input_feats))
+
+ # 2nd step : reshape
+ self.feats_reshaped = []
+ for i, frontend in enumerate(self.frontends):
+ input_feats_proj = self.feats_proj[i]
+ bs, nf, dim = input_feats_proj.shape
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/frontends/fused.py#L110" target="_blank" class="src-more">View full source →</a></details></div></div>
+<div class="api-detail" id="e281"><span class="dtag badge-m">method</span><h2><code>FusedFrontends.output_size()</code></h2><div class="dmod">funasr.frontends.fused.FusedFrontends · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/frontends/fused.py#L106" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Output size.</p>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def output_size(self) -&gt; int:
+ &quot;&quot;&quot;Output size.&quot;&quot;&quot;
+ return len(self.frontends) * self.proj_dim
+</code></pre></details></div>
+<div class="api-detail" id="e282"><span class="dtag badge-m">method</span><h2><code>FusedFrontends.forward(input, input_lengths)</code></h2><div class="dmod">funasr.frontends.fused.FusedFrontends · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/frontends/fused.py#L110" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Forward pass for training.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>input</code> — Input audio/text data.</li>
+<li><code>input_lengths</code> — Lengths of input.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def forward(
+ self, input: torch.Tensor, input_lengths: torch.Tensor
+ ) -&gt; Tuple[torch.Tensor, torch.Tensor]:
+
+ # step 0 : get all frontends features
+ &quot;&quot;&quot;Forward pass for training.
+
+ Args:
+ input: Input audio/text data.
+ input_lengths: Lengths of input.
+ &quot;&quot;&quot;
+ self.feats = []
+ for frontend in self.frontends:
+ with torch.no_grad():
+ input_feats, feats_lens = frontend.forward(input, input_lengths)
+ self.feats.append([input_feats, feats_lens])
+
+ if self.align_method == &quot;linear_projection&quot;: # TODO(Dan): to add other align methods
+
+ # first step : projections
+ self.feats_proj = []
+ for i, frontend in enumerate(self.frontends):
+ input_feats = self.feats[i][0]
+ self.feats_proj.append(self.projection_layers[i](input_feats))
+
+ # 2nd step : reshape
+ self.feats_reshaped = []
+ for i, frontend in enumerate(self.frontends):
+ input_feats_proj = self.feats_proj[i]
+ bs, nf, dim = input_feats_proj.shape
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/frontends/fused.py#L110" target="_blank" class="src-more">View full source on GitHub →</a></details></div>
+<div class="api-detail" id="e283"><span class="dtag badge-f">function</span><h2><code>base_s3prl_setup(args)</code></h2><div class="dmod">funasr.frontends.s3prl · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/frontends/s3prl.py#L17" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Base s3prl setup.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>args</code> — TODO.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code>def base_s3prl_setup(args):
+ &quot;&quot;&quot;Base s3prl setup.
+
+ Args:
+ args: TODO.
+ &quot;&quot;&quot;
+ args.upstream_feature_selection = getattr(args, &quot;upstream_feature_selection&quot;, None)
+ args.upstream_model_config = getattr(args, &quot;upstream_model_config&quot;, None)
+ args.upstream_refresh = getattr(args, &quot;upstream_refresh&quot;, False)
+ args.upstream_ckpt = getattr(args, &quot;upstream_ckpt&quot;, None)
+ args.init_ckpt = getattr(args, &quot;init_ckpt&quot;, None)
+ args.verbose = getattr(args, &quot;verbose&quot;, False)
+ args.tile_factor = getattr(args, &quot;tile_factor&quot;, 1)
+ return args
+</code></pre></details></div>
+<div class="api-detail" id="e284"><span class="dtag badge-c">class</span><h2>S3prlFrontend</h2><div class="dmod">funasr.frontends.s3prl · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/frontends/s3prl.py#L33" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Speech Pretrained Representation frontend structure for ASR.</p>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code>class S3prlFrontend(nn.Module):
+ &quot;&quot;&quot;Speech Pretrained Representation frontend structure for ASR.&quot;&quot;&quot;
+
+ def __init__(
+ self,
+ fs: Union[int, str] = 16000,
+ frontend_conf: Optional[dict] = None,
+ download_dir: str = None,
+ multilayer_feature: bool = False,
+ ):
+ &quot;&quot;&quot;Initialize S3prlFrontend.
+
+ Args:
+ fs: TODO.
+ frontend_conf: Configuration dict for frontend.
+ download_dir: TODO.
+ multilayer_feature: TODO.
+ &quot;&quot;&quot;
+ super().__init__()
+ if isinstance(fs, str):
+ fs = humanfriendly.parse_size(fs)
+
+ if download_dir is not None:
+ torch.hub.set_dir(download_dir)
+
+ self.multilayer_feature = multilayer_feature
+ self.upstream, self.featurizer = self._get_upstream(frontend_conf)
+ self.pretrained_params = copy.deepcopy(self.upstream.state_dict())
+ self.output_dim = self.featurizer.output_dim
+ self.frontend_type = &quot;s3prl&quot;
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/frontends/s3prl.py#L33" target="_blank" class="src-more">View full source on GitHub →</a></details><h3 class="mt">Methods</h3><div class="mblk"><code>.output_size()</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/frontends/s3prl.py#L127" target="_blank" class="gh-link">L127</a><div class="mdoc"><p>Output size.</p>
+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def output_size(self) -&gt; int:
+ &quot;&quot;&quot;Output size.&quot;&quot;&quot;
+ return self.output_dim
+</code></pre></details></div><div class="mblk"><code>.forward(input, input_lengths)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/frontends/s3prl.py#L131" target="_blank" class="gh-link">L131</a><div class="mdoc"><p>Forward pass for training.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>input</code> — Input audio/text data.</li>
+<li><code>input_lengths</code> — Lengths of input.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def forward(
+ self, input: torch.Tensor, input_lengths: torch.Tensor
+ ) -&gt; Tuple[torch.Tensor, torch.Tensor]:
+ &quot;&quot;&quot;Forward pass for training.
+
+ Args:
+ input: Input audio/text data.
+ input_lengths: Lengths of input.
+ &quot;&quot;&quot;
+ wavs = [wav[: input_lengths[i]] for i, wav in enumerate(input)]
+ self.upstream.eval()
+ with torch.no_grad():
+ feats = self.upstream(wavs)
+ feats = self.featurizer(wavs, feats)
+
+ if self.args.tile_factor != 1:
+ feats = self._tile_representations(feats)
+
+ input_feats = pad_list(feats, 0.0)
+ feats_lens = torch.tensor([f.shape[0] for f in feats], dtype=torch.long)
+
+ # Saving CUDA Memory
+ del feats
+
+ return input_feats, feats_lens
+</code></pre></details></div><div class="mblk"><code>.reload_pretrained_parameters()</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/frontends/s3prl.py#L157" target="_blank" class="gh-link">L157</a><div class="mdoc"><p>Reload pretrained parameters.</p>
+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def reload_pretrained_parameters(self):
+ &quot;&quot;&quot;Reload pretrained parameters.&quot;&quot;&quot;
+ self.upstream.load_state_dict(self.pretrained_params)
+ logging.info(&quot;Pretrained S3PRL frontend model parameters reloaded!&quot;)
+</code></pre></details></div></div>
+<div class="api-detail" id="e285"><span class="dtag badge-m">method</span><h2><code>S3prlFrontend.output_size()</code></h2><div class="dmod">funasr.frontends.s3prl.S3prlFrontend · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/frontends/s3prl.py#L127" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Output size.</p>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def output_size(self) -&gt; int:
+ &quot;&quot;&quot;Output size.&quot;&quot;&quot;
+ return self.output_dim
+</code></pre></details></div>
+<div class="api-detail" id="e286"><span class="dtag badge-m">method</span><h2><code>S3prlFrontend.forward(input, input_lengths)</code></h2><div class="dmod">funasr.frontends.s3prl.S3prlFrontend · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/frontends/s3prl.py#L131" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Forward pass for training.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>input</code> — Input audio/text data.</li>
+<li><code>input_lengths</code> — Lengths of input.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def forward(
+ self, input: torch.Tensor, input_lengths: torch.Tensor
+ ) -&gt; Tuple[torch.Tensor, torch.Tensor]:
+ &quot;&quot;&quot;Forward pass for training.
+
+ Args:
+ input: Input audio/text data.
+ input_lengths: Lengths of input.
+ &quot;&quot;&quot;
+ wavs = [wav[: input_lengths[i]] for i, wav in enumerate(input)]
+ self.upstream.eval()
+ with torch.no_grad():
+ feats = self.upstream(wavs)
+ feats = self.featurizer(wavs, feats)
+
+ if self.args.tile_factor != 1:
+ feats = self._tile_representations(feats)
+
+ input_feats = pad_list(feats, 0.0)
+ feats_lens = torch.tensor([f.shape[0] for f in feats], dtype=torch.long)
+
+ # Saving CUDA Memory
+ del feats
+
+ return input_feats, feats_lens
+</code></pre></details></div>
+<div class="api-detail" id="e287"><span class="dtag badge-m">method</span><h2><code>S3prlFrontend.reload_pretrained_parameters()</code></h2><div class="dmod">funasr.frontends.s3prl.S3prlFrontend · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/frontends/s3prl.py#L157" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Reload pretrained parameters.</p>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def reload_pretrained_parameters(self):
+ &quot;&quot;&quot;Reload pretrained parameters.&quot;&quot;&quot;
+ self.upstream.load_state_dict(self.pretrained_params)
+ logging.info(&quot;Pretrained S3PRL frontend model parameters reloaded!&quot;)
+</code></pre></details></div>
+<div class="api-detail" id="e288"><span class="dtag badge-f">function</span><h2><code>load_cmvn(cmvn_file)</code></h2><div class="dmod">funasr.frontends.wav_frontend · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/frontends/wav_frontend.py#L15" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Load cmvn.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>cmvn_file</code> — TODO.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code>def load_cmvn(cmvn_file):
+ &quot;&quot;&quot;Load cmvn.
+
+ Args:
+ cmvn_file: TODO.
+ &quot;&quot;&quot;
+ with open(cmvn_file, &quot;r&quot;, encoding=&quot;utf-8&quot;) as f:
+ lines = f.readlines()
+ means_list = []
+ vars_list = []
+ for i in range(len(lines)):
+ line_item = lines[i].split()
+ if line_item[0] == &quot;&lt;AddShift&gt;&quot;:
+ line_item = lines[i + 1].split()
+ if line_item[0] == &quot;&lt;LearnRateCoef&gt;&quot;:
+ add_shift_line = line_item[3 : (len(line_item) - 1)]
+ means_list = list(add_shift_line)
+ continue
+ elif line_item[0] == &quot;&lt;Rescale&gt;&quot;:
+ line_item = lines[i + 1].split()
+ if line_item[0] == &quot;&lt;LearnRateCoef&gt;&quot;:
+ rescale_line = line_item[3 : (len(line_item) - 1)]
+ vars_list = list(rescale_line)
+ continue
+ means = np.array(means_list).astype(np.float32)
+ vars = np.array(vars_list).astype(np.float32)
+ cmvn = np.array([means, vars])
+ cmvn = torch.as_tensor(cmvn, dtype=torch.float32)
+ return cmvn
+</code></pre></details></div>
+<div class="api-detail" id="e289"><span class="dtag badge-f">function</span><h2><code>apply_cmvn(inputs, cmvn)</code></h2><div class="dmod">funasr.frontends.wav_frontend · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/frontends/wav_frontend.py#L46" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Apply CMVN with mvn data</p>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code>def apply_cmvn(inputs, cmvn): # noqa
+ &quot;&quot;&quot;
+ Apply CMVN with mvn data
+ &quot;&quot;&quot;
+
+ device = inputs.device
+ dtype = inputs.dtype
+ frame, dim = inputs.shape
+
+ means = cmvn[0:1, :dim]
+ vars = cmvn[1:2, :dim]
+ inputs += means.to(device)
+ inputs *= vars.to(device)
+
+ return inputs.type(torch.float32)
+</code></pre></details></div>
+<div class="api-detail" id="e290"><span class="dtag badge-f">function</span><h2><code>apply_lfr(inputs, lfr_m, lfr_n)</code></h2><div class="dmod">funasr.frontends.wav_frontend · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/frontends/wav_frontend.py#L63" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Apply lfr.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>inputs</code> — TODO.</li>
+<li><code>lfr_m</code> — TODO.</li>
+<li><code>lfr_n</code> — TODO.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code>def apply_lfr(inputs, lfr_m, lfr_n):
+ &quot;&quot;&quot;Apply lfr.
+
+ Args:
+ inputs: TODO.
+ lfr_m: TODO.
+ lfr_n: TODO.
+ &quot;&quot;&quot;
+ LFR_inputs = []
+ T = inputs.shape[0]
+ T_lfr = int(np.ceil(T / lfr_n))
+ left_padding = inputs[0].repeat((lfr_m - 1) // 2, 1)
+ inputs = torch.vstack((left_padding, inputs))
+ T = T + (lfr_m - 1) // 2
+ feat_dim = inputs.shape[-1]
+ strides = (lfr_n * feat_dim, 1)
+ sizes = (T_lfr, lfr_m * feat_dim)
+ last_idx = (T - lfr_m) // lfr_n + 1
+ num_padding = lfr_m - (T - last_idx * lfr_n)
+ if num_padding &gt; 0:
+ num_padding = (2 * lfr_m - 2 * T + (T_lfr - 1 + last_idx) * lfr_n) / 2 * (T_lfr - last_idx)
+ inputs = torch.vstack([inputs] + [inputs[-1:]] * int(num_padding))
+ LFR_outputs = inputs.as_strided(sizes, strides)
+ return LFR_outputs.clone().type(torch.float32)
+</code></pre></details></div>
+<div class="api-detail" id="e291"><span class="dtag badge-c">class</span><h2>WavFrontend</h2><div class="dmod">funasr.frontends.wav_frontend · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/frontends/wav_frontend.py#L91" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Conventional frontend structure for ASR.</p>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code>class WavFrontend(nn.Module):
+ &quot;&quot;&quot;Conventional frontend structure for ASR.&quot;&quot;&quot;
+
+ def __init__(
+ self,
+ cmvn_file: str = None,
+ fs: int = 16000,
+ window: str = &quot;hamming&quot;,
+ n_mels: int = 80,
+ frame_length: int = 25,
+ frame_shift: int = 10,
+ filter_length_min: int = -1,
+ filter_length_max: int = -1,
+ lfr_m: int = 1,
+ lfr_n: int = 1,
+ dither: float = 1.0,
+ snip_edges: bool = True,
+ upsacle_samples: bool = True,
+ **kwargs,
+ ):
+ &quot;&quot;&quot;Initialize WavFrontend.
+
+ Args:
+ cmvn_file: TODO.
+ fs: TODO.
+ window: TODO.
+ n_mels: TODO.
+ frame_length: TODO.
+ frame_shift: TODO.
+ filter_length_min: TODO.
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/frontends/wav_frontend.py#L91" target="_blank" class="src-more">View full source on GitHub →</a></details><h3 class="mt">Methods</h3><div class="mblk"><code>.output_size()</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/frontends/wav_frontend.py#L145" target="_blank" class="gh-link">L145</a><div class="mdoc"><p>Output size.</p>
+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def output_size(self) -&gt; int:
+ &quot;&quot;&quot;Output size.&quot;&quot;&quot;
+ return self.n_mels * self.lfr_m
+</code></pre></details></div><div class="mblk"><code>.forward(input, input_lengths, **kwargs)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/frontends/wav_frontend.py#L149" target="_blank" class="gh-link">L149</a><div class="mdoc"><p>Forward pass for training.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>input</code> — Input audio/text data.</li>
+<li><code>input_lengths</code> — Lengths of input.</li>
+<li><code>**kwargs</code> — Additional keyword arguments.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def forward(
+ self,
+ input: torch.Tensor,
+ input_lengths,
+ **kwargs,
+ ) -&gt; Tuple[torch.Tensor, torch.Tensor]:
+ &quot;&quot;&quot;Forward pass for training.
+
+ Args:
+ input: Input audio/text data.
+ input_lengths: Lengths of input.
+ **kwargs: Additional keyword arguments.
+ &quot;&quot;&quot;
+ batch_size = input.size(0)
+ feats = []
+ feats_lens = []
+ for i in range(batch_size):
+ waveform_length = input_lengths[i]
+ waveform = input[i][:waveform_length]
+ if self.upsacle_samples:
+ waveform = waveform * (1 &lt;&lt; 15)
+ waveform = waveform.unsqueeze(0)
+ mat = kaldi.fbank(
+ waveform,
+ num_mel_bins=self.n_mels,
+ frame_length=min(self.frame_length,waveform_length/self.fs*1000),
+ frame_shift=self.frame_shift,
+ dither=self.dither,
+ energy_floor=0.0,
+ window_type=self.window,
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/frontends/wav_frontend.py#L149" target="_blank" class="src-more">View full source →</a></details></div><div class="mblk"><code>.forward_fbank(input, input_lengths)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/frontends/wav_frontend.py#L198" target="_blank" class="gh-link">L198</a><div class="mdoc"><p>Forward fbank.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>input</code> — Input audio/text data.</li>
+<li><code>input_lengths</code> — Lengths of input.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def forward_fbank(
+ self, input: torch.Tensor, input_lengths: torch.Tensor
+ ) -&gt; Tuple[torch.Tensor, torch.Tensor]:
+ &quot;&quot;&quot;Forward fbank.
+
+ Args:
+ input: Input audio/text data.
+ input_lengths: Lengths of input.
+ &quot;&quot;&quot;
+ batch_size = input.size(0)
+ feats = []
+ feats_lens = []
+ for i in range(batch_size):
+ waveform_length = input_lengths[i]
+ waveform = input[i][:waveform_length]
+ waveform = waveform * (1 &lt;&lt; 15)
+ waveform = waveform.unsqueeze(0)
+ mat = kaldi.fbank(
+ waveform,
+ num_mel_bins=self.n_mels,
+ frame_length=self.frame_length,
+ frame_shift=self.frame_shift,
+ dither=self.dither,
+ energy_floor=0.0,
+ window_type=self.window,
+ sample_frequency=self.fs,
+ )
+
+ feat_length = mat.size(0)
+ feats.append(mat)
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/frontends/wav_frontend.py#L198" target="_blank" class="src-more">View full source →</a></details></div><div class="mblk"><code>.forward_lfr_cmvn(input, input_lengths)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/frontends/wav_frontend.py#L234" target="_blank" class="gh-link">L234</a><div class="mdoc"><p>Forward lfr cmvn.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>input</code> — Input audio/text data.</li>
+<li><code>input_lengths</code> — Lengths of input.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def forward_lfr_cmvn(
+ self, input: torch.Tensor, input_lengths: torch.Tensor
+ ) -&gt; Tuple[torch.Tensor, torch.Tensor]:
+ &quot;&quot;&quot;Forward lfr cmvn.
+
+ Args:
+ input: Input audio/text data.
+ input_lengths: Lengths of input.
+ &quot;&quot;&quot;
+ batch_size = input.size(0)
+ feats = []
+ feats_lens = []
+ for i in range(batch_size):
+ mat = input[i, : input_lengths[i], :]
+ if self.lfr_m != 1 or self.lfr_n != 1:
+ mat = apply_lfr(mat, self.lfr_m, self.lfr_n)
+ if self.cmvn is not None:
+ mat = apply_cmvn(mat, self.cmvn)
+ feat_length = mat.size(0)
+ feats.append(mat)
+ feats_lens.append(feat_length)
+
+ feats_lens = torch.as_tensor(feats_lens)
+ feats_pad = pad_sequence(feats, batch_first=True, padding_value=0.0)
+ return feats_pad, feats_lens
+</code></pre></details></div></div>
+<div class="api-detail" id="e292"><span class="dtag badge-m">method</span><h2><code>WavFrontend.output_size()</code></h2><div class="dmod">funasr.frontends.wav_frontend.WavFrontend · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/frontends/wav_frontend.py#L145" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Output size.</p>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def output_size(self) -&gt; int:
+ &quot;&quot;&quot;Output size.&quot;&quot;&quot;
+ return self.n_mels * self.lfr_m
+</code></pre></details></div>
+<div class="api-detail" id="e293"><span class="dtag badge-m">method</span><h2><code>WavFrontend.forward(input, input_lengths, **kwargs)</code></h2><div class="dmod">funasr.frontends.wav_frontend.WavFrontend · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/frontends/wav_frontend.py#L149" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Forward pass for training.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>input</code> — Input audio/text data.</li>
+<li><code>input_lengths</code> — Lengths of input.</li>
+<li><code>**kwargs</code> — Additional keyword arguments.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def forward(
+ self,
+ input: torch.Tensor,
+ input_lengths,
+ **kwargs,
+ ) -&gt; Tuple[torch.Tensor, torch.Tensor]:
+ &quot;&quot;&quot;Forward pass for training.
+
+ Args:
+ input: Input audio/text data.
+ input_lengths: Lengths of input.
+ **kwargs: Additional keyword arguments.
+ &quot;&quot;&quot;
+ batch_size = input.size(0)
+ feats = []
+ feats_lens = []
+ for i in range(batch_size):
+ waveform_length = input_lengths[i]
+ waveform = input[i][:waveform_length]
+ if self.upsacle_samples:
+ waveform = waveform * (1 &lt;&lt; 15)
+ waveform = waveform.unsqueeze(0)
+ mat = kaldi.fbank(
+ waveform,
+ num_mel_bins=self.n_mels,
+ frame_length=min(self.frame_length,waveform_length/self.fs*1000),
+ frame_shift=self.frame_shift,
+ dither=self.dither,
+ energy_floor=0.0,
+ window_type=self.window,
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/frontends/wav_frontend.py#L149" target="_blank" class="src-more">View full source on GitHub →</a></details></div>
+<div class="api-detail" id="e294"><span class="dtag badge-m">method</span><h2><code>WavFrontend.forward_fbank(input, input_lengths)</code></h2><div class="dmod">funasr.frontends.wav_frontend.WavFrontend · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/frontends/wav_frontend.py#L198" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Forward fbank.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>input</code> — Input audio/text data.</li>
+<li><code>input_lengths</code> — Lengths of input.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def forward_fbank(
+ self, input: torch.Tensor, input_lengths: torch.Tensor
+ ) -&gt; Tuple[torch.Tensor, torch.Tensor]:
+ &quot;&quot;&quot;Forward fbank.
+
+ Args:
+ input: Input audio/text data.
+ input_lengths: Lengths of input.
+ &quot;&quot;&quot;
+ batch_size = input.size(0)
+ feats = []
+ feats_lens = []
+ for i in range(batch_size):
+ waveform_length = input_lengths[i]
+ waveform = input[i][:waveform_length]
+ waveform = waveform * (1 &lt;&lt; 15)
+ waveform = waveform.unsqueeze(0)
+ mat = kaldi.fbank(
+ waveform,
+ num_mel_bins=self.n_mels,
+ frame_length=self.frame_length,
+ frame_shift=self.frame_shift,
+ dither=self.dither,
+ energy_floor=0.0,
+ window_type=self.window,
+ sample_frequency=self.fs,
+ )
+
+ feat_length = mat.size(0)
+ feats.append(mat)
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/frontends/wav_frontend.py#L198" target="_blank" class="src-more">View full source on GitHub →</a></details></div>
+<div class="api-detail" id="e295"><span class="dtag badge-m">method</span><h2><code>WavFrontend.forward_lfr_cmvn(input, input_lengths)</code></h2><div class="dmod">funasr.frontends.wav_frontend.WavFrontend · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/frontends/wav_frontend.py#L234" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Forward lfr cmvn.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>input</code> — Input audio/text data.</li>
+<li><code>input_lengths</code> — Lengths of input.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def forward_lfr_cmvn(
+ self, input: torch.Tensor, input_lengths: torch.Tensor
+ ) -&gt; Tuple[torch.Tensor, torch.Tensor]:
+ &quot;&quot;&quot;Forward lfr cmvn.
+
+ Args:
+ input: Input audio/text data.
+ input_lengths: Lengths of input.
+ &quot;&quot;&quot;
+ batch_size = input.size(0)
+ feats = []
+ feats_lens = []
+ for i in range(batch_size):
+ mat = input[i, : input_lengths[i], :]
+ if self.lfr_m != 1 or self.lfr_n != 1:
+ mat = apply_lfr(mat, self.lfr_m, self.lfr_n)
+ if self.cmvn is not None:
+ mat = apply_cmvn(mat, self.cmvn)
+ feat_length = mat.size(0)
+ feats.append(mat)
+ feats_lens.append(feat_length)
+
+ feats_lens = torch.as_tensor(feats_lens)
+ feats_pad = pad_sequence(feats, batch_first=True, padding_value=0.0)
+ return feats_pad, feats_lens
+</code></pre></details></div>
+<div class="api-detail" id="e296"><span class="dtag badge-c">class</span><h2>WavFrontendOnline</h2><div class="dmod">funasr.frontends.wav_frontend · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/frontends/wav_frontend.py#L262" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Conventional frontend structure for streaming ASR/VAD.</p>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code>class WavFrontendOnline(nn.Module):
+ &quot;&quot;&quot;Conventional frontend structure for streaming ASR/VAD.&quot;&quot;&quot;
+
+ def __init__(
+ self,
+ cmvn_file: str = None,
+ fs: int = 16000,
+ window: str = &quot;hamming&quot;,
+ n_mels: int = 80,
+ frame_length: int = 25,
+ frame_shift: int = 10,
+ filter_length_min: int = -1,
+ filter_length_max: int = -1,
+ lfr_m: int = 1,
+ lfr_n: int = 1,
+ dither: float = 1.0,
+ snip_edges: bool = True,
+ upsacle_samples: bool = True,
+ **kwargs,
+ ):
+ &quot;&quot;&quot;Initialize WavFrontendOnline.
+
+ Args:
+ cmvn_file: TODO.
+ fs: TODO.
+ window: TODO.
+ n_mels: TODO.
+ frame_length: TODO.
+ frame_shift: TODO.
+ filter_length_min: TODO.
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/frontends/wav_frontend.py#L262" target="_blank" class="src-more">View full source on GitHub →</a></details><h3 class="mt">Methods</h3><div class="mblk"><code>.output_size()</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/frontends/wav_frontend.py#L324" target="_blank" class="gh-link">L324</a><div class="mdoc"><p>Output size.</p>
+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def output_size(self) -&gt; int:
+ &quot;&quot;&quot;Output size.&quot;&quot;&quot;
+ return self.n_mels * self.lfr_m
+</code></pre></details></div><div class="mblk"><code>.apply_cmvn(inputs, cmvn)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/frontends/wav_frontend.py#L329" target="_blank" class="gh-link">L329</a><div class="mdoc"><p>Apply CMVN with mvn data</p>
+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def apply_cmvn(inputs: torch.Tensor, cmvn: torch.Tensor) -&gt; torch.Tensor:
+ &quot;&quot;&quot;
+ Apply CMVN with mvn data
+ &quot;&quot;&quot;
+
+ device = inputs.device
+ dtype = inputs.dtype
+ frame, dim = inputs.shape
+
+ means = np.tile(cmvn[0:1, :dim], (frame, 1))
+ vars = np.tile(cmvn[1:2, :dim], (frame, 1))
+ inputs += torch.from_numpy(means).type(dtype).to(device)
+ inputs *= torch.from_numpy(vars).type(dtype).to(device)
+
+ return inputs.type(torch.float32)
+</code></pre></details></div><div class="mblk"><code>.apply_lfr(inputs, lfr_m, lfr_n, is_final)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/frontends/wav_frontend.py#L346" target="_blank" class="gh-link">L346</a><div class="mdoc"><p>Apply lfr with data</p>
+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def apply_lfr(
+ inputs: torch.Tensor, lfr_m: int, lfr_n: int, is_final: bool = False
+ ) -&gt; Tuple[torch.Tensor, torch.Tensor, int]:
+ &quot;&quot;&quot;
+ Apply lfr with data
+ &quot;&quot;&quot;
+
+ LFR_inputs = []
+ # inputs = torch.vstack((inputs_lfr_cache, inputs))
+ T = inputs.shape[0] # include the right context
+ T_lfr = int(
+ np.ceil((T - (lfr_m - 1) // 2) / lfr_n)
+ ) # minus the right context: (lfr_m - 1) // 2
+ splice_idx = T_lfr
+ feat_dim = inputs.shape[-1]
+ ori_inputs = inputs
+ strides = (lfr_n * feat_dim, 1)
+ sizes = (T_lfr, lfr_m * feat_dim)
+ last_idx = (T - lfr_m) // lfr_n + 1
+ num_padding = lfr_m - (T - last_idx * lfr_n)
+ if is_final:
+ if num_padding &gt; 0:
+ num_padding = (2 * lfr_m - 2 * T + (T_lfr - 1 + last_idx) * lfr_n) / 2 * (T_lfr - last_idx)
+ inputs = torch.vstack([inputs] + [inputs[-1:]] * int(num_padding))
+ else:
+ if num_padding &gt; 0:
+ sizes = (last_idx, lfr_m * feat_dim)
+ splice_idx = last_idx
+ splice_idx = min(T - 1, splice_idx * lfr_n)
+ LFR_outputs = inputs[:splice_idx].as_strided(sizes, strides)
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/frontends/wav_frontend.py#L346" target="_blank" class="src-more">View full source →</a></details></div><div class="mblk"><code>.compute_frame_num(sample_length, frame_sample_length, frame_shift_sample_length)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/frontends/wav_frontend.py#L380" target="_blank" class="gh-link">L380</a><div class="mdoc"><p>Compute frame num.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>sample_length</code> — TODO.</li>
+<li><code>frame_sample_length</code> — TODO.</li>
+<li><code>frame_shift_sample_length</code> — TODO.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def compute_frame_num(
+ sample_length: int, frame_sample_length: int, frame_shift_sample_length: int
+ ) -&gt; int:
+ &quot;&quot;&quot;Compute frame num.
+
+ Args:
+ sample_length: TODO.
+ frame_sample_length: TODO.
+ frame_shift_sample_length: TODO.
+ &quot;&quot;&quot;
+ frame_num = int((sample_length - frame_sample_length) / frame_shift_sample_length + 1)
+ return frame_num if frame_num &gt;= 1 and sample_length &gt;= frame_sample_length else 0
+</code></pre></details></div><div class="mblk"><code>.forward_fbank(input, input_lengths, cache, **kwargs)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/frontends/wav_frontend.py#L393" target="_blank" class="gh-link">L393</a><div class="mdoc"><p>Forward fbank.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>input</code> — Input audio/text data.</li>
+<li><code>input_lengths</code> — Lengths of input.</li>
+<li><code>cache</code> — State cache dict for streaming inference.</li>
+<li><code>**kwargs</code> — Additional keyword arguments.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def forward_fbank(
+ self,
+ input: torch.Tensor,
+ input_lengths: torch.Tensor,
+ cache: dict = None,
+ **kwargs,
+ ) -&gt; Tuple[torch.Tensor, torch.Tensor, torch.Tensor]:
+ &quot;&quot;&quot;Forward fbank.
+
+ Args:
+ input: Input audio/text data.
+ input_lengths: Lengths of input.
+ cache: State cache dict for streaming inference.
+ **kwargs: Additional keyword arguments.
+ &quot;&quot;&quot;
+ if cache is None:
+ cache = {}
+ batch_size = input.size(0)
+
+ input = torch.cat((cache[&quot;input_cache&quot;], input), dim=1)
+ frame_num = self.compute_frame_num(
+ input.shape[-1], self.frame_sample_length, self.frame_shift_sample_length
+ )
+ # update self.in_cache
+ cache[&quot;input_cache&quot;] = input[
+ :, -(input.shape[-1] - frame_num * self.frame_shift_sample_length) :
+ ]
+ waveforms = torch.empty(0)
+ feats_pad = torch.empty(0)
+ feats_lens = torch.empty(0)
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/frontends/wav_frontend.py#L393" target="_blank" class="src-more">View full source →</a></details></div><div class="mblk"><code>.forward_lfr_cmvn(input, input_lengths, is_final, cache, **kwargs)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/frontends/wav_frontend.py#L462" target="_blank" class="gh-link">L462</a><div class="mdoc"><p>Forward lfr cmvn.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>input</code> — Input audio/text data.</li>
+<li><code>input_lengths</code> — Lengths of input.</li>
+<li><code>is_final</code> — Whether this is the final chunk in streaming.</li>
+<li><code>cache</code> — State cache dict for streaming inference.</li>
+<li><code>**kwargs</code> — Additional keyword arguments.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def forward_lfr_cmvn(
+ self,
+ input: torch.Tensor,
+ input_lengths: torch.Tensor,
+ is_final: bool = False,
+ cache: dict = None,
+ **kwargs,
+ ):
+ &quot;&quot;&quot;Forward lfr cmvn.
+
+ Args:
+ input: Input audio/text data.
+ input_lengths: Lengths of input.
+ is_final: Whether this is the final chunk in streaming.
+ cache: State cache dict for streaming inference.
+ **kwargs: Additional keyword arguments.
+ &quot;&quot;&quot;
+ if cache is None:
+ cache = {}
+ batch_size = input.size(0)
+ feats = []
+ feats_lens = []
+ lfr_splice_frame_idxs = []
+ for i in range(batch_size):
+ mat = input[i, : input_lengths[i], :]
+ if self.lfr_m != 1 or self.lfr_n != 1:
+ # update self.lfr_splice_cache in self.apply_lfr
+ # mat, self.lfr_splice_cache[i], lfr_splice_frame_idx = self.apply_lfr(mat, self.lfr_m, self.lfr_n, self.lfr_splice_cache[i],
+ mat, cache[&quot;lfr_splice_cache&quot;][i], lfr_splice_frame_idx = self.apply_lfr(
+ mat, self.lfr_m, self.lfr_n, is_final
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/frontends/wav_frontend.py#L462" target="_blank" class="src-more">View full source →</a></details></div><div class="mblk"><code>.forward(input, input_lengths, **kwargs)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/frontends/wav_frontend.py#L505" target="_blank" class="gh-link">L505</a><div class="mdoc"><p>Forward pass for training.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>input</code> — Input audio/text data.</li>
+<li><code>input_lengths</code> — Lengths of input.</li>
+<li><code>**kwargs</code> — Additional keyword arguments.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def forward(self, input: torch.Tensor, input_lengths: torch.Tensor, **kwargs):
+ &quot;&quot;&quot;Forward pass for training.
+
+ Args:
+ input: Input audio/text data.
+ input_lengths: Lengths of input.
+ **kwargs: Additional keyword arguments.
+ &quot;&quot;&quot;
+ is_final = kwargs.get(&quot;is_final&quot;, False)
+ cache = kwargs.get(&quot;cache&quot;, {})
+ if len(cache) == 0:
+ self.init_cache(cache)
+
+ batch_size = input.shape[0]
+ assert (
+ batch_size == 1
+ ), &quot;we support to extract feature online only when the batch size is equal to 1 now&quot;
+
+ waveforms, feats, feats_lengths = self.forward_fbank(
+ input, input_lengths, cache=cache
+ ) # input shape: B T D
+
+ if feats.shape[0]:
+
+ cache[&quot;waveforms&quot;] = torch.cat((cache[&quot;reserve_waveforms&quot;], waveforms), dim=1)
+
+ if not cache[&quot;lfr_splice_cache&quot;]: # 初始化splice_cache
+ for i in range(batch_size):
+ cache[&quot;lfr_splice_cache&quot;].append(
+ feats[i][0, :].unsqueeze(dim=0).repeat((self.lfr_m - 1) // 2, 1)
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/frontends/wav_frontend.py#L505" target="_blank" class="src-more">View full source →</a></details></div><div class="mblk"><code>.init_cache(cache)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/frontends/wav_frontend.py#L594" target="_blank" class="gh-link">L594</a><div class="mdoc"><p>Init cache.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>cache</code> — State cache dict for streaming inference.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def init_cache(self, cache: dict = None):
+ &quot;&quot;&quot;Init cache.
+
+ Args:
+ cache: State cache dict for streaming inference.
+ &quot;&quot;&quot;
+ if cache is None:
+ cache = {}
+ cache[&quot;reserve_waveforms&quot;] = torch.empty(0)
+ cache[&quot;input_cache&quot;] = torch.empty(0)
+ cache[&quot;lfr_splice_cache&quot;] = []
+ cache[&quot;waveforms&quot;] = None
+ cache[&quot;fbanks&quot;] = None
+ cache[&quot;fbanks_lens&quot;] = None
+ return cache
+</code></pre></details></div></div>
+<div class="api-detail" id="e297"><span class="dtag badge-m">method</span><h2><code>WavFrontendOnline.output_size()</code></h2><div class="dmod">funasr.frontends.wav_frontend.WavFrontendOnline · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/frontends/wav_frontend.py#L324" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Output size.</p>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def output_size(self) -&gt; int:
+ &quot;&quot;&quot;Output size.&quot;&quot;&quot;
+ return self.n_mels * self.lfr_m
+</code></pre></details></div>
+<div class="api-detail" id="e298"><span class="dtag badge-m">method</span><h2><code>WavFrontendOnline.apply_cmvn(inputs, cmvn)</code></h2><div class="dmod">funasr.frontends.wav_frontend.WavFrontendOnline · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/frontends/wav_frontend.py#L329" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Apply CMVN with mvn data</p>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def apply_cmvn(inputs: torch.Tensor, cmvn: torch.Tensor) -&gt; torch.Tensor:
+ &quot;&quot;&quot;
+ Apply CMVN with mvn data
+ &quot;&quot;&quot;
+
+ device = inputs.device
+ dtype = inputs.dtype
+ frame, dim = inputs.shape
+
+ means = np.tile(cmvn[0:1, :dim], (frame, 1))
+ vars = np.tile(cmvn[1:2, :dim], (frame, 1))
+ inputs += torch.from_numpy(means).type(dtype).to(device)
+ inputs *= torch.from_numpy(vars).type(dtype).to(device)
+
+ return inputs.type(torch.float32)
+</code></pre></details></div>
+<div class="api-detail" id="e299"><span class="dtag badge-m">method</span><h2><code>WavFrontendOnline.apply_lfr(inputs, lfr_m, lfr_n, is_final)</code></h2><div class="dmod">funasr.frontends.wav_frontend.WavFrontendOnline · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/frontends/wav_frontend.py#L346" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Apply lfr with data</p>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def apply_lfr(
+ inputs: torch.Tensor, lfr_m: int, lfr_n: int, is_final: bool = False
+ ) -&gt; Tuple[torch.Tensor, torch.Tensor, int]:
+ &quot;&quot;&quot;
+ Apply lfr with data
+ &quot;&quot;&quot;
+
+ LFR_inputs = []
+ # inputs = torch.vstack((inputs_lfr_cache, inputs))
+ T = inputs.shape[0] # include the right context
+ T_lfr = int(
+ np.ceil((T - (lfr_m - 1) // 2) / lfr_n)
+ ) # minus the right context: (lfr_m - 1) // 2
+ splice_idx = T_lfr
+ feat_dim = inputs.shape[-1]
+ ori_inputs = inputs
+ strides = (lfr_n * feat_dim, 1)
+ sizes = (T_lfr, lfr_m * feat_dim)
+ last_idx = (T - lfr_m) // lfr_n + 1
+ num_padding = lfr_m - (T - last_idx * lfr_n)
+ if is_final:
+ if num_padding &gt; 0:
+ num_padding = (2 * lfr_m - 2 * T + (T_lfr - 1 + last_idx) * lfr_n) / 2 * (T_lfr - last_idx)
+ inputs = torch.vstack([inputs] + [inputs[-1:]] * int(num_padding))
+ else:
+ if num_padding &gt; 0:
+ sizes = (last_idx, lfr_m * feat_dim)
+ splice_idx = last_idx
+ splice_idx = min(T - 1, splice_idx * lfr_n)
+ LFR_outputs = inputs[:splice_idx].as_strided(sizes, strides)
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/frontends/wav_frontend.py#L346" target="_blank" class="src-more">View full source on GitHub →</a></details></div>
+<div class="api-detail" id="e300"><span class="dtag badge-m">method</span><h2><code>WavFrontendOnline.compute_frame_num(sample_length, frame_sample_length, frame_shift_sample_length)</code></h2><div class="dmod">funasr.frontends.wav_frontend.WavFrontendOnline · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/frontends/wav_frontend.py#L380" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Compute frame num.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>sample_length</code> — TODO.</li>
+<li><code>frame_sample_length</code> — TODO.</li>
+<li><code>frame_shift_sample_length</code> — TODO.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def compute_frame_num(
+ sample_length: int, frame_sample_length: int, frame_shift_sample_length: int
+ ) -&gt; int:
+ &quot;&quot;&quot;Compute frame num.
+
+ Args:
+ sample_length: TODO.
+ frame_sample_length: TODO.
+ frame_shift_sample_length: TODO.
+ &quot;&quot;&quot;
+ frame_num = int((sample_length - frame_sample_length) / frame_shift_sample_length + 1)
+ return frame_num if frame_num &gt;= 1 and sample_length &gt;= frame_sample_length else 0
+</code></pre></details></div>
+<div class="api-detail" id="e301"><span class="dtag badge-m">method</span><h2><code>WavFrontendOnline.forward_fbank(input, input_lengths, cache, **kwargs)</code></h2><div class="dmod">funasr.frontends.wav_frontend.WavFrontendOnline · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/frontends/wav_frontend.py#L393" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Forward fbank.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>input</code> — Input audio/text data.</li>
+<li><code>input_lengths</code> — Lengths of input.</li>
+<li><code>cache</code> — State cache dict for streaming inference.</li>
+<li><code>**kwargs</code> — Additional keyword arguments.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def forward_fbank(
+ self,
+ input: torch.Tensor,
+ input_lengths: torch.Tensor,
+ cache: dict = None,
+ **kwargs,
+ ) -&gt; Tuple[torch.Tensor, torch.Tensor, torch.Tensor]:
+ &quot;&quot;&quot;Forward fbank.
+
+ Args:
+ input: Input audio/text data.
+ input_lengths: Lengths of input.
+ cache: State cache dict for streaming inference.
+ **kwargs: Additional keyword arguments.
+ &quot;&quot;&quot;
+ if cache is None:
+ cache = {}
+ batch_size = input.size(0)
+
+ input = torch.cat((cache[&quot;input_cache&quot;], input), dim=1)
+ frame_num = self.compute_frame_num(
+ input.shape[-1], self.frame_sample_length, self.frame_shift_sample_length
+ )
+ # update self.in_cache
+ cache[&quot;input_cache&quot;] = input[
+ :, -(input.shape[-1] - frame_num * self.frame_shift_sample_length) :
+ ]
+ waveforms = torch.empty(0)
+ feats_pad = torch.empty(0)
+ feats_lens = torch.empty(0)
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/frontends/wav_frontend.py#L393" target="_blank" class="src-more">View full source on GitHub →</a></details></div>
+<div class="api-detail" id="e302"><span class="dtag badge-m">method</span><h2><code>WavFrontendOnline.forward_lfr_cmvn(input, input_lengths, is_final, cache, **kwargs)</code></h2><div class="dmod">funasr.frontends.wav_frontend.WavFrontendOnline · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/frontends/wav_frontend.py#L462" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Forward lfr cmvn.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>input</code> — Input audio/text data.</li>
+<li><code>input_lengths</code> — Lengths of input.</li>
+<li><code>is_final</code> — Whether this is the final chunk in streaming.</li>
+<li><code>cache</code> — State cache dict for streaming inference.</li>
+<li><code>**kwargs</code> — Additional keyword arguments.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def forward_lfr_cmvn(
+ self,
+ input: torch.Tensor,
+ input_lengths: torch.Tensor,
+ is_final: bool = False,
+ cache: dict = None,
+ **kwargs,
+ ):
+ &quot;&quot;&quot;Forward lfr cmvn.
+
+ Args:
+ input: Input audio/text data.
+ input_lengths: Lengths of input.
+ is_final: Whether this is the final chunk in streaming.
+ cache: State cache dict for streaming inference.
+ **kwargs: Additional keyword arguments.
+ &quot;&quot;&quot;
+ if cache is None:
+ cache = {}
+ batch_size = input.size(0)
+ feats = []
+ feats_lens = []
+ lfr_splice_frame_idxs = []
+ for i in range(batch_size):
+ mat = input[i, : input_lengths[i], :]
+ if self.lfr_m != 1 or self.lfr_n != 1:
+ # update self.lfr_splice_cache in self.apply_lfr
+ # mat, self.lfr_splice_cache[i], lfr_splice_frame_idx = self.apply_lfr(mat, self.lfr_m, self.lfr_n, self.lfr_splice_cache[i],
+ mat, cache[&quot;lfr_splice_cache&quot;][i], lfr_splice_frame_idx = self.apply_lfr(
+ mat, self.lfr_m, self.lfr_n, is_final
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/frontends/wav_frontend.py#L462" target="_blank" class="src-more">View full source on GitHub →</a></details></div>
+<div class="api-detail" id="e303"><span class="dtag badge-m">method</span><h2><code>WavFrontendOnline.forward(input, input_lengths, **kwargs)</code></h2><div class="dmod">funasr.frontends.wav_frontend.WavFrontendOnline · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/frontends/wav_frontend.py#L505" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Forward pass for training.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>input</code> — Input audio/text data.</li>
+<li><code>input_lengths</code> — Lengths of input.</li>
+<li><code>**kwargs</code> — Additional keyword arguments.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def forward(self, input: torch.Tensor, input_lengths: torch.Tensor, **kwargs):
+ &quot;&quot;&quot;Forward pass for training.
+
+ Args:
+ input: Input audio/text data.
+ input_lengths: Lengths of input.
+ **kwargs: Additional keyword arguments.
+ &quot;&quot;&quot;
+ is_final = kwargs.get(&quot;is_final&quot;, False)
+ cache = kwargs.get(&quot;cache&quot;, {})
+ if len(cache) == 0:
+ self.init_cache(cache)
+
+ batch_size = input.shape[0]
+ assert (
+ batch_size == 1
+ ), &quot;we support to extract feature online only when the batch size is equal to 1 now&quot;
+
+ waveforms, feats, feats_lengths = self.forward_fbank(
+ input, input_lengths, cache=cache
+ ) # input shape: B T D
+
+ if feats.shape[0]:
+
+ cache[&quot;waveforms&quot;] = torch.cat((cache[&quot;reserve_waveforms&quot;], waveforms), dim=1)
+
+ if not cache[&quot;lfr_splice_cache&quot;]: # 初始化splice_cache
+ for i in range(batch_size):
+ cache[&quot;lfr_splice_cache&quot;].append(
+ feats[i][0, :].unsqueeze(dim=0).repeat((self.lfr_m - 1) // 2, 1)
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/frontends/wav_frontend.py#L505" target="_blank" class="src-more">View full source on GitHub →</a></details></div>
+<div class="api-detail" id="e304"><span class="dtag badge-m">method</span><h2><code>WavFrontendOnline.init_cache(cache)</code></h2><div class="dmod">funasr.frontends.wav_frontend.WavFrontendOnline · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/frontends/wav_frontend.py#L594" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Init cache.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>cache</code> — State cache dict for streaming inference.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def init_cache(self, cache: dict = None):
+ &quot;&quot;&quot;Init cache.
+
+ Args:
+ cache: State cache dict for streaming inference.
+ &quot;&quot;&quot;
+ if cache is None:
+ cache = {}
+ cache[&quot;reserve_waveforms&quot;] = torch.empty(0)
+ cache[&quot;input_cache&quot;] = torch.empty(0)
+ cache[&quot;lfr_splice_cache&quot;] = []
+ cache[&quot;waveforms&quot;] = None
+ cache[&quot;fbanks&quot;] = None
+ cache[&quot;fbanks_lens&quot;] = None
+ return cache
+</code></pre></details></div>
+<div class="api-detail" id="e305"><span class="dtag badge-c">class</span><h2>WavFrontendMel23</h2><div class="dmod">funasr.frontends.wav_frontend · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/frontends/wav_frontend.py#L611" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Conventional frontend structure for ASR.</p>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code>class WavFrontendMel23(nn.Module):
+ &quot;&quot;&quot;Conventional frontend structure for ASR.&quot;&quot;&quot;
+
+ def __init__(
+ self,
+ fs: int = 16000,
+ frame_length: int = 25,
+ frame_shift: int = 10,
+ lfr_m: int = 1,
+ lfr_n: int = 1,
+ **kwargs,
+ ):
+ &quot;&quot;&quot;Initialize WavFrontendMel23.
+
+ Args:
+ fs: TODO.
+ frame_length: TODO.
+ frame_shift: TODO.
+ lfr_m: TODO.
+ lfr_n: TODO.
+ **kwargs: Additional keyword arguments.
+ &quot;&quot;&quot;
+ super().__init__()
+ self.fs = fs
+ self.frame_length = frame_length
+ self.frame_shift = frame_shift
+ self.lfr_m = lfr_m
+ self.lfr_n = lfr_n
+ self.n_mels = 23
+
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/frontends/wav_frontend.py#L611" target="_blank" class="src-more">View full source on GitHub →</a></details><h3 class="mt">Methods</h3><div class="mblk"><code>.output_size()</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/frontends/wav_frontend.py#L641" target="_blank" class="gh-link">L641</a><div class="mdoc"><p>Output size.</p>
+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def output_size(self) -&gt; int:
+ &quot;&quot;&quot;Output size.&quot;&quot;&quot;
+ return self.n_mels * (2 * self.lfr_m + 1)
+</code></pre></details></div><div class="mblk"><code>.forward(input, input_lengths)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/frontends/wav_frontend.py#L645" target="_blank" class="gh-link">L645</a><div class="mdoc"><p>Forward pass for training.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>input</code> — Input audio/text data.</li>
+<li><code>input_lengths</code> — Lengths of input.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def forward(
+ self, input: torch.Tensor, input_lengths: torch.Tensor
+ ) -&gt; Tuple[torch.Tensor, torch.Tensor]:
+ &quot;&quot;&quot;Forward pass for training.
+
+ Args:
+ input: Input audio/text data.
+ input_lengths: Lengths of input.
+ &quot;&quot;&quot;
+ batch_size = input.size(0)
+ feats = []
+ feats_lens = []
+ for i in range(batch_size):
+ waveform_length = input_lengths[i]
+ waveform = input[i][:waveform_length]
+ waveform = waveform.numpy()
+ mat = eend_ola_feature.stft(waveform, self.frame_length, self.frame_shift)
+ mat = eend_ola_feature.transform(mat)
+ mat = eend_ola_feature.splice(mat, context_size=self.lfr_m)
+ mat = mat[:: self.lfr_n]
+ mat = torch.from_numpy(mat)
+ feat_length = mat.size(0)
+ feats.append(mat)
+ feats_lens.append(feat_length)
+
+ feats_lens = torch.as_tensor(feats_lens)
+ feats_pad = pad_sequence(feats, batch_first=True, padding_value=0.0)
+ return feats_pad, feats_lens
+</code></pre></details></div></div>
+<div class="api-detail" id="e306"><span class="dtag badge-m">method</span><h2><code>WavFrontendMel23.output_size()</code></h2><div class="dmod">funasr.frontends.wav_frontend.WavFrontendMel23 · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/frontends/wav_frontend.py#L641" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Output size.</p>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def output_size(self) -&gt; int:
+ &quot;&quot;&quot;Output size.&quot;&quot;&quot;
+ return self.n_mels * (2 * self.lfr_m + 1)
+</code></pre></details></div>
+<div class="api-detail" id="e307"><span class="dtag badge-m">method</span><h2><code>WavFrontendMel23.forward(input, input_lengths)</code></h2><div class="dmod">funasr.frontends.wav_frontend.WavFrontendMel23 · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/frontends/wav_frontend.py#L645" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Forward pass for training.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>input</code> — Input audio/text data.</li>
+<li><code>input_lengths</code> — Lengths of input.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def forward(
+ self, input: torch.Tensor, input_lengths: torch.Tensor
+ ) -&gt; Tuple[torch.Tensor, torch.Tensor]:
+ &quot;&quot;&quot;Forward pass for training.
+
+ Args:
+ input: Input audio/text data.
+ input_lengths: Lengths of input.
+ &quot;&quot;&quot;
+ batch_size = input.size(0)
+ feats = []
+ feats_lens = []
+ for i in range(batch_size):
+ waveform_length = input_lengths[i]
+ waveform = input[i][:waveform_length]
+ waveform = waveform.numpy()
+ mat = eend_ola_feature.stft(waveform, self.frame_length, self.frame_shift)
+ mat = eend_ola_feature.transform(mat)
+ mat = eend_ola_feature.splice(mat, context_size=self.lfr_m)
+ mat = mat[:: self.lfr_n]
+ mat = torch.from_numpy(mat)
+ feat_length = mat.size(0)
+ feats.append(mat)
+ feats_lens.append(feat_length)
+
+ feats_lens = torch.as_tensor(feats_lens)
+ feats_pad = pad_sequence(feats, batch_first=True, padding_value=0.0)
+ return feats_pad, feats_lens
+</code></pre></details></div>
+<div class="api-detail" id="e308"><span class="dtag badge-c">class</span><h2>WhisperFrontend</h2><div class="dmod">funasr.frontends.whisper_frontend · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/frontends/whisper_frontend.py#L11" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Speech Representation Using Encoder Outputs from OpenAI&#x27;s Whisper Model:</p>
+<ul>
+<li><code>URL</code> — https://github.com/openai/whisper</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code>class WhisperFrontend(nn.Module):
+ &quot;&quot;&quot;Speech Representation Using Encoder Outputs from OpenAI&#x27;s Whisper Model:
+
+ URL: https://github.com/openai/whisper
+ &quot;&quot;&quot;
+
+ def __init__(
+ self,
+ fs: int = 16000,
+ whisper_model: str = None,
+ do_pad_trim: bool = True,
+ n_mels: int = 80,
+ permute: bool = False,
+ **kwargs,
+ ):
+ &quot;&quot;&quot;Initialize WhisperFrontend.
+
+ Args:
+ fs: TODO.
+ whisper_model: Whisper Model instance.
+ do_pad_trim: TODO.
+ n_mels: TODO.
+ permute: TODO.
+ **kwargs: Additional keyword arguments.
+ &quot;&quot;&quot;
+ super().__init__()
+ assert fs == 16000
+ self.fs = fs
+ import whisper
+ from whisper.audio import HOP_LENGTH, N_FFT, N_SAMPLES
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/frontends/whisper_frontend.py#L11" target="_blank" class="src-more">View full source on GitHub →</a></details><h3 class="mt">Methods</h3><div class="mblk"><code>.output_size()</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/frontends/whisper_frontend.py#L67" target="_blank" class="gh-link">L67</a><div class="mdoc"><p>Output size.</p>
+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def output_size(self) -&gt; int:
+ &quot;&quot;&quot;Output size.&quot;&quot;&quot;
+ return self.n_mels
+</code></pre></details></div><div class="mblk"><code>.log_mel_spectrogram(audio, ilens)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/frontends/whisper_frontend.py#L71" target="_blank" class="gh-link">L71</a><div class="mdoc"><p>Log mel spectrogram.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>audio</code> — TODO.</li>
+<li><code>ilens</code> — TODO.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def log_mel_spectrogram(
+ self,
+ audio: torch.Tensor,
+ ilens: torch.Tensor = None,
+ ) -&gt; torch.Tensor:
+ &quot;&quot;&quot;Log mel spectrogram.
+
+ Args:
+ audio: TODO.
+ ilens: TODO.
+ &quot;&quot;&quot;
+ window = torch.hann_window(self.win_length).to(audio.device)
+ stft = torch.stft(audio, self.n_fft, self.hop_length, window=window, return_complex=True)
+
+ # whisper deletes the last frame by default (Shih-Lun)
+ magnitudes = stft[..., :-1].abs() ** 2
+ if self.filters_path is not None:
+ filters = self.mel_filters(audio.device, self.n_mels, self.filters_path)
+ else:
+ filters = self.mel_filters(audio.device, self.n_mels)
+ mel_spec = filters @ magnitudes
+
+ log_spec = torch.clamp(mel_spec, min=1e-10).log10()
+
+ if ilens is not None:
+ olens = ilens // self.hop_length
+ else:
+ olens = None
+
+ log_spec = torch.maximum(
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/frontends/whisper_frontend.py#L71" target="_blank" class="src-more">View full source →</a></details></div><div class="mblk"><code>.forward(input, input_lengths, **kwargs)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/frontends/whisper_frontend.py#L108" target="_blank" class="gh-link">L108</a><div class="mdoc"><p>Forward pass for training.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>input</code> — Input audio/text data.</li>
+<li><code>input_lengths</code> — Lengths of input.</li>
+<li><code>**kwargs</code> — Additional keyword arguments.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def forward(
+ self,
+ input: torch.Tensor,
+ input_lengths: torch.Tensor,
+ **kwargs,
+ ) -&gt; Tuple[torch.Tensor, torch.Tensor]:
+ &quot;&quot;&quot;Forward pass for training.
+
+ Args:
+ input: Input audio/text data.
+ input_lengths: Lengths of input.
+ **kwargs: Additional keyword arguments.
+ &quot;&quot;&quot;
+ batch_size = input.size(0)
+ feats = []
+ feats_lens = []
+ input = input.to(torch.float32)
+ for i in range(batch_size):
+ if self.do_pad_trim:
+ feat = self.pad_or_trim(input[i], self.pad_samples)
+ else:
+ feat = input[i]
+ feat, feat_len = self.log_mel_spectrogram(feat[None, :], input_lengths[0])
+ feats.append(feat[0])
+ feats_lens.append(feat_len)
+ feats_lens = torch.as_tensor(feats_lens)
+
+ if batch_size == 1:
+ feats_pad = feats[0][None, :, :]
+ else:
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/frontends/whisper_frontend.py#L108" target="_blank" class="src-more">View full source →</a></details></div></div>
+<div class="api-detail" id="e309"><span class="dtag badge-m">method</span><h2><code>WhisperFrontend.output_size()</code></h2><div class="dmod">funasr.frontends.whisper_frontend.WhisperFrontend · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/frontends/whisper_frontend.py#L67" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Output size.</p>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def output_size(self) -&gt; int:
+ &quot;&quot;&quot;Output size.&quot;&quot;&quot;
+ return self.n_mels
+</code></pre></details></div>
+<div class="api-detail" id="e310"><span class="dtag badge-m">method</span><h2><code>WhisperFrontend.log_mel_spectrogram(audio, ilens)</code></h2><div class="dmod">funasr.frontends.whisper_frontend.WhisperFrontend · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/frontends/whisper_frontend.py#L71" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Log mel spectrogram.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>audio</code> — TODO.</li>
+<li><code>ilens</code> — TODO.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def log_mel_spectrogram(
+ self,
+ audio: torch.Tensor,
+ ilens: torch.Tensor = None,
+ ) -&gt; torch.Tensor:
+ &quot;&quot;&quot;Log mel spectrogram.
+
+ Args:
+ audio: TODO.
+ ilens: TODO.
+ &quot;&quot;&quot;
+ window = torch.hann_window(self.win_length).to(audio.device)
+ stft = torch.stft(audio, self.n_fft, self.hop_length, window=window, return_complex=True)
+
+ # whisper deletes the last frame by default (Shih-Lun)
+ magnitudes = stft[..., :-1].abs() ** 2
+ if self.filters_path is not None:
+ filters = self.mel_filters(audio.device, self.n_mels, self.filters_path)
+ else:
+ filters = self.mel_filters(audio.device, self.n_mels)
+ mel_spec = filters @ magnitudes
+
+ log_spec = torch.clamp(mel_spec, min=1e-10).log10()
+
+ if ilens is not None:
+ olens = ilens // self.hop_length
+ else:
+ olens = None
+
+ log_spec = torch.maximum(
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/frontends/whisper_frontend.py#L71" target="_blank" class="src-more">View full source on GitHub →</a></details></div>
+<div class="api-detail" id="e311"><span class="dtag badge-m">method</span><h2><code>WhisperFrontend.forward(input, input_lengths, **kwargs)</code></h2><div class="dmod">funasr.frontends.whisper_frontend.WhisperFrontend · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/frontends/whisper_frontend.py#L108" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Forward pass for training.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>input</code> — Input audio/text data.</li>
+<li><code>input_lengths</code> — Lengths of input.</li>
+<li><code>**kwargs</code> — Additional keyword arguments.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def forward(
+ self,
+ input: torch.Tensor,
+ input_lengths: torch.Tensor,
+ **kwargs,
+ ) -&gt; Tuple[torch.Tensor, torch.Tensor]:
+ &quot;&quot;&quot;Forward pass for training.
+
+ Args:
+ input: Input audio/text data.
+ input_lengths: Lengths of input.
+ **kwargs: Additional keyword arguments.
+ &quot;&quot;&quot;
+ batch_size = input.size(0)
+ feats = []
+ feats_lens = []
+ input = input.to(torch.float32)
+ for i in range(batch_size):
+ if self.do_pad_trim:
+ feat = self.pad_or_trim(input[i], self.pad_samples)
+ else:
+ feat = input[i]
+ feat, feat_len = self.log_mel_spectrogram(feat[None, :], input_lengths[0])
+ feats.append(feat[0])
+ feats_lens.append(feat_len)
+ feats_lens = torch.as_tensor(feats_lens)
+
+ if batch_size == 1:
+ feats_pad = feats[0][None, :, :]
+ else:
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/frontends/whisper_frontend.py#L108" target="_blank" class="src-more">View full source on GitHub →</a></details></div>
+<div class="api-detail" id="e312"><span class="dtag badge-c">class</span><h2>SlidingWindow</h2><div class="dmod">funasr.frontends.windowing · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/frontends/windowing.py#L12" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Sliding Window.</p>
+<p>Provides a sliding window over a batched continuous raw audio tensor.</p>
+<p>Optionally, provides padding (Currently not implemented).</p>
+<p>Combine this module with a pre-encoder compatible with raw audio data,</p>
+<p>for example Sinc convolutions.</p>
+<p>Known issues:</p>
+<p>Output length is calculated incorrectly if audio shorter than win_length.</p>
+<ul>
+<li><code>WARNING</code> — trailing values are discarded - padding not implemented yet.</li>
+</ul>
+<p>There is currently no additional window function applied to input values.</p>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code>class SlidingWindow(nn.Module):
+ &quot;&quot;&quot;Sliding Window.
+ Provides a sliding window over a batched continuous raw audio tensor.
+ Optionally, provides padding (Currently not implemented).
+ Combine this module with a pre-encoder compatible with raw audio data,
+ for example Sinc convolutions.
+ Known issues:
+ Output length is calculated incorrectly if audio shorter than win_length.
+ WARNING: trailing values are discarded - padding not implemented yet.
+ There is currently no additional window function applied to input values.
+ &quot;&quot;&quot;
+
+ def __init__(
+ self,
+ win_length: int = 400,
+ hop_length: int = 160,
+ channels: int = 1,
+ padding: int = None,
+ fs=None,
+ ):
+ &quot;&quot;&quot;Initialize.
+ Args:
+ win_length: Length of frame.
+ hop_length: Relative starting point of next frame.
+ channels: Number of input channels.
+ padding: Padding (placeholder, currently not implemented).
+ fs: Sampling rate (placeholder for compatibility, not used).
+ &quot;&quot;&quot;
+ super().__init__()
+ self.fs = fs
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/frontends/windowing.py#L12" target="_blank" class="src-more">View full source on GitHub →</a></details><h3 class="mt">Methods</h3><div class="mblk"><code>.forward(input, input_lengths)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/frontends/windowing.py#L47" target="_blank" class="gh-link">L47</a><div class="mdoc"><p>Apply a sliding window on the input.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>input</code> — Input (B, T, C*D) or (B, T*C*D), with D=C=1.</li>
+<li><code>input_lengths</code> — Input lengths within batch.</li>
+</ul>
+<h4>Returns:</h4>
+<ul>
+<li><code>Tensor</code> — Output with dimensions (B, T, C, D), with D=win_length.</li>
+<li><code>Tensor</code> — Output lengths within batch.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def forward(
+ self, input: torch.Tensor, input_lengths: torch.Tensor
+ ) -&gt; Tuple[torch.Tensor, torch.Tensor]:
+ &quot;&quot;&quot;Apply a sliding window on the input.
+ Args:
+ input: Input (B, T, C*D) or (B, T*C*D), with D=C=1.
+ input_lengths: Input lengths within batch.
+ Returns:
+ Tensor: Output with dimensions (B, T, C, D), with D=win_length.
+ Tensor: Output lengths within batch.
+ &quot;&quot;&quot;
+ input_size = input.size()
+ B = input_size[0]
+ T = input_size[1]
+ C = self.channels
+ D = self.win_length
+ # (B, T, C) --&gt; (T, B, C)
+ continuous = input.view(B, T, C).permute(1, 0, 2)
+ windowed = continuous.unfold(0, D, self.hop_length)
+ # (T, B, C, D) --&gt; (B, T, C, D)
+ output = windowed.permute(1, 0, 2, 3).contiguous()
+ # After unfold(), windowed lengths change:
+ output_lengths = (input_lengths - self.win_length) // self.hop_length + 1
+ return output, output_lengths
+</code></pre></details></div><div class="mblk"><code>.output_size()</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/frontends/windowing.py#L72" target="_blank" class="gh-link">L72</a><div class="mdoc"><p>Return output length of feature dimension D, i.e. the window length.</p>
+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def output_size(self) -&gt; int:
+ &quot;&quot;&quot;Return output length of feature dimension D, i.e. the window length.&quot;&quot;&quot;
+ return self.win_length
+</code></pre></details></div></div>
+<div class="api-detail" id="e313"><span class="dtag badge-m">method</span><h2><code>SlidingWindow.forward(input, input_lengths)</code></h2><div class="dmod">funasr.frontends.windowing.SlidingWindow · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/frontends/windowing.py#L47" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Apply a sliding window on the input.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>input</code> — Input (B, T, C*D) or (B, T*C*D), with D=C=1.</li>
+<li><code>input_lengths</code> — Input lengths within batch.</li>
+</ul>
+<h4>Returns:</h4>
+<ul>
+<li><code>Tensor</code> — Output with dimensions (B, T, C, D), with D=win_length.</li>
+<li><code>Tensor</code> — Output lengths within batch.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def forward(
+ self, input: torch.Tensor, input_lengths: torch.Tensor
+ ) -&gt; Tuple[torch.Tensor, torch.Tensor]:
+ &quot;&quot;&quot;Apply a sliding window on the input.
+ Args:
+ input: Input (B, T, C*D) or (B, T*C*D), with D=C=1.
+ input_lengths: Input lengths within batch.
+ Returns:
+ Tensor: Output with dimensions (B, T, C, D), with D=win_length.
+ Tensor: Output lengths within batch.
+ &quot;&quot;&quot;
+ input_size = input.size()
+ B = input_size[0]
+ T = input_size[1]
+ C = self.channels
+ D = self.win_length
+ # (B, T, C) --&gt; (T, B, C)
+ continuous = input.view(B, T, C).permute(1, 0, 2)
+ windowed = continuous.unfold(0, D, self.hop_length)
+ # (T, B, C, D) --&gt; (B, T, C, D)
+ output = windowed.permute(1, 0, 2, 3).contiguous()
+ # After unfold(), windowed lengths change:
+ output_lengths = (input_lengths - self.win_length) // self.hop_length + 1
+ return output, output_lengths
+</code></pre></details></div>
+<div class="api-detail" id="e314"><span class="dtag badge-m">method</span><h2><code>SlidingWindow.output_size()</code></h2><div class="dmod">funasr.frontends.windowing.SlidingWindow · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/frontends/windowing.py#L72" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Return output length of feature dimension D, i.e. the window length.</p>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def output_size(self) -&gt; int:
+ &quot;&quot;&quot;Return output length of feature dimension D, i.e. the window length.&quot;&quot;&quot;
+ return self.win_length
+</code></pre></details></div>
+<div class="api-detail" id="e315"><span class="dtag badge-f">function</span><h2><code>build_tokenizer(token_type, bpemodel, non_linguistic_symbols, remove_non_linguistic_symbols, space_symbol, delimiter, g2p_type)</code></h2><div class="dmod">funasr.tokenizer.build_tokenizer · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/tokenizer/build_tokenizer.py#L13" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>A helper function to instantiate Tokenizer</p>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code>def build_tokenizer(
+ token_type: str,
+ bpemodel: Union[Path, str, Iterable[str]] = None,
+ non_linguistic_symbols: Union[Path, str, Iterable[str]] = None,
+ remove_non_linguistic_symbols: bool = False,
+ space_symbol: str = &quot;&lt;space&gt;&quot;,
+ delimiter: str = None,
+ g2p_type: str = None,
+) -&gt; AbsTokenizer:
+ &quot;&quot;&quot;A helper function to instantiate Tokenizer&quot;&quot;&quot;
+ if token_type == &quot;bpe&quot;:
+ if bpemodel is None:
+ raise ValueError(&#x27;bpemodel is required if token_type = &quot;bpe&quot;&#x27;)
+
+ if remove_non_linguistic_symbols:
+ raise RuntimeError(
+ &quot;remove_non_linguistic_symbols is not implemented for token_type=bpe&quot;
+ )
+ return SentencepiecesTokenizer(bpemodel)
+
+ elif token_type == &quot;word&quot;:
+ if remove_non_linguistic_symbols and non_linguistic_symbols is not None:
+ return WordTokenizer(
+ delimiter=delimiter,
+ non_linguistic_symbols=non_linguistic_symbols,
+ remove_non_linguistic_symbols=True,
+ )
+ else:
+ return WordTokenizer(delimiter=delimiter)
+
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/tokenizer/build_tokenizer.py#L13" target="_blank" class="src-more">View full source on GitHub →</a></details></div>
+<div class="api-detail" id="e316"><span class="dtag badge-c">class</span><h2>CharTokenizer</h2><div class="dmod">funasr.tokenizer.char_tokenizer · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/tokenizer/char_tokenizer.py#L13" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p class="muted">No documentation yet.</p></div><details class="src-block"><summary>📄 Source code</summary><pre><code>class CharTokenizer(BaseTokenizer):
+ def __init__(
+ self,
+ non_linguistic_symbols: Union[Path, str, Iterable[str]] = None,
+ space_symbol: str = &quot;&lt;space&gt;&quot;,
+ remove_non_linguistic_symbols: bool = False,
+ split_with_space: bool = False,
+ seg_dict: str = None,
+ **kwargs,
+ ):
+ &quot;&quot;&quot;Initialize CharTokenizer.
+
+ Args:
+ non_linguistic_symbols: TODO.
+ space_symbol: TODO.
+ remove_non_linguistic_symbols: TODO.
+ split_with_space: TODO.
+ seg_dict: TODO.
+ **kwargs: Additional keyword arguments.
+ &quot;&quot;&quot;
+ super().__init__(**kwargs)
+ self.space_symbol = space_symbol
+ if non_linguistic_symbols is None:
+ self.non_linguistic_symbols = set()
+ elif isinstance(non_linguistic_symbols, (Path, str)):
+ non_linguistic_symbols = Path(non_linguistic_symbols)
+ try:
+ with non_linguistic_symbols.open(&quot;r&quot;, encoding=&quot;utf-8&quot;) as f:
+ self.non_linguistic_symbols = set(line.rstrip() for line in f)
+ except FileNotFoundError:
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/tokenizer/char_tokenizer.py#L13" target="_blank" class="src-more">View full source on GitHub →</a></details><h3 class="mt">Methods</h3><div class="mblk"><code>.text2tokens(line)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/tokenizer/char_tokenizer.py#L63" target="_blank" class="gh-link">L63</a><div class="mdoc"><p>Text2tokens.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>line</code> — TODO.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def text2tokens(self, line: Union[str, list]) -&gt; List[str]:
+ # if self.split_with_space:
+ &quot;&quot;&quot;Text2tokens.
+
+ Args:
+ line: TODO.
+ &quot;&quot;&quot;
+ if self.seg_dict is not None:
+ tokens = line.strip().split(&quot; &quot;)
+ tokens = seg_tokenize(tokens, self.seg_dict)
+ else:
+ tokens = []
+ while len(line) != 0:
+ for w in self.non_linguistic_symbols:
+ if line.startswith(w):
+ if not self.remove_non_linguistic_symbols:
+ tokens.append(line[: len(w)])
+ line = line[len(w) :]
+ break
+ else:
+ t = line[0]
+ if t == &quot; &quot;:
+ # t = &quot;&lt;space&gt;&quot;
+ line = line[1:]
+ continue
+ tokens.append(t)
+ line = line[1:]
+ return tokens
+</code></pre></details></div><div class="mblk"><code>.tokens2text(tokens)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/tokenizer/char_tokenizer.py#L92" target="_blank" class="gh-link">L92</a><div class="mdoc"><p>Tokens2text.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>tokens</code> — TODO.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def tokens2text(self, tokens: Iterable[str]) -&gt; str:
+ &quot;&quot;&quot;Tokens2text.
+
+ Args:
+ tokens: TODO.
+ &quot;&quot;&quot;
+ tokens = [t if t != self.space_symbol else &quot; &quot; for t in tokens]
+ return &quot;&quot;.join(tokens)
+</code></pre></details></div></div>
+<div class="api-detail" id="e317"><span class="dtag badge-m">method</span><h2><code>CharTokenizer.text2tokens(line)</code></h2><div class="dmod">funasr.tokenizer.char_tokenizer.CharTokenizer · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/tokenizer/char_tokenizer.py#L63" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Text2tokens.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>line</code> — TODO.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def text2tokens(self, line: Union[str, list]) -&gt; List[str]:
+ # if self.split_with_space:
+ &quot;&quot;&quot;Text2tokens.
+
+ Args:
+ line: TODO.
+ &quot;&quot;&quot;
+ if self.seg_dict is not None:
+ tokens = line.strip().split(&quot; &quot;)
+ tokens = seg_tokenize(tokens, self.seg_dict)
+ else:
+ tokens = []
+ while len(line) != 0:
+ for w in self.non_linguistic_symbols:
+ if line.startswith(w):
+ if not self.remove_non_linguistic_symbols:
+ tokens.append(line[: len(w)])
+ line = line[len(w) :]
+ break
+ else:
+ t = line[0]
+ if t == &quot; &quot;:
+ # t = &quot;&lt;space&gt;&quot;
+ line = line[1:]
+ continue
+ tokens.append(t)
+ line = line[1:]
+ return tokens
+</code></pre></details></div>
+<div class="api-detail" id="e318"><span class="dtag badge-m">method</span><h2><code>CharTokenizer.tokens2text(tokens)</code></h2><div class="dmod">funasr.tokenizer.char_tokenizer.CharTokenizer · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/tokenizer/char_tokenizer.py#L92" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Tokens2text.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>tokens</code> — TODO.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def tokens2text(self, tokens: Iterable[str]) -&gt; str:
+ &quot;&quot;&quot;Tokens2text.
+
+ Args:
+ tokens: TODO.
+ &quot;&quot;&quot;
+ tokens = [t if t != self.space_symbol else &quot; &quot; for t in tokens]
+ return &quot;&quot;.join(tokens)
+</code></pre></details></div>
+<div class="api-detail" id="e319"><span class="dtag badge-f">function</span><h2><code>load_seg_dict(seg_dict_file)</code></h2><div class="dmod">funasr.tokenizer.char_tokenizer · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/tokenizer/char_tokenizer.py#L102" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Load seg dict.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>seg_dict_file</code> — TODO.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code>def load_seg_dict(seg_dict_file):
+ &quot;&quot;&quot;Load seg dict.
+
+ Args:
+ seg_dict_file: TODO.
+ &quot;&quot;&quot;
+ seg_dict = {}
+ assert isinstance(seg_dict_file, str)
+ with open(seg_dict_file, &quot;r&quot;, encoding=&quot;utf8&quot;) as f:
+ lines = f.readlines()
+ for line in lines:
+ s = line.strip().split()
+ key = s[0]
+ value = s[1:]
+ seg_dict[key] = &quot; &quot;.join(value)
+ return seg_dict
+</code></pre></details></div>
+<div class="api-detail" id="e320"><span class="dtag badge-f">function</span><h2><code>seg_tokenize(txt, seg_dict)</code></h2><div class="dmod">funasr.tokenizer.char_tokenizer · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/tokenizer/char_tokenizer.py#L120" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Seg tokenize.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>txt</code> — TODO.</li>
+<li><code>seg_dict</code> — TODO.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code>def seg_tokenize(txt, seg_dict):
+ # pattern = re.compile(r&#x27;^[\u4E00-\u9FA50-9]+$&#x27;)
+ &quot;&quot;&quot;Seg tokenize.
+
+ Args:
+ txt: TODO.
+ seg_dict: TODO.
+ &quot;&quot;&quot;
+ pattern = re.compile(r&quot;([\u4E00-\u9FA5A-Za-z0-9])&quot;)
+ out_txt = &quot;&quot;
+ for word in txt:
+ word = word.lower()
+ if word in seg_dict:
+ out_txt += seg_dict[word] + &quot; &quot;
+ else:
+ if pattern.match(word):
+ for char in word:
+ if char in seg_dict:
+ out_txt += seg_dict[char] + &quot; &quot;
+ else:
+ out_txt += &quot;&lt;unk&gt;&quot; + &quot; &quot;
+ else:
+ out_txt += &quot;&lt;unk&gt;&quot; + &quot; &quot;
+ return out_txt.strip().split()
+</code></pre></details></div>
+<div class="api-detail" id="e321"><span class="dtag badge-c">class</span><h2>TextCleaner</h2><div class="dmod">funasr.tokenizer.cleaner · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/tokenizer/cleaner.py#L13" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Text cleaner.</p>
+<h4>Examples:</h4>
+<p>&gt;&gt;&gt; cleaner = TextCleaner(&quot;tacotron&quot;)</p>
+<p>&gt;&gt;&gt; cleaner(&quot;(Hello-World); &amp; jr. &amp; dr.&quot;)</p>
+<p>&#x27;HELLO WORLD, AND JUNIOR AND DOCTOR&#x27;</p>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code>class TextCleaner:
+ &quot;&quot;&quot;Text cleaner.
+
+ Examples:
+ &gt;&gt;&gt; cleaner = TextCleaner(&quot;tacotron&quot;)
+ &gt;&gt;&gt; cleaner(&quot;(Hello-World); &amp; jr. &amp; dr.&quot;)
+ &#x27;HELLO WORLD, AND JUNIOR AND DOCTOR&#x27;
+
+ &quot;&quot;&quot;
+
+ def __init__(self, cleaner_types: Collection[str] = None):
+
+ &quot;&quot;&quot;Initialize TextCleaner.
+
+ Args:
+ cleaner_types: TODO.
+ &quot;&quot;&quot;
+ if cleaner_types is None:
+ self.cleaner_types = []
+ elif isinstance(cleaner_types, str):
+ self.cleaner_types = [cleaner_types]
+ else:
+ self.cleaner_types = list(cleaner_types)
+
+ def __call__(self, text: str) -&gt; str:
+ &quot;&quot;&quot;Internal: call .
+
+ Args:
+ text: Text tensor or string input.
+ &quot;&quot;&quot;
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/tokenizer/cleaner.py#L13" target="_blank" class="src-more">View full source on GitHub →</a></details></div>
+<div class="api-detail" id="e322"><span class="dtag badge-f">function</span><h2><code>HuggingfaceTokenizer(init_param_path, **kwargs)</code></h2><div class="dmod">funasr.tokenizer.hf_tokenizer · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/tokenizer/hf_tokenizer.py#L5" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Huggingfacetokenizer.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>init_param_path</code> — TODO.</li>
+<li><code>**kwargs</code> — Additional keyword arguments.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code>def HuggingfaceTokenizer(init_param_path, **kwargs):
+ &quot;&quot;&quot;Huggingfacetokenizer.
+
+ Args:
+ init_param_path: TODO.
+ **kwargs: Additional keyword arguments.
+ &quot;&quot;&quot;
+ try:
+ from transformers import AutoTokenizer
+ except Exception as e:
+ raise ImportError(
+ &quot;HuggingfaceTokenizer requires &#x27;transformers&#x27;. &quot;
+ &quot;Please install it with: pip install -U transformers&quot;
+ ) from e
+
+ tokenizer = AutoTokenizer.from_pretrained(init_param_path)
+
+ return tokenizer
+</code></pre></details></div>
+<div class="api-detail" id="e323"><span class="dtag badge-c">class</span><h2>KoreanCleaner</h2><div class="dmod">funasr.tokenizer.korean_cleaner · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/tokenizer/korean_cleaner.py#L6" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p class="muted">No documentation yet.</p></div><details class="src-block"><summary>📄 Source code</summary><pre><code>class KoreanCleaner:
+ @classmethod
+ def _normalize_numbers(cls, text):
+ &quot;&quot;&quot;Internal: normalize numbers.
+
+ Args:
+ text: Text tensor or string input.
+ &quot;&quot;&quot;
+ number_to_kor = {
+ &quot;0&quot;: &quot;영&quot;,
+ &quot;1&quot;: &quot;일&quot;,
+ &quot;2&quot;: &quot;이&quot;,
+ &quot;3&quot;: &quot;삼&quot;,
+ &quot;4&quot;: &quot;사&quot;,
+ &quot;5&quot;: &quot;오&quot;,
+ &quot;6&quot;: &quot;육&quot;,
+ &quot;7&quot;: &quot;칠&quot;,
+ &quot;8&quot;: &quot;팔&quot;,
+ &quot;9&quot;: &quot;구&quot;,
+ }
+ new_text = &quot;&quot;.join(
+ number_to_kor[char] if char in number_to_kor.keys() else char for char in text
+ )
+ return new_text
+
+ @classmethod
+ def _normalize_english_text(cls, text):
+ &quot;&quot;&quot;Internal: normalize english text.
+
+ Args:
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/tokenizer/korean_cleaner.py#L6" target="_blank" class="src-more">View full source on GitHub →</a></details><h3 class="mt">Methods</h3><div class="mblk"><code>.normalize_text(cls, text)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/tokenizer/korean_cleaner.py#L75" target="_blank" class="gh-link">L75</a><div class="mdoc"><p>Normalize text.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>text</code> — Text tensor or string input.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def normalize_text(cls, text):
+ # stage 0 : text strip
+ &quot;&quot;&quot;Normalize text.
+
+ Args:
+ text: Text tensor or string input.
+ &quot;&quot;&quot;
+ text = text.strip()
+
+ # stage 1 : normalize numbers
+ text = cls._normalize_numbers(text)
+
+ # stage 2 : normalize english text
+ text = cls._normalize_english_text(text)
+ return text
+</code></pre></details></div></div>
+<div class="api-detail" id="e324"><span class="dtag badge-m">method</span><h2><code>KoreanCleaner.normalize_text(cls, text)</code></h2><div class="dmod">funasr.tokenizer.korean_cleaner.KoreanCleaner · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/tokenizer/korean_cleaner.py#L75" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Normalize text.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>text</code> — Text tensor or string input.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def normalize_text(cls, text):
+ # stage 0 : text strip
+ &quot;&quot;&quot;Normalize text.
+
+ Args:
+ text: Text tensor or string input.
+ &quot;&quot;&quot;
+ text = text.strip()
+
+ # stage 1 : normalize numbers
+ text = cls._normalize_numbers(text)
+
+ # stage 2 : normalize english text
+ text = cls._normalize_english_text(text)
+ return text
+</code></pre></details></div>
+<div class="api-detail" id="e325"><span class="dtag badge-f">function</span><h2><code>split_by_space(text)</code></h2><div class="dmod">funasr.tokenizer.phoneme_tokenizer · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/tokenizer/phoneme_tokenizer.py#L45" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Split by space.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>text</code> — Text tensor or string input.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code>def split_by_space(text) -&gt; List[str]:
+ &quot;&quot;&quot;Split by space.
+
+ Args:
+ text: Text tensor or string input.
+ &quot;&quot;&quot;
+ if &quot; &quot; in text:
+ text = text.replace(&quot; &quot;, &quot; &lt;space&gt; &quot;)
+ return [c.replace(&quot;&lt;space&gt;&quot;, &quot; &quot;) for c in text.split(&quot; &quot;)]
+ else:
+ return text.split(&quot; &quot;)
+</code></pre></details></div>
+<div class="api-detail" id="e326"><span class="dtag badge-f">function</span><h2><code>pyopenjtalk_g2p(text)</code></h2><div class="dmod">funasr.tokenizer.phoneme_tokenizer · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/tokenizer/phoneme_tokenizer.py#L58" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Pyopenjtalk g2p.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>text</code> — Text tensor or string input.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code>def pyopenjtalk_g2p(text) -&gt; List[str]:
+ &quot;&quot;&quot;Pyopenjtalk g2p.
+
+ Args:
+ text: Text tensor or string input.
+ &quot;&quot;&quot;
+ import pyopenjtalk
+
+ # phones is a str object separated by space
+ phones = pyopenjtalk.g2p(text, kana=False)
+ phones = phones.split(&quot; &quot;)
+ return phones
+</code></pre></details></div>
+<div class="api-detail" id="e327"><span class="dtag badge-f">function</span><h2><code>pyopenjtalk_g2p_accent(text)</code></h2><div class="dmod">funasr.tokenizer.phoneme_tokenizer · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/tokenizer/phoneme_tokenizer.py#L72" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Pyopenjtalk g2p accent.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>text</code> — Text tensor or string input.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code>def pyopenjtalk_g2p_accent(text) -&gt; List[str]:
+ &quot;&quot;&quot;Pyopenjtalk g2p accent.
+
+ Args:
+ text: Text tensor or string input.
+ &quot;&quot;&quot;
+ import pyopenjtalk
+ import re
+
+ phones = []
+ for labels in pyopenjtalk.run_frontend(text)[1]:
+ p = re.findall(r&quot;\-(.*?)\+.*?\/A:([0-9\-]+).*?\/F:.*?_([0-9]+)&quot;, labels)
+ if len(p) == 1:
+ phones += [p[0][0], p[0][2], p[0][1]]
+ return phones
+</code></pre></details></div>
+<div class="api-detail" id="e328"><span class="dtag badge-f">function</span><h2><code>pyopenjtalk_g2p_accent_with_pause(text)</code></h2><div class="dmod">funasr.tokenizer.phoneme_tokenizer · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/tokenizer/phoneme_tokenizer.py#L89" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Pyopenjtalk g2p accent with pause.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>text</code> — Text tensor or string input.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code>def pyopenjtalk_g2p_accent_with_pause(text) -&gt; List[str]:
+ &quot;&quot;&quot;Pyopenjtalk g2p accent with pause.
+
+ Args:
+ text: Text tensor or string input.
+ &quot;&quot;&quot;
+ import pyopenjtalk
+ import re
+
+ phones = []
+ for labels in pyopenjtalk.run_frontend(text)[1]:
+ if labels.split(&quot;-&quot;)[1].split(&quot;+&quot;)[0] == &quot;pau&quot;:
+ phones += [&quot;pau&quot;]
+ continue
+ p = re.findall(r&quot;\-(.*?)\+.*?\/A:([0-9\-]+).*?\/F:.*?_([0-9]+)&quot;, labels)
+ if len(p) == 1:
+ phones += [p[0][0], p[0][2], p[0][1]]
+ return phones
+</code></pre></details></div>
+<div class="api-detail" id="e329"><span class="dtag badge-f">function</span><h2><code>pyopenjtalk_g2p_kana(text)</code></h2><div class="dmod">funasr.tokenizer.phoneme_tokenizer · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/tokenizer/phoneme_tokenizer.py#L109" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Pyopenjtalk g2p kana.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>text</code> — Text tensor or string input.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code>def pyopenjtalk_g2p_kana(text) -&gt; List[str]:
+ &quot;&quot;&quot;Pyopenjtalk g2p kana.
+
+ Args:
+ text: Text tensor or string input.
+ &quot;&quot;&quot;
+ import pyopenjtalk
+
+ kanas = pyopenjtalk.g2p(text, kana=True)
+ return list(kanas)
+</code></pre></details></div>
+<div class="api-detail" id="e330"><span class="dtag badge-f">function</span><h2><code>pyopenjtalk_g2p_prosody(text, drop_unvoiced_vowels)</code></h2><div class="dmod">funasr.tokenizer.phoneme_tokenizer · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/tokenizer/phoneme_tokenizer.py#L121" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Extract phoneme + prosoody symbol sequence from input full-context labels.</p>
+<p>The algorithm is based on `Prosodic features control by symbols as input of</p>
+<ul>
+<li><code>sequence</code> — to-sequence acoustic modeling for neural TTS`_ with some r9y9&#x27;s tweaks.</li>
+</ul>
+<h4>Args:</h4>
+<ul>
+<li><code>text (str)</code> — Input text.</li>
+<li><code>drop_unvoiced_vowels (bool)</code> — whether to drop unvoiced vowels.</li>
+</ul>
+<h4>Returns:</h4>
+<p>List[str]: List of phoneme + prosody symbols.</p>
+<h4>Examples:</h4>
+<p>&gt;&gt;&gt; from funasr.tokenizer.phoneme_tokenizer import pyopenjtalk_g2p_prosody</p>
+<p>&gt;&gt;&gt; pyopenjtalk_g2p_prosody(&quot;こんにちは。&quot;)</p>
+<p>[&#x27;^&#x27;, &#x27;k&#x27;, &#x27;o&#x27;, &#x27;[&#x27;, &#x27;N&#x27;, &#x27;n&#x27;, &#x27;i&#x27;, &#x27;ch&#x27;, &#x27;i&#x27;, &#x27;w&#x27;, &#x27;a&#x27;, &#x27;$&#x27;]</p>
+<p>.. _`Prosodic features control by symbols as input of sequence-to-sequence acoustic</p>
+<p>modeling for neural TTS`: https://doi.org/10.1587/transinf.2020EDP7104</p>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code>def pyopenjtalk_g2p_prosody(text: str, drop_unvoiced_vowels: bool = True) -&gt; List[str]:
+ &quot;&quot;&quot;Extract phoneme + prosoody symbol sequence from input full-context labels.
+
+ The algorithm is based on `Prosodic features control by symbols as input of
+ sequence-to-sequence acoustic modeling for neural TTS`_ with some r9y9&#x27;s tweaks.
+
+ Args:
+ text (str): Input text.
+ drop_unvoiced_vowels (bool): whether to drop unvoiced vowels.
+
+ Returns:
+ List[str]: List of phoneme + prosody symbols.
+
+ Examples:
+ &gt;&gt;&gt; from funasr.tokenizer.phoneme_tokenizer import pyopenjtalk_g2p_prosody
+ &gt;&gt;&gt; pyopenjtalk_g2p_prosody(&quot;こんにちは。&quot;)
+ [&#x27;^&#x27;, &#x27;k&#x27;, &#x27;o&#x27;, &#x27;[&#x27;, &#x27;N&#x27;, &#x27;n&#x27;, &#x27;i&#x27;, &#x27;ch&#x27;, &#x27;i&#x27;, &#x27;w&#x27;, &#x27;a&#x27;, &#x27;$&#x27;]
+
+ .. _`Prosodic features control by symbols as input of sequence-to-sequence acoustic
+ modeling for neural TTS`: https://doi.org/10.1587/transinf.2020EDP7104
+
+ &quot;&quot;&quot;
+ import pyopenjtalk
+
+ labels = pyopenjtalk.run_frontend(text)[1]
+ N = len(labels)
+
+ phones = []
+ for n in range(N):
+ lab_curr = labels[n]
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/tokenizer/phoneme_tokenizer.py#L121" target="_blank" class="src-more">View full source on GitHub →</a></details></div>
+<div class="api-detail" id="e331"><span class="dtag badge-f">function</span><h2><code>pypinyin_g2p(text)</code></h2><div class="dmod">funasr.tokenizer.phoneme_tokenizer · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/tokenizer/phoneme_tokenizer.py#L213" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Pypinyin g2p.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>text</code> — Text tensor or string input.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code>def pypinyin_g2p(text) -&gt; List[str]:
+ &quot;&quot;&quot;Pypinyin g2p.
+
+ Args:
+ text: Text tensor or string input.
+ &quot;&quot;&quot;
+ from pypinyin import pinyin
+ from pypinyin import Style
+
+ phones = [phone[0] for phone in pinyin(text, style=Style.TONE3)]
+ return phones
+</code></pre></details></div>
+<div class="api-detail" id="e332"><span class="dtag badge-f">function</span><h2><code>pypinyin_g2p_phone(text)</code></h2><div class="dmod">funasr.tokenizer.phoneme_tokenizer · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/tokenizer/phoneme_tokenizer.py#L226" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Pypinyin g2p phone.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>text</code> — Text tensor or string input.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code>def pypinyin_g2p_phone(text) -&gt; List[str]:
+ &quot;&quot;&quot;Pypinyin g2p phone.
+
+ Args:
+ text: Text tensor or string input.
+ &quot;&quot;&quot;
+ from pypinyin import pinyin
+ from pypinyin import Style
+ from pypinyin.style._utils import get_finals
+ from pypinyin.style._utils import get_initials
+
+ phones = [
+ p
+ for phone in pinyin(text, style=Style.TONE3)
+ for p in [
+ get_initials(phone[0], strict=True),
+ get_finals(phone[0], strict=True),
+ ]
+ if len(p) != 0
+ ]
+ return phones
+</code></pre></details></div>
+<div class="api-detail" id="e333"><span class="dtag badge-c">class</span><h2>G2p_en</h2><div class="dmod">funasr.tokenizer.phoneme_tokenizer · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/tokenizer/phoneme_tokenizer.py#L249" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>On behalf of g2p_en.G2p.</p>
+<p>g2p_en.G2p isn&#x27;t pickalable and it can&#x27;t be copied to the other processes</p>
+<p>via multiprocessing module.</p>
+<p>As a workaround, g2p_en.G2p is instantiated upon calling this class.</p>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code>class G2p_en:
+ &quot;&quot;&quot;On behalf of g2p_en.G2p.
+
+ g2p_en.G2p isn&#x27;t pickalable and it can&#x27;t be copied to the other processes
+ via multiprocessing module.
+ As a workaround, g2p_en.G2p is instantiated upon calling this class.
+
+ &quot;&quot;&quot;
+
+ def __init__(self, no_space: bool = False):
+ &quot;&quot;&quot;Initialize G2p_en.
+
+ Args:
+ no_space: TODO.
+ &quot;&quot;&quot;
+ self.no_space = no_space
+ self.g2p = None
+
+ def __call__(self, text) -&gt; List[str]:
+ &quot;&quot;&quot;Internal: call .
+
+ Args:
+ text: Text tensor or string input.
+ &quot;&quot;&quot;
+ if self.g2p is None:
+ self.g2p = g2p_en.G2p()
+
+ phones = self.g2p(text)
+ if self.no_space:
+ # remove space which represents word serapater
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/tokenizer/phoneme_tokenizer.py#L249" target="_blank" class="src-more">View full source on GitHub →</a></details></div>
+<div class="api-detail" id="e334"><span class="dtag badge-c">class</span><h2>G2pk</h2><div class="dmod">funasr.tokenizer.phoneme_tokenizer · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/tokenizer/phoneme_tokenizer.py#L283" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>On behalf of g2pk.G2p.</p>
+<p>g2pk.G2p isn&#x27;t pickalable and it can&#x27;t be copied to the other processes</p>
+<p>via multiprocessing module.</p>
+<p>As a workaround, g2pk.G2p is instantiated upon calling this class.</p>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code>class G2pk:
+ &quot;&quot;&quot;On behalf of g2pk.G2p.
+
+ g2pk.G2p isn&#x27;t pickalable and it can&#x27;t be copied to the other processes
+ via multiprocessing module.
+ As a workaround, g2pk.G2p is instantiated upon calling this class.
+
+ &quot;&quot;&quot;
+
+ def __init__(self, descritive=False, group_vowels=False, to_syl=False, no_space=False):
+ &quot;&quot;&quot;Initialize G2pk.
+
+ Args:
+ descritive: TODO.
+ group_vowels: TODO.
+ to_syl: TODO.
+ no_space: TODO.
+ &quot;&quot;&quot;
+ self.descritive = descritive
+ self.group_vowels = group_vowels
+ self.to_syl = to_syl
+ self.no_space = no_space
+ self.g2p = None
+
+ def __call__(self, text) -&gt; List[str]:
+ &quot;&quot;&quot;Internal: call .
+
+ Args:
+ text: Text tensor or string input.
+ &quot;&quot;&quot;
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/tokenizer/phoneme_tokenizer.py#L283" target="_blank" class="src-more">View full source on GitHub →</a></details></div>
+<div class="api-detail" id="e335"><span class="dtag badge-c">class</span><h2>Jaso</h2><div class="dmod">funasr.tokenizer.phoneme_tokenizer · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/tokenizer/phoneme_tokenizer.py#L332" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p class="muted">No documentation yet.</p></div><details class="src-block"><summary>📄 Source code</summary><pre><code>class Jaso:
+ PUNC = &quot;!&#x27;(),-.:;?&quot;
+ SPACE = &quot; &quot;
+
+ JAMO_LEADS = &quot;&quot;.join([chr(_) for _ in range(0x1100, 0x1113)])
+ JAMO_VOWELS = &quot;&quot;.join([chr(_) for _ in range(0x1161, 0x1176)])
+ JAMO_TAILS = &quot;&quot;.join([chr(_) for _ in range(0x11A8, 0x11C3)])
+
+ VALID_CHARS = JAMO_LEADS + JAMO_VOWELS + JAMO_TAILS + PUNC + SPACE
+
+ def __init__(self, space_symbol=&quot; &quot;, no_space=False):
+ &quot;&quot;&quot;Initialize Jaso.
+
+ Args:
+ space_symbol: TODO.
+ no_space: TODO.
+ &quot;&quot;&quot;
+ self.space_symbol = space_symbol
+ self.no_space = no_space
+
+ def _text_to_jaso(self, line: str) -&gt; List[str]:
+ &quot;&quot;&quot;Internal: text to jaso.
+
+ Args:
+ line: TODO.
+ &quot;&quot;&quot;
+ jasos = list(jamo.hangul_to_jamo(line))
+ return jasos
+
+ def _remove_non_korean_characters(self, tokens):
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/tokenizer/phoneme_tokenizer.py#L332" target="_blank" class="src-more">View full source on GitHub →</a></details></div>
+<div class="api-detail" id="e336"><span class="dtag badge-c">class</span><h2>Phonemizer</h2><div class="dmod">funasr.tokenizer.phoneme_tokenizer · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/tokenizer/phoneme_tokenizer.py#L386" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Phonemizer module for various languages.</p>
+<p>This is wrapper module of https://github.com/bootphon/phonemizer.</p>
+<p>You can define various g2p modules by specifying options for phonemizer.</p>
+<p>See available options:</p>
+<p>https://github.com/bootphon/phonemizer/blob/master/phonemizer/phonemize.py#L32</p>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code>class Phonemizer:
+ &quot;&quot;&quot;Phonemizer module for various languages.
+
+ This is wrapper module of https://github.com/bootphon/phonemizer.
+ You can define various g2p modules by specifying options for phonemizer.
+
+ See available options:
+ https://github.com/bootphon/phonemizer/blob/master/phonemizer/phonemize.py#L32
+
+ &quot;&quot;&quot;
+
+ def __init__(
+ self,
+ backend,
+ word_separator: Optional[str] = None,
+ syllable_separator: Optional[str] = None,
+ phone_separator: Optional[str] = &quot; &quot;,
+ strip=False,
+ split_by_single_token: bool = False,
+ **phonemizer_kwargs,
+ ):
+ # delayed import
+ &quot;&quot;&quot;Initialize Phonemizer.
+
+ Args:
+ backend: TODO.
+ word_separator: TODO.
+ syllable_separator: TODO.
+ phone_separator: TODO.
+ strip: TODO.
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/tokenizer/phoneme_tokenizer.py#L386" target="_blank" class="src-more">View full source on GitHub →</a></details></div>
+<div class="api-detail" id="e337"><span class="dtag badge-c">class</span><h2>PhonemeTokenizer</h2><div class="dmod">funasr.tokenizer.phoneme_tokenizer · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/tokenizer/phoneme_tokenizer.py#L458" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p class="muted">No documentation yet.</p></div><details class="src-block"><summary>📄 Source code</summary><pre><code>class PhonemeTokenizer(AbsTokenizer):
+ def __init__(
+ self,
+ g2p_type: Union[None, str],
+ non_linguistic_symbols: Union[Path, str, Iterable[str]] = None,
+ space_symbol: str = &quot;&lt;space&gt;&quot;,
+ remove_non_linguistic_symbols: bool = False,
+ ):
+ &quot;&quot;&quot;Initialize PhonemeTokenizer.
+
+ Args:
+ g2p_type: TODO.
+ non_linguistic_symbols: TODO.
+ space_symbol: TODO.
+ remove_non_linguistic_symbols: TODO.
+ &quot;&quot;&quot;
+ if g2p_type is None:
+ self.g2p = split_by_space
+ elif g2p_type == &quot;g2p_en&quot;:
+ self.g2p = G2p_en(no_space=False)
+ elif g2p_type == &quot;g2p_en_no_space&quot;:
+ self.g2p = G2p_en(no_space=True)
+ elif g2p_type == &quot;pyopenjtalk&quot;:
+ self.g2p = pyopenjtalk_g2p
+ elif g2p_type == &quot;pyopenjtalk_kana&quot;:
+ self.g2p = pyopenjtalk_g2p_kana
+ elif g2p_type == &quot;pyopenjtalk_accent&quot;:
+ self.g2p = pyopenjtalk_g2p_accent
+ elif g2p_type == &quot;pyopenjtalk_accent_with_pause&quot;:
+ self.g2p = pyopenjtalk_g2p_accent_with_pause
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/tokenizer/phoneme_tokenizer.py#L458" target="_blank" class="src-more">View full source on GitHub →</a></details><h3 class="mt">Methods</h3><div class="mblk"><code>.text2tokens(line)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/tokenizer/phoneme_tokenizer.py#L614" target="_blank" class="gh-link">L614</a><div class="mdoc"><p>Text2tokens.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>line</code> — TODO.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def text2tokens(self, line: str) -&gt; List[str]:
+ &quot;&quot;&quot;Text2tokens.
+
+ Args:
+ line: TODO.
+ &quot;&quot;&quot;
+ tokens = []
+ while len(line) != 0:
+ for w in self.non_linguistic_symbols:
+ if line.startswith(w):
+ if not self.remove_non_linguistic_symbols:
+ tokens.append(line[: len(w)])
+ line = line[len(w) :]
+ break
+ else:
+ t = line[0]
+ tokens.append(t)
+ line = line[1:]
+
+ line = &quot;&quot;.join(tokens)
+ tokens = self.g2p(line)
+ return tokens
+</code></pre></details></div><div class="mblk"><code>.tokens2text(tokens)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/tokenizer/phoneme_tokenizer.py#L637" target="_blank" class="gh-link">L637</a><div class="mdoc"><p>Tokens2text.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>tokens</code> — TODO.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def tokens2text(self, tokens: Iterable[str]) -&gt; str:
+ # phoneme type is not invertible
+ &quot;&quot;&quot;Tokens2text.
+
+ Args:
+ tokens: TODO.
+ &quot;&quot;&quot;
+ return &quot;&quot;.join(tokens)
+</code></pre></details></div></div>
+<div class="api-detail" id="e338"><span class="dtag badge-m">method</span><h2><code>PhonemeTokenizer.text2tokens(line)</code></h2><div class="dmod">funasr.tokenizer.phoneme_tokenizer.PhonemeTokenizer · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/tokenizer/phoneme_tokenizer.py#L614" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Text2tokens.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>line</code> — TODO.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def text2tokens(self, line: str) -&gt; List[str]:
+ &quot;&quot;&quot;Text2tokens.
+
+ Args:
+ line: TODO.
+ &quot;&quot;&quot;
+ tokens = []
+ while len(line) != 0:
+ for w in self.non_linguistic_symbols:
+ if line.startswith(w):
+ if not self.remove_non_linguistic_symbols:
+ tokens.append(line[: len(w)])
+ line = line[len(w) :]
+ break
+ else:
+ t = line[0]
+ tokens.append(t)
+ line = line[1:]
+
+ line = &quot;&quot;.join(tokens)
+ tokens = self.g2p(line)
+ return tokens
+</code></pre></details></div>
+<div class="api-detail" id="e339"><span class="dtag badge-m">method</span><h2><code>PhonemeTokenizer.tokens2text(tokens)</code></h2><div class="dmod">funasr.tokenizer.phoneme_tokenizer.PhonemeTokenizer · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/tokenizer/phoneme_tokenizer.py#L637" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Tokens2text.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>tokens</code> — TODO.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def tokens2text(self, tokens: Iterable[str]) -&gt; str:
+ # phoneme type is not invertible
+ &quot;&quot;&quot;Tokens2text.
+
+ Args:
+ tokens: TODO.
+ &quot;&quot;&quot;
+ return &quot;&quot;.join(tokens)
+</code></pre></details></div>
+<div class="api-detail" id="e340"><span class="dtag badge-c">class</span><h2>SentencepiecesTokenizer</h2><div class="dmod">funasr.tokenizer.sentencepiece_tokenizer · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/tokenizer/sentencepiece_tokenizer.py#L13" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p class="muted">No documentation yet.</p></div><details class="src-block"><summary>📄 Source code</summary><pre><code>class SentencepiecesTokenizer(BaseTokenizer):
+ def __init__(self, bpemodel: Union[Path, str], **kwargs):
+ &quot;&quot;&quot;Initialize SentencepiecesTokenizer.
+
+ Args:
+ bpemodel: TODO.
+ **kwargs: Additional keyword arguments.
+ &quot;&quot;&quot;
+ super().__init__(**kwargs)
+ self.bpemodel = str(bpemodel)
+ # NOTE(kamo):
+ # Don&#x27;t build SentencePieceProcessor in __init__()
+ # because it&#x27;s not picklable and it may cause following error,
+ # &quot;TypeError: can&#x27;t pickle SwigPyObject objects&quot;,
+ # when giving it as argument of &quot;multiprocessing.Process()&quot;.
+ self.sp = None
+ self._build_sentence_piece_processor()
+
+ def __repr__(self):
+ &quot;&quot;&quot;Internal: repr .&quot;&quot;&quot;
+ return f&#x27;{self.__class__.__name__}(model=&quot;{self.bpemodel}&quot;)&#x27;
+
+ def _build_sentence_piece_processor(self):
+ # Build SentencePieceProcessor lazily.
+ &quot;&quot;&quot;Internal: build sentence piece processor.&quot;&quot;&quot;
+ if self.sp is None:
+ self.sp = spm.SentencePieceProcessor()
+ self.sp.load(self.bpemodel)
+
+ def text2tokens(self, line: str) -&gt; List[str]:
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/tokenizer/sentencepiece_tokenizer.py#L13" target="_blank" class="src-more">View full source on GitHub →</a></details><h3 class="mt">Methods</h3><div class="mblk"><code>.text2tokens(line)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/tokenizer/sentencepiece_tokenizer.py#L42" target="_blank" class="gh-link">L42</a><div class="mdoc"><p>Text2tokens.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>line</code> — TODO.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def text2tokens(self, line: str) -&gt; List[str]:
+ &quot;&quot;&quot;Text2tokens.
+
+ Args:
+ line: TODO.
+ &quot;&quot;&quot;
+ self._build_sentence_piece_processor()
+ return self.sp.EncodeAsPieces(line)
+</code></pre></details></div><div class="mblk"><code>.tokens2text(tokens)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/tokenizer/sentencepiece_tokenizer.py#L51" target="_blank" class="gh-link">L51</a><div class="mdoc"><p>Tokens2text.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>tokens</code> — TODO.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def tokens2text(self, tokens: Iterable[str]) -&gt; str:
+ &quot;&quot;&quot;Tokens2text.
+
+ Args:
+ tokens: TODO.
+ &quot;&quot;&quot;
+ self._build_sentence_piece_processor()
+ return self.sp.DecodePieces(list(tokens))
+</code></pre></details></div><div class="mblk"><code>.encode(line, **kwargs)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/tokenizer/sentencepiece_tokenizer.py#L60" target="_blank" class="gh-link">L60</a><div class="mdoc"><p>Encode.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>line</code> — TODO.</li>
+<li><code>**kwargs</code> — Additional keyword arguments.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def encode(self, line: str, **kwargs) -&gt; List[int]:
+ &quot;&quot;&quot;Encode.
+
+ Args:
+ line: TODO.
+ **kwargs: Additional keyword arguments.
+ &quot;&quot;&quot;
+ self._build_sentence_piece_processor()
+ return self.sp.EncodeAsIds(line)
+</code></pre></details></div><div class="mblk"><code>.decode(line, **kwargs)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/tokenizer/sentencepiece_tokenizer.py#L70" target="_blank" class="gh-link">L70</a><div class="mdoc"><p>Decode.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>line</code> — TODO.</li>
+<li><code>**kwargs</code> — Additional keyword arguments.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def decode(self, line: List[int], **kwargs):
+ &quot;&quot;&quot;Decode.
+
+ Args:
+ line: TODO.
+ **kwargs: Additional keyword arguments.
+ &quot;&quot;&quot;
+ self._build_sentence_piece_processor()
+ return self.sp.DecodeIds(line)
+</code></pre></details></div><div class="mblk"><code>.get_vocab_size()</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/tokenizer/sentencepiece_tokenizer.py#L80" target="_blank" class="gh-link">L80</a><div class="mdoc"><p>Get vocab size.</p>
+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def get_vocab_size(self):
+ &quot;&quot;&quot;Get vocab size.&quot;&quot;&quot;
+ return self.sp.GetPieceSize()
+</code></pre></details></div><div class="mblk"><code>.ids2tokens(*args, **kwargs)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/tokenizer/sentencepiece_tokenizer.py#L84" target="_blank" class="gh-link">L84</a><div class="mdoc"><p>Ids2tokens.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>*args</code> — Variable positional arguments.</li>
+<li><code>**kwargs</code> — Additional keyword arguments.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def ids2tokens(self, *args, **kwargs):
+ &quot;&quot;&quot;Ids2tokens.
+
+ Args:
+ *args: Variable positional arguments.
+ **kwargs: Additional keyword arguments.
+ &quot;&quot;&quot;
+ return self.decode(*args, **kwargs)
+</code></pre></details></div><div class="mblk"><code>.tokens2ids(*args, **kwargs)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/tokenizer/sentencepiece_tokenizer.py#L93" target="_blank" class="gh-link">L93</a><div class="mdoc"><p>Tokens2ids.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>*args</code> — Variable positional arguments.</li>
+<li><code>**kwargs</code> — Additional keyword arguments.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def tokens2ids(self, *args, **kwargs):
+ &quot;&quot;&quot;Tokens2ids.
+
+ Args:
+ *args: Variable positional arguments.
+ **kwargs: Additional keyword arguments.
+ &quot;&quot;&quot;
+ return self.encode(*args, **kwargs)
+</code></pre></details></div></div>
+<div class="api-detail" id="e341"><span class="dtag badge-m">method</span><h2><code>SentencepiecesTokenizer.text2tokens(line)</code></h2><div class="dmod">funasr.tokenizer.sentencepiece_tokenizer.SentencepiecesTokenizer · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/tokenizer/sentencepiece_tokenizer.py#L42" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Text2tokens.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>line</code> — TODO.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def text2tokens(self, line: str) -&gt; List[str]:
+ &quot;&quot;&quot;Text2tokens.
+
+ Args:
+ line: TODO.
+ &quot;&quot;&quot;
+ self._build_sentence_piece_processor()
+ return self.sp.EncodeAsPieces(line)
+</code></pre></details></div>
+<div class="api-detail" id="e342"><span class="dtag badge-m">method</span><h2><code>SentencepiecesTokenizer.tokens2text(tokens)</code></h2><div class="dmod">funasr.tokenizer.sentencepiece_tokenizer.SentencepiecesTokenizer · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/tokenizer/sentencepiece_tokenizer.py#L51" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Tokens2text.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>tokens</code> — TODO.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def tokens2text(self, tokens: Iterable[str]) -&gt; str:
+ &quot;&quot;&quot;Tokens2text.
+
+ Args:
+ tokens: TODO.
+ &quot;&quot;&quot;
+ self._build_sentence_piece_processor()
+ return self.sp.DecodePieces(list(tokens))
+</code></pre></details></div>
+<div class="api-detail" id="e343"><span class="dtag badge-m">method</span><h2><code>SentencepiecesTokenizer.encode(line, **kwargs)</code></h2><div class="dmod">funasr.tokenizer.sentencepiece_tokenizer.SentencepiecesTokenizer · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/tokenizer/sentencepiece_tokenizer.py#L60" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Encode.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>line</code> — TODO.</li>
+<li><code>**kwargs</code> — Additional keyword arguments.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def encode(self, line: str, **kwargs) -&gt; List[int]:
+ &quot;&quot;&quot;Encode.
+
+ Args:
+ line: TODO.
+ **kwargs: Additional keyword arguments.
+ &quot;&quot;&quot;
+ self._build_sentence_piece_processor()
+ return self.sp.EncodeAsIds(line)
+</code></pre></details></div>
+<div class="api-detail" id="e344"><span class="dtag badge-m">method</span><h2><code>SentencepiecesTokenizer.decode(line, **kwargs)</code></h2><div class="dmod">funasr.tokenizer.sentencepiece_tokenizer.SentencepiecesTokenizer · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/tokenizer/sentencepiece_tokenizer.py#L70" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Decode.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>line</code> — TODO.</li>
+<li><code>**kwargs</code> — Additional keyword arguments.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def decode(self, line: List[int], **kwargs):
+ &quot;&quot;&quot;Decode.
+
+ Args:
+ line: TODO.
+ **kwargs: Additional keyword arguments.
+ &quot;&quot;&quot;
+ self._build_sentence_piece_processor()
+ return self.sp.DecodeIds(line)
+</code></pre></details></div>
+<div class="api-detail" id="e345"><span class="dtag badge-m">method</span><h2><code>SentencepiecesTokenizer.get_vocab_size()</code></h2><div class="dmod">funasr.tokenizer.sentencepiece_tokenizer.SentencepiecesTokenizer · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/tokenizer/sentencepiece_tokenizer.py#L80" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Get vocab size.</p>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def get_vocab_size(self):
+ &quot;&quot;&quot;Get vocab size.&quot;&quot;&quot;
+ return self.sp.GetPieceSize()
+</code></pre></details></div>
+<div class="api-detail" id="e346"><span class="dtag badge-m">method</span><h2><code>SentencepiecesTokenizer.ids2tokens(*args, **kwargs)</code></h2><div class="dmod">funasr.tokenizer.sentencepiece_tokenizer.SentencepiecesTokenizer · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/tokenizer/sentencepiece_tokenizer.py#L84" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Ids2tokens.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>*args</code> — Variable positional arguments.</li>
+<li><code>**kwargs</code> — Additional keyword arguments.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def ids2tokens(self, *args, **kwargs):
+ &quot;&quot;&quot;Ids2tokens.
+
+ Args:
+ *args: Variable positional arguments.
+ **kwargs: Additional keyword arguments.
+ &quot;&quot;&quot;
+ return self.decode(*args, **kwargs)
+</code></pre></details></div>
+<div class="api-detail" id="e347"><span class="dtag badge-m">method</span><h2><code>SentencepiecesTokenizer.tokens2ids(*args, **kwargs)</code></h2><div class="dmod">funasr.tokenizer.sentencepiece_tokenizer.SentencepiecesTokenizer · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/tokenizer/sentencepiece_tokenizer.py#L93" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Tokens2ids.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>*args</code> — Variable positional arguments.</li>
+<li><code>**kwargs</code> — Additional keyword arguments.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def tokens2ids(self, *args, **kwargs):
+ &quot;&quot;&quot;Tokens2ids.
+
+ Args:
+ *args: Variable positional arguments.
+ **kwargs: Additional keyword arguments.
+ &quot;&quot;&quot;
+ return self.encode(*args, **kwargs)
+</code></pre></details></div>
+<div class="api-detail" id="e348"><span class="dtag badge-c">class</span><h2>TokenIDConverter</h2><div class="dmod">funasr.tokenizer.token_id_converter · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/tokenizer/token_id_converter.py#L10" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p class="muted">No documentation yet.</p></div><details class="src-block"><summary>📄 Source code</summary><pre><code>class TokenIDConverter:
+ def __init__(
+ self,
+ token_list: Union[Path, str, Iterable[str]],
+ unk_symbol: str = &quot;&lt;unk&gt;&quot;,
+ ):
+
+ &quot;&quot;&quot;Initialize TokenIDConverter.
+
+ Args:
+ token_list: TODO.
+ unk_symbol: TODO.
+ &quot;&quot;&quot;
+ if isinstance(token_list, (Path, str)):
+ token_list = Path(token_list)
+ self.token_list_repr = str(token_list)
+ self.token_list: List[str] = []
+
+ with token_list.open(&quot;r&quot;, encoding=&quot;utf-8&quot;) as f:
+ for idx, line in enumerate(f):
+ line = line.rstrip()
+ self.token_list.append(line)
+
+ else:
+ self.token_list: List[str] = list(token_list)
+ self.token_list_repr = &quot;&quot;
+ for i, t in enumerate(self.token_list):
+ if i == 3:
+ break
+ self.token_list_repr += f&quot;{t}, &quot;
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/tokenizer/token_id_converter.py#L10" target="_blank" class="src-more">View full source on GitHub →</a></details><h3 class="mt">Methods</h3><div class="mblk"><code>.get_num_vocabulary_size()</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/tokenizer/token_id_converter.py#L53" target="_blank" class="gh-link">L53</a><div class="mdoc"><p>Get num vocabulary size.</p>
+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def get_num_vocabulary_size(self) -&gt; int:
+ &quot;&quot;&quot;Get num vocabulary size.&quot;&quot;&quot;
+ return len(self.token_list)
+</code></pre></details></div><div class="mblk"><code>.ids2tokens(integers)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/tokenizer/token_id_converter.py#L57" target="_blank" class="gh-link">L57</a><div class="mdoc"><p>Ids2tokens.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>integers</code> — TODO.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def ids2tokens(self, integers: Union[np.ndarray, Iterable[int]]) -&gt; List[str]:
+ &quot;&quot;&quot;Ids2tokens.
+
+ Args:
+ integers: TODO.
+ &quot;&quot;&quot;
+ if isinstance(integers, np.ndarray) and integers.ndim != 1:
+ raise ValueError(f&quot;Must be 1 dim ndarray, but got {integers.ndim}&quot;)
+ return [self.token_list[i] for i in integers]
+</code></pre></details></div><div class="mblk"><code>.tokens2ids(tokens)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/tokenizer/token_id_converter.py#L67" target="_blank" class="gh-link">L67</a><div class="mdoc"><p>Tokens2ids.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>tokens</code> — TODO.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def tokens2ids(self, tokens: Iterable[str]) -&gt; List[int]:
+ &quot;&quot;&quot;Tokens2ids.
+
+ Args:
+ tokens: TODO.
+ &quot;&quot;&quot;
+ return [self.token2id.get(i, self.unk_id) for i in tokens]
+</code></pre></details></div></div>
+<div class="api-detail" id="e349"><span class="dtag badge-m">method</span><h2><code>TokenIDConverter.get_num_vocabulary_size()</code></h2><div class="dmod">funasr.tokenizer.token_id_converter.TokenIDConverter · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/tokenizer/token_id_converter.py#L53" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Get num vocabulary size.</p>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def get_num_vocabulary_size(self) -&gt; int:
+ &quot;&quot;&quot;Get num vocabulary size.&quot;&quot;&quot;
+ return len(self.token_list)
+</code></pre></details></div>
+<div class="api-detail" id="e350"><span class="dtag badge-m">method</span><h2><code>TokenIDConverter.ids2tokens(integers)</code></h2><div class="dmod">funasr.tokenizer.token_id_converter.TokenIDConverter · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/tokenizer/token_id_converter.py#L57" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Ids2tokens.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>integers</code> — TODO.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def ids2tokens(self, integers: Union[np.ndarray, Iterable[int]]) -&gt; List[str]:
+ &quot;&quot;&quot;Ids2tokens.
+
+ Args:
+ integers: TODO.
+ &quot;&quot;&quot;
+ if isinstance(integers, np.ndarray) and integers.ndim != 1:
+ raise ValueError(f&quot;Must be 1 dim ndarray, but got {integers.ndim}&quot;)
+ return [self.token_list[i] for i in integers]
+</code></pre></details></div>
+<div class="api-detail" id="e351"><span class="dtag badge-m">method</span><h2><code>TokenIDConverter.tokens2ids(tokens)</code></h2><div class="dmod">funasr.tokenizer.token_id_converter.TokenIDConverter · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/tokenizer/token_id_converter.py#L67" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Tokens2ids.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>tokens</code> — TODO.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def tokens2ids(self, tokens: Iterable[str]) -&gt; List[int]:
+ &quot;&quot;&quot;Tokens2ids.
+
+ Args:
+ tokens: TODO.
+ &quot;&quot;&quot;
+ return [self.token2id.get(i, self.unk_id) for i in tokens]
+</code></pre></details></div>
+<div class="api-detail" id="e352"><span class="dtag badge-f">function</span><h2><code>WhisperTokenizer(**kwargs)</code></h2><div class="dmod">funasr.tokenizer.whisper_tokenizer · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/tokenizer/whisper_tokenizer.py#L5" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Whispertokenizer.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>**kwargs</code> — Additional keyword arguments.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code>def WhisperTokenizer(**kwargs):
+ &quot;&quot;&quot;Whispertokenizer.
+
+ Args:
+ **kwargs: Additional keyword arguments.
+ &quot;&quot;&quot;
+ try:
+ from whisper.tokenizer import get_tokenizer
+ except:
+ print(&quot;Notice: If you want to use whisper, please `pip install -U openai-whisper`&quot;)
+
+ language = kwargs.get(&quot;language&quot;, None)
+ task = kwargs.get(&quot;task&quot;, &quot;transcribe&quot;)
+ is_multilingual = kwargs.get(&quot;is_multilingual&quot;, True)
+ num_languages = kwargs.get(&quot;num_languages&quot;, 99)
+ tokenizer = get_tokenizer(
+ multilingual=is_multilingual,
+ num_languages=num_languages,
+ language=language,
+ task=task,
+ )
+
+ return tokenizer
+</code></pre></details></div>
+<div class="api-detail" id="e353"><span class="dtag badge-f">function</span><h2><code>SenseVoiceTokenizer(**kwargs)</code></h2><div class="dmod">funasr.tokenizer.whisper_tokenizer · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/tokenizer/whisper_tokenizer.py#L31" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Sensevoicetokenizer.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>**kwargs</code> — Additional keyword arguments.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code>def SenseVoiceTokenizer(**kwargs):
+ &quot;&quot;&quot;Sensevoicetokenizer.
+
+ Args:
+ **kwargs: Additional keyword arguments.
+ &quot;&quot;&quot;
+ from funasr.models.sense_voice.whisper_lib.tokenizer import get_tokenizer
+
+ language = kwargs.get(&quot;language&quot;, None)
+ task = kwargs.get(&quot;task&quot;, None)
+ is_multilingual = kwargs.get(&quot;is_multilingual&quot;, True)
+ num_languages = kwargs.get(&quot;num_languages&quot;, 8749)
+ vocab_path = kwargs.get(&quot;vocab_path&quot;, None)
+ tokenizer = get_tokenizer(
+ multilingual=is_multilingual,
+ num_languages=num_languages,
+ language=language,
+ task=task,
+ vocab_path=vocab_path,
+ )
+
+ return tokenizer
+</code></pre></details></div>
+<div class="api-detail" id="e354"><span class="dtag badge-c">class</span><h2>WordTokenizer</h2><div class="dmod">funasr.tokenizer.word_tokenizer · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/tokenizer/word_tokenizer.py#L11" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p class="muted">No documentation yet.</p></div><details class="src-block"><summary>📄 Source code</summary><pre><code>class WordTokenizer(AbsTokenizer):
+ def __init__(
+ self,
+ delimiter: str = None,
+ non_linguistic_symbols: Union[Path, str, Iterable[str]] = None,
+ remove_non_linguistic_symbols: bool = False,
+ ):
+ &quot;&quot;&quot;Initialize WordTokenizer.
+
+ Args:
+ delimiter: TODO.
+ non_linguistic_symbols: TODO.
+ remove_non_linguistic_symbols: TODO.
+ &quot;&quot;&quot;
+ self.delimiter = delimiter
+
+ if not remove_non_linguistic_symbols and non_linguistic_symbols is not None:
+ warnings.warn(
+ &quot;non_linguistic_symbols is only used &quot; &quot;when remove_non_linguistic_symbols = True&quot;
+ )
+
+ if non_linguistic_symbols is None:
+ self.non_linguistic_symbols = set()
+ elif isinstance(non_linguistic_symbols, (Path, str)):
+ non_linguistic_symbols = Path(non_linguistic_symbols)
+ try:
+ with non_linguistic_symbols.open(&quot;r&quot;, encoding=&quot;utf-8&quot;) as f:
+ self.non_linguistic_symbols = set(line.rstrip() for line in f)
+ except FileNotFoundError:
+ warnings.warn(f&quot;{non_linguistic_symbols} doesn&#x27;t exist.&quot;)
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/tokenizer/word_tokenizer.py#L11" target="_blank" class="src-more">View full source on GitHub →</a></details><h3 class="mt">Methods</h3><div class="mblk"><code>.text2tokens(line)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/tokenizer/word_tokenizer.py#L50" target="_blank" class="gh-link">L50</a><div class="mdoc"><p>Text2tokens.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>line</code> — TODO.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def text2tokens(self, line: str) -&gt; List[str]:
+ &quot;&quot;&quot;Text2tokens.
+
+ Args:
+ line: TODO.
+ &quot;&quot;&quot;
+ tokens = []
+ for t in line.split(self.delimiter):
+ if self.remove_non_linguistic_symbols and t in self.non_linguistic_symbols:
+ continue
+ tokens.append(t)
+ return tokens
+</code></pre></details></div><div class="mblk"><code>.tokens2text(tokens)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/tokenizer/word_tokenizer.py#L63" target="_blank" class="gh-link">L63</a><div class="mdoc"><p>Tokens2text.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>tokens</code> — TODO.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def tokens2text(self, tokens: Iterable[str]) -&gt; str:
+ &quot;&quot;&quot;Tokens2text.
+
+ Args:
+ tokens: TODO.
+ &quot;&quot;&quot;
+ if self.delimiter is None:
+ delimiter = &quot; &quot;
+ else:
+ delimiter = self.delimiter
+ return delimiter.join(tokens)
+</code></pre></details></div></div>
+<div class="api-detail" id="e355"><span class="dtag badge-m">method</span><h2><code>WordTokenizer.text2tokens(line)</code></h2><div class="dmod">funasr.tokenizer.word_tokenizer.WordTokenizer · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/tokenizer/word_tokenizer.py#L50" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Text2tokens.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>line</code> — TODO.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def text2tokens(self, line: str) -&gt; List[str]:
+ &quot;&quot;&quot;Text2tokens.
+
+ Args:
+ line: TODO.
+ &quot;&quot;&quot;
+ tokens = []
+ for t in line.split(self.delimiter):
+ if self.remove_non_linguistic_symbols and t in self.non_linguistic_symbols:
+ continue
+ tokens.append(t)
+ return tokens
+</code></pre></details></div>
+<div class="api-detail" id="e356"><span class="dtag badge-m">method</span><h2><code>WordTokenizer.tokens2text(tokens)</code></h2><div class="dmod">funasr.tokenizer.word_tokenizer.WordTokenizer · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/tokenizer/word_tokenizer.py#L63" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Tokens2text.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>tokens</code> — TODO.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def tokens2text(self, tokens: Iterable[str]) -&gt; str:
+ &quot;&quot;&quot;Tokens2text.
+
+ Args:
+ tokens: TODO.
+ &quot;&quot;&quot;
+ if self.delimiter is None:
+ delimiter = &quot; &quot;
+ else:
+ delimiter = self.delimiter
+ return delimiter.join(tokens)
+</code></pre></details></div>
+<div class="api-detail" id="e357"><span class="dtag badge-c">class</span><h2>thread_wrapper</h2><div class="dmod">funasr.utils.compute_det_ctc · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/utils/compute_det_ctc.py#L14" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p class="muted">No documentation yet.</p></div><details class="src-block"><summary>📄 Source code</summary><pre><code>class thread_wrapper(threading.Thread):
+ def __init__(self, func, args=()):
+ &quot;&quot;&quot;Initialize thread_wrapper.
+
+ Args:
+ func: TODO.
+ args: TODO.
+ &quot;&quot;&quot;
+ super(thread_wrapper, self).__init__()
+ self.func = func
+ self.args = args
+ self.result = []
+
+ def run(self):
+ &quot;&quot;&quot;Run.&quot;&quot;&quot;
+ self.result = self.func(*self.args)
+
+ def get_result(self):
+ &quot;&quot;&quot;Get result.&quot;&quot;&quot;
+ try:
+ return self.result
+ except Exception:
+ return None
+</code></pre></details><h3 class="mt">Methods</h3><div class="mblk"><code>.run()</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/utils/compute_det_ctc.py#L27" target="_blank" class="gh-link">L27</a><div class="mdoc"><p>Run.</p>
+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def run(self):
+ &quot;&quot;&quot;Run.&quot;&quot;&quot;
+ self.result = self.func(*self.args)
+</code></pre></details></div><div class="mblk"><code>.get_result()</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/utils/compute_det_ctc.py#L31" target="_blank" class="gh-link">L31</a><div class="mdoc"><p>Get result.</p>
+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def get_result(self):
+ &quot;&quot;&quot;Get result.&quot;&quot;&quot;
+ try:
+ return self.result
+ except Exception:
+ return None
+</code></pre></details></div></div>
+<div class="api-detail" id="e358"><span class="dtag badge-m">method</span><h2><code>thread_wrapper.run()</code></h2><div class="dmod">funasr.utils.compute_det_ctc.thread_wrapper · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/utils/compute_det_ctc.py#L27" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Run.</p>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def run(self):
+ &quot;&quot;&quot;Run.&quot;&quot;&quot;
+ self.result = self.func(*self.args)
+</code></pre></details></div>
+<div class="api-detail" id="e359"><span class="dtag badge-m">method</span><h2><code>thread_wrapper.get_result()</code></h2><div class="dmod">funasr.utils.compute_det_ctc.thread_wrapper · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/utils/compute_det_ctc.py#L31" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Get result.</p>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def get_result(self):
+ &quot;&quot;&quot;Get result.&quot;&quot;&quot;
+ try:
+ return self.result
+ except Exception:
+ return None
+</code></pre></details></div>
+<div class="api-detail" id="e360"><span class="dtag badge-f">function</span><h2><code>space_mixed_label(input_str)</code></h2><div class="dmod">funasr.utils.compute_det_ctc · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/utils/compute_det_ctc.py#L39" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Space mixed label.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>input_str</code> — TODO.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code>def space_mixed_label(input_str):
+ &quot;&quot;&quot;Space mixed label.
+
+ Args:
+ input_str: TODO.
+ &quot;&quot;&quot;
+ splits = split_mixed_label(input_str)
+ space_str = &#x27;&#x27;.join(f&#x27;{sub} &#x27; for sub in splits)
+ return space_str.strip()
+</code></pre></details></div>
+<div class="api-detail" id="e361"><span class="dtag badge-f">function</span><h2><code>read_lists(list_file)</code></h2><div class="dmod">funasr.utils.compute_det_ctc · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/utils/compute_det_ctc.py#L50" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Read lists.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>list_file</code> — TODO.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code>def read_lists(list_file):
+ &quot;&quot;&quot;Read lists.
+
+ Args:
+ list_file: TODO.
+ &quot;&quot;&quot;
+ lists = []
+ with open(list_file, &#x27;r&#x27;, encoding=&#x27;utf8&#x27;) as fin:
+ for line in fin:
+ if line.strip() != &#x27;&#x27;:
+ lists.append(line.strip())
+ return lists
+</code></pre></details></div>
+<div class="api-detail" id="e362"><span class="dtag badge-f">function</span><h2><code>make_pair(wav_lists, trans_lists)</code></h2><div class="dmod">funasr.utils.compute_det_ctc · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/utils/compute_det_ctc.py#L64" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Make pair.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>wav_lists</code> — TODO.</li>
+<li><code>trans_lists</code> — TODO.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code>def make_pair(wav_lists, trans_lists):
+ &quot;&quot;&quot;Make pair.
+
+ Args:
+ wav_lists: TODO.
+ trans_lists: TODO.
+ &quot;&quot;&quot;
+ logging.info(&#x27;make pair for wav-trans list&#x27;)
+
+ trans_table = {}
+ for line in trans_lists:
+ arr = line.strip().replace(&#x27;\t&#x27;, &#x27; &#x27;).split()
+ if len(arr) &lt; 2:
+ logging.debug(&#x27;invalid line in trans file: {}&#x27;.format(
+ line.strip()))
+ continue
+
+ trans_table[arr[0]] = line.replace(arr[0],&#x27;&#x27;).strip()
+
+ lists = []
+ for line in wav_lists:
+ arr = line.strip().replace(&#x27;\t&#x27;, &#x27; &#x27;).split()
+ if len(arr) == 2 and arr[0] in trans_table:
+ lists.append(
+ dict(key=arr[0],
+ txt=trans_table[arr[0]],
+ wav=arr[1],
+ sample_rate=16000))
+ else:
+ logging.debug(&quot;can&#x27;t find corresponding trans for key: {}&quot;.format(
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/utils/compute_det_ctc.py#L64" target="_blank" class="src-more">View full source on GitHub →</a></details></div>
+<div class="api-detail" id="e363"><span class="dtag badge-f">function</span><h2><code>count_duration(tid, data_lists)</code></h2><div class="dmod">funasr.utils.compute_det_ctc · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/utils/compute_det_ctc.py#L100" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Count duration.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>tid</code> — TODO.</li>
+<li><code>data_lists</code> — TODO.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code>def count_duration(tid, data_lists):
+ &quot;&quot;&quot;Count duration.
+
+ Args:
+ tid: TODO.
+ data_lists: TODO.
+ &quot;&quot;&quot;
+ results = []
+
+ for obj in data_lists:
+ assert &#x27;key&#x27; in obj
+ assert &#x27;wav&#x27; in obj
+ assert &#x27;txt&#x27; in obj
+ key = obj[&#x27;key&#x27;]
+ wav_file = obj[&#x27;wav&#x27;]
+ txt = obj[&#x27;txt&#x27;]
+
+ try:
+ rate, waveform = kaldiio.load_mat(wav_file)
+ waveform = torch.tensor(waveform, dtype=torch.float32)
+ waveform = waveform.unsqueeze(0)
+ frames = len(waveform[0])
+ duration = frames / float(rate)
+ except:
+ logging.info(f&#x27;load file failed: {wav_file}&#x27;)
+ duration = 0.0
+
+ obj[&#x27;duration&#x27;] = duration
+ results.append(obj)
+
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/utils/compute_det_ctc.py#L100" target="_blank" class="src-more">View full source on GitHub →</a></details></div>
+<div class="api-detail" id="e364"><span class="dtag badge-f">function</span><h2><code>load_data_and_score(keywords_list, data_file, trans_file, score_file)</code></h2><div class="dmod">funasr.utils.compute_det_ctc · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/utils/compute_det_ctc.py#L133" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Load data and score.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>keywords_list</code> — TODO.</li>
+<li><code>data_file</code> — TODO.</li>
+<li><code>trans_file</code> — TODO.</li>
+<li><code>score_file</code> — TODO.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code>def load_data_and_score(keywords_list, data_file, trans_file, score_file):
+ # score_table: {uttid: [keywordlist]}
+ &quot;&quot;&quot;Load data and score.
+
+ Args:
+ keywords_list: TODO.
+ data_file: TODO.
+ trans_file: TODO.
+ score_file: TODO.
+ &quot;&quot;&quot;
+ score_table = {}
+ with open(score_file, &#x27;r&#x27;, encoding=&#x27;utf8&#x27;) as fin:
+ # read score file and store in table
+ for line in fin:
+ arr = line.strip().split()
+ key = arr[0]
+ is_detected = arr[1]
+ if is_detected == &#x27;detected&#x27;:
+ if key not in score_table:
+ score_table.update(
+ {key: {
+ &#x27;kw&#x27;: space_mixed_label(arr[2]),
+ &#x27;confi&#x27;: float(arr[3])
+ }})
+ else:
+ if key not in score_table:
+ score_table.update({key: {&#x27;kw&#x27;: &#x27;unknown&#x27;, &#x27;confi&#x27;: -1.0}})
+
+ wav_lists = read_lists(data_file)
+ trans_lists = read_lists(trans_file)
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/utils/compute_det_ctc.py#L133" target="_blank" class="src-more">View full source on GitHub →</a></details></div>
+<div class="api-detail" id="e365"><span class="dtag badge-c">class</span><h2>DatadirWriter</h2><div class="dmod">funasr.utils.datadir_writer · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/utils/datadir_writer.py#L6" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Writer class to create kaldi like data directory.</p>
+<h4>Examples:</h4>
+<p>&gt;&gt;&gt; with DatadirWriter(&quot;output&quot;) as writer:</p>
+<p>... # output/sub.txt is created here</p>
+<p>... subwriter = writer[&quot;sub.txt&quot;]</p>
+<p>... # Write &quot;uttidA some/where/a.wav&quot;</p>
+<p>... subwriter[&quot;uttidA&quot;] = &quot;some/where/a.wav&quot;</p>
+<p>... subwriter[&quot;uttidB&quot;] = &quot;some/where/b.wav&quot;</p>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code>class DatadirWriter:
+ &quot;&quot;&quot;Writer class to create kaldi like data directory.
+
+ Examples:
+ &gt;&gt;&gt; with DatadirWriter(&quot;output&quot;) as writer:
+ ... # output/sub.txt is created here
+ ... subwriter = writer[&quot;sub.txt&quot;]
+ ... # Write &quot;uttidA some/where/a.wav&quot;
+ ... subwriter[&quot;uttidA&quot;] = &quot;some/where/a.wav&quot;
+ ... subwriter[&quot;uttidB&quot;] = &quot;some/where/b.wav&quot;
+
+ &quot;&quot;&quot;
+
+ def __init__(self, p: Union[Path, str]):
+ &quot;&quot;&quot;Initialize DatadirWriter.
+
+ Args:
+ p: TODO.
+ &quot;&quot;&quot;
+ self.path = Path(p)
+ self.chilidren = {}
+ self.fd = None
+ self.has_children = False
+ self.keys = set()
+
+ def __enter__(self):
+ &quot;&quot;&quot;Internal: enter .&quot;&quot;&quot;
+ return self
+
+ def __getitem__(self, key: str) -&gt; &quot;DatadirWriter&quot;:
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/utils/datadir_writer.py#L6" target="_blank" class="src-more">View full source on GitHub →</a></details><h3 class="mt">Methods</h3><div class="mblk"><code>.close()</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/utils/datadir_writer.py#L82" target="_blank" class="gh-link">L82</a><div class="mdoc"><p>Close.</p>
+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def close(self):
+ &quot;&quot;&quot;Close.&quot;&quot;&quot;
+ if self.has_children:
+ prev_child = None
+ for child in self.chilidren.values():
+ child.close()
+ if prev_child is not None and prev_child.keys != child.keys:
+ warnings.warn(
+ f&quot;Ids are mismatching between &quot; f&quot;{prev_child.path} and {child.path}&quot;
+ )
+ prev_child = child
+
+ elif self.fd is not None:
+ self.fd.close()
+</code></pre></details></div></div>
+<div class="api-detail" id="e366"><span class="dtag badge-m">method</span><h2><code>DatadirWriter.close()</code></h2><div class="dmod">funasr.utils.datadir_writer.DatadirWriter · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/utils/datadir_writer.py#L82" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Close.</p>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def close(self):
+ &quot;&quot;&quot;Close.&quot;&quot;&quot;
+ if self.has_children:
+ prev_child = None
+ for child in self.chilidren.values():
+ child.close()
+ if prev_child is not None and prev_child.keys != child.keys:
+ warnings.warn(
+ f&quot;Ids are mismatching between &quot; f&quot;{prev_child.path} and {child.path}&quot;
+ )
+ prev_child = child
+
+ elif self.fd is not None:
+ self.fd.close()
+</code></pre></details></div>
+<div class="api-detail" id="e367"><span class="dtag badge-f">function</span><h2><code>load_module_from_path(file_path)</code></h2><div class="dmod">funasr.utils.dynamic_import · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/utils/dynamic_import.py#L9" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>从给定的文件路径动态加载模块。</p>
+<p>:param file_path: 模块文件的绝对路径。</p>
+<p>:return: 加载的模块</p>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code>def load_module_from_path(file_path):
+ &quot;&quot;&quot;
+ 从给定的文件路径动态加载模块。
+
+ :param file_path: 模块文件的绝对路径。
+ :return: 加载的模块
+ &quot;&quot;&quot;
+ module_name = file_path.split(&quot;/&quot;)[-1].replace(&quot;.py&quot;, &quot;&quot;)
+ spec = importlib.util.spec_from_file_location(module_name, file_path)
+ module = importlib.util.module_from_spec(spec)
+ spec.loader.exec_module(module)
+ return module
+</code></pre></details></div>
+<div class="api-detail" id="e368"><span class="dtag badge-f">function</span><h2><code>import_module_from_path(file_path)</code></h2><div class="dmod">funasr.utils.dynamic_import · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/utils/dynamic_import.py#L23" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Import module from path.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>file_path</code> — TODO.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code>def import_module_from_path(file_path: str):
+
+ &quot;&quot;&quot;Import module from path.
+
+ Args:
+ file_path: TODO.
+ &quot;&quot;&quot;
+ if file_path.startswith(&quot;http&quot;):
+ from funasr.download.file import download_from_url
+
+ file_path = download_from_url(file_path)
+
+ file_dir = os.path.dirname(file_path)
+ # file_name = os.path.basename(file_path)
+ module_name = file_path.split(&quot;/&quot;)[-1].replace(&quot;.py&quot;, &quot;&quot;)
+ if len(file_dir) &lt; 1:
+ file_dir = &quot;./&quot;
+ sys.path.append(file_dir)
+ try:
+ importlib.import_module(module_name)
+ print(f&quot;Loading remote code successfully: {file_path}&quot;)
+ except Exception as e:
+ print(f&quot;Loading remote code failed: {file_path}, {e}&quot;)
+</code></pre></details></div>
+<div class="api-detail" id="e369"><span class="dtag badge-f">function</span><h2><code>export(model, data_in, quantize, opset_version, type, **kwargs)</code></h2><div class="dmod">funasr.utils.export_utils · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/utils/export_utils.py#L10" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Export.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>model</code> — Model instance or model name.</li>
+<li><code>data_in</code> — Input data (audio samples, file paths, or text).</li>
+<li><code>quantize</code> — TODO.</li>
+<li><code>opset_version</code> — TODO.</li>
+<li><code>type</code> — TODO.</li>
+<li><code>**kwargs</code> — Additional keyword arguments.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code>def export(
+ model, data_in=None, quantize: bool = False, opset_version: int = 14, type=&quot;onnx&quot;, **kwargs
+):
+ &quot;&quot;&quot;Export.
+
+ Args:
+ model: Model instance or model name.
+ data_in: Input data (audio samples, file paths, or text).
+ quantize: TODO.
+ opset_version: TODO.
+ type: TODO.
+ **kwargs: Additional keyword arguments.
+ &quot;&quot;&quot;
+ model_scripts = model.export(**kwargs)
+ export_dir = kwargs.get(&quot;output_dir&quot;, os.path.dirname(kwargs.get(&quot;init_param&quot;)))
+ os.makedirs(export_dir, exist_ok=True)
+
+ if not isinstance(model_scripts, (list, tuple)):
+ model_scripts = (model_scripts,)
+ for m in model_scripts:
+ m.eval()
+ if type == &quot;onnx&quot;:
+ _onnx(
+ m,
+ data_in=data_in,
+ quantize=quantize,
+ opset_version=opset_version,
+ export_dir=export_dir,
+ **kwargs,
+ )
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/utils/export_utils.py#L10" target="_blank" class="src-more">View full source on GitHub →</a></details></div>
+<div class="api-detail" id="e370"><span class="dtag badge-f">function</span><h2><code>install_requirements(requirements_path)</code></h2><div class="dmod">funasr.utils.install_model_requirements · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/utils/install_model_requirements.py#L5" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Install requirements.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>requirements_path</code> — TODO.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code>def install_requirements(requirements_path):
+ &quot;&quot;&quot;Install requirements.
+
+ Args:
+ requirements_path: TODO.
+ &quot;&quot;&quot;
+ try:
+ result = pip_install_r(requirements_path)
+ # check status
+ if result.returncode == 0:
+ print(&quot;install model requirements successfully&quot;)
+ return True
+ else:
+ print(&quot;fail to install model requirements! &quot;)
+ print(&quot;error&quot;, result.stderr)
+ return False
+ except Exception as e:
+ result = pip_install_r(requirements_path)
+ # check status
+ if result.returncode == 0:
+ print(&quot;install model requirements successfully&quot;)
+ return True
+ else:
+ print(&quot;fail to install model requirements! &quot;)
+ print(&quot;error&quot;, result.stderr)
+ return False
+</code></pre></details></div>
+<div class="api-detail" id="e371"><span class="dtag badge-f">function</span><h2><code>pip_install_r(requirements_path)</code></h2><div class="dmod">funasr.utils.install_model_requirements · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/utils/install_model_requirements.py#L33" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Pip install r.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>requirements_path</code> — TODO.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code>def pip_install_r(requirements_path):
+ &quot;&quot;&quot;Pip install r.
+
+ Args:
+ requirements_path: TODO.
+ &quot;&quot;&quot;
+ cmd = []
+ if shutil.which(&quot;pip&quot;) is not None:
+ cmd = [&quot;pip&quot;]
+ elif shutil.which(&quot;uv&quot;) is not None:
+ cmd = [&quot;uv&quot;, &quot;pip&quot;]
+ else:
+ raise RuntimeError(&quot;pip not found, failed to install model requirements&quot;)
+ cmd += [&quot;install&quot;, &quot;-r&quot;, requirements_path]
+ return subprocess.run(
+ cmd,
+ stdout=subprocess.PIPE,
+ stderr=subprocess.PIPE,
+ text=True,
+ )
+</code></pre></details></div>
+<div class="api-detail" id="e372"><span class="dtag badge-f">function</span><h2><code>split_mixed_label(input_str)</code></h2><div class="dmod">funasr.utils.kws_utils · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/utils/kws_utils.py#L13" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Split mixed label.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>input_str</code> — TODO.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code>def split_mixed_label(input_str):
+ &quot;&quot;&quot;Split mixed label.
+
+ Args:
+ input_str: TODO.
+ &quot;&quot;&quot;
+ tokens = []
+ s = input_str.lower()
+ while len(s) &gt; 0:
+ match = re.match(r&#x27;[A-Za-z!?,&lt;&gt;()\&#x27;]+&#x27;, s)
+ if match is not None:
+ word = match.group(0)
+ else:
+ word = s[0:1]
+ tokens.append(word)
+ s = s.replace(word, &#x27;&#x27;, 1).strip(&#x27; &#x27;)
+ return tokens
+</code></pre></details></div>
+<div class="api-detail" id="e373"><span class="dtag badge-f">function</span><h2><code>query_token_set(txt, symbol_table, lexicon_table)</code></h2><div class="dmod">funasr.utils.kws_utils · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/utils/kws_utils.py#L32" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Query token set.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>txt</code> — TODO.</li>
+<li><code>symbol_table</code> — TODO.</li>
+<li><code>lexicon_table</code> — TODO.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code>def query_token_set(txt, symbol_table, lexicon_table):
+ &quot;&quot;&quot;Query token set.
+
+ Args:
+ txt: TODO.
+ symbol_table: TODO.
+ lexicon_table: TODO.
+ &quot;&quot;&quot;
+ tokens_str = tuple()
+ tokens_idx = tuple()
+
+ if txt in symbol_table:
+ tokens_str = tokens_str + (txt, )
+ tokens_idx = tokens_idx + (symbol_table[txt], )
+ return tokens_str, tokens_idx
+
+ parts = split_mixed_label(txt)
+ for part in parts:
+ if part == &#x27;!sil&#x27; or part == &#x27;(sil)&#x27; or part == &#x27;&lt;sil&gt;&#x27;:
+ tokens_str = tokens_str + (&#x27;!sil&#x27;, )
+ elif part == &#x27;&lt;blank&gt;&#x27; or part == &#x27;&lt;blank&gt;&#x27;:
+ tokens_str = tokens_str + (&#x27;&lt;blank&gt;&#x27;, )
+ elif part == &#x27;(noise)&#x27; or part == &#x27;noise)&#x27; or part == &#x27;(noise&#x27; or part == &#x27;&lt;noise&gt;&#x27;:
+ tokens_str = tokens_str + (&#x27;&lt;unk&gt;&#x27;, )
+ elif part in symbol_table:
+ tokens_str = tokens_str + (part, )
+ elif part in lexicon_table:
+ for ch in lexicon_table[part]:
+ tokens_str = tokens_str + (ch, )
+ else:
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/utils/kws_utils.py#L32" target="_blank" class="src-more">View full source on GitHub →</a></details></div>
+<div class="api-detail" id="e374"><span class="dtag badge-c">class</span><h2>KwsCtcPrefixDecoder</h2><div class="dmod">funasr.utils.kws_utils · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/utils/kws_utils.py#L90" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Decoder interface wrapper for CTCPrefixDecode.</p>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code>class KwsCtcPrefixDecoder():
+ &quot;&quot;&quot;Decoder interface wrapper for CTCPrefixDecode.&quot;&quot;&quot;
+
+ def __init__(
+ self,
+ ctc: torch.nn.Module,
+ keywords: str,
+ token_list: list,
+ seg_dict: dict,
+ ):
+ &quot;&quot;&quot;Initialize class.
+
+ Args:
+ ctc (torch.nn.Module): The CTC implementation.
+ For example, :class:`espnet.nets.pytorch_backend.ctc.CTC`
+
+ &quot;&quot;&quot;
+ self.ctc = ctc
+ self.token_list = token_list
+
+ token_table = {}
+ for token in token_list:
+ token_table[token] = token_list.index(token)
+
+ self.keywords_idxset = {0}
+ self.keywords_token = {}
+ self.keywords_str = keywords
+ keywords_list = self.keywords_str.strip().replace(&#x27; &#x27;, &#x27;&#x27;).split(&#x27;,&#x27;)
+ for keyword in keywords_list:
+ strs, indexs = query_token_set(keyword, token_table, seg_dict)
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/utils/kws_utils.py#L90" target="_blank" class="src-more">View full source on GitHub →</a></details><h3 class="mt">Methods</h3><div class="mblk"><code>.beam_search(logits, logits_lengths, keywords_tokenset, score_beam_size, path_beam_size)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/utils/kws_utils.py#L125" target="_blank" class="gh-link">L125</a><div class="mdoc"><p>CTC prefix beam search inner implementation</p>
+<h4>Args:</h4>
+<ul>
+<li><code>logits (torch.Tensor)</code> — (1, max_len, vocab_size)</li>
+<li><code>logits_lengths (torch.Tensor)</code> — (1, )</li>
+<li><code>keywords_tokenset (set)</code> — token set for filtering score</li>
+<li><code>score_beam_size (int)</code> — beam size for score</li>
+<li><code>path_beam_size (int)</code> — beam size for path</li>
+</ul>
+<h4>Returns:</h4>
+<p>List[List[int]]: nbest results</p>
+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def beam_search(
+ self,
+ logits: torch.Tensor,
+ logits_lengths: torch.Tensor,
+ keywords_tokenset: set = None,
+ score_beam_size: int = 3,
+ path_beam_size: int = 20,
+ ) -&gt; Tuple[List[List[int]], torch.Tensor]:
+ &quot;&quot;&quot; CTC prefix beam search inner implementation
+
+ Args:
+ logits (torch.Tensor): (1, max_len, vocab_size)
+ logits_lengths (torch.Tensor): (1, )
+ keywords_tokenset (set): token set for filtering score
+ score_beam_size (int): beam size for score
+ path_beam_size (int): beam size for path
+
+ Returns:
+ List[List[int]]: nbest results
+ &quot;&quot;&quot;
+
+ maxlen = logits.size(0)
+ ctc_probs = logits
+ cur_hyps = [(tuple(), (1.0, 0.0, []))]
+
+ # CTC beam search step by step
+ for t in range(0, maxlen):
+ probs = ctc_probs[t] # (vocab_size,)
+ # key: prefix, value (pb, pnb), default value(-inf, -inf)
+ next_hyps = defaultdict(lambda: (0.0, 0.0, []))
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/utils/kws_utils.py#L125" target="_blank" class="src-more">View full source →</a></details></div><div class="mblk"><code>.is_sublist(main_list, check_list)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/utils/kws_utils.py#L232" target="_blank" class="gh-link">L232</a><div class="mdoc"><p>Is sublist.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>main_list</code> — TODO.</li>
+<li><code>check_list</code> — TODO.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def is_sublist(self, main_list, check_list):
+ &quot;&quot;&quot;Is sublist.
+
+ Args:
+ main_list: TODO.
+ check_list: TODO.
+ &quot;&quot;&quot;
+ if len(main_list) &lt; len(check_list):
+ return -1
+
+ if len(main_list) == len(check_list):
+ return 0 if main_list == check_list else -1
+
+ for i in range(len(main_list) - len(check_list) + 1):
+ if main_list[i] == check_list[0]:
+ for j in range(len(check_list)):
+ if main_list[i + j] != check_list[j]:
+ break
+ else:
+ return i
+ else:
+ return -1
+</code></pre></details></div><div class="mblk"><code>.decode(x)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/utils/kws_utils.py#L295" target="_blank" class="gh-link">L295</a><div class="mdoc"><p>Get an initial state for decoding.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>x (torch.Tensor)</code> — The encoded feature tensor</li>
+<li><code>Returns</code> — decode result</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def decode(self, x: torch.Tensor):
+ &quot;&quot;&quot;Get an initial state for decoding.
+
+ Args:
+ x (torch.Tensor): The encoded feature tensor
+
+ Returns: decode result
+
+ &quot;&quot;&quot;
+
+ raw_logp = self.ctc.softmax(x.unsqueeze(0)).detach().squeeze(0).cpu()
+ xlen = torch.tensor([raw_logp.size(1)])
+
+ return self._decode_inside(raw_logp, xlen)
+</code></pre></details></div></div>
+<div class="api-detail" id="e375"><span class="dtag badge-m">method</span><h2><code>KwsCtcPrefixDecoder.beam_search(logits, logits_lengths, keywords_tokenset, score_beam_size, path_beam_size)</code></h2><div class="dmod">funasr.utils.kws_utils.KwsCtcPrefixDecoder · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/utils/kws_utils.py#L125" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>CTC prefix beam search inner implementation</p>
+<h4>Args:</h4>
+<ul>
+<li><code>logits (torch.Tensor)</code> — (1, max_len, vocab_size)</li>
+<li><code>logits_lengths (torch.Tensor)</code> — (1, )</li>
+<li><code>keywords_tokenset (set)</code> — token set for filtering score</li>
+<li><code>score_beam_size (int)</code> — beam size for score</li>
+<li><code>path_beam_size (int)</code> — beam size for path</li>
+</ul>
+<h4>Returns:</h4>
+<p>List[List[int]]: nbest results</p>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def beam_search(
+ self,
+ logits: torch.Tensor,
+ logits_lengths: torch.Tensor,
+ keywords_tokenset: set = None,
+ score_beam_size: int = 3,
+ path_beam_size: int = 20,
+ ) -&gt; Tuple[List[List[int]], torch.Tensor]:
+ &quot;&quot;&quot; CTC prefix beam search inner implementation
+
+ Args:
+ logits (torch.Tensor): (1, max_len, vocab_size)
+ logits_lengths (torch.Tensor): (1, )
+ keywords_tokenset (set): token set for filtering score
+ score_beam_size (int): beam size for score
+ path_beam_size (int): beam size for path
+
+ Returns:
+ List[List[int]]: nbest results
+ &quot;&quot;&quot;
+
+ maxlen = logits.size(0)
+ ctc_probs = logits
+ cur_hyps = [(tuple(), (1.0, 0.0, []))]
+
+ # CTC beam search step by step
+ for t in range(0, maxlen):
+ probs = ctc_probs[t] # (vocab_size,)
+ # key: prefix, value (pb, pnb), default value(-inf, -inf)
+ next_hyps = defaultdict(lambda: (0.0, 0.0, []))
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/utils/kws_utils.py#L125" target="_blank" class="src-more">View full source on GitHub →</a></details></div>
+<div class="api-detail" id="e376"><span class="dtag badge-m">method</span><h2><code>KwsCtcPrefixDecoder.is_sublist(main_list, check_list)</code></h2><div class="dmod">funasr.utils.kws_utils.KwsCtcPrefixDecoder · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/utils/kws_utils.py#L232" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Is sublist.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>main_list</code> — TODO.</li>
+<li><code>check_list</code> — TODO.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def is_sublist(self, main_list, check_list):
+ &quot;&quot;&quot;Is sublist.
+
+ Args:
+ main_list: TODO.
+ check_list: TODO.
+ &quot;&quot;&quot;
+ if len(main_list) &lt; len(check_list):
+ return -1
+
+ if len(main_list) == len(check_list):
+ return 0 if main_list == check_list else -1
+
+ for i in range(len(main_list) - len(check_list) + 1):
+ if main_list[i] == check_list[0]:
+ for j in range(len(check_list)):
+ if main_list[i + j] != check_list[j]:
+ break
+ else:
+ return i
+ else:
+ return -1
+</code></pre></details></div>
+<div class="api-detail" id="e377"><span class="dtag badge-m">method</span><h2><code>KwsCtcPrefixDecoder.decode(x)</code></h2><div class="dmod">funasr.utils.kws_utils.KwsCtcPrefixDecoder · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/utils/kws_utils.py#L295" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Get an initial state for decoding.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>x (torch.Tensor)</code> — The encoded feature tensor</li>
+<li><code>Returns</code> — decode result</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def decode(self, x: torch.Tensor):
+ &quot;&quot;&quot;Get an initial state for decoding.
+
+ Args:
+ x (torch.Tensor): The encoded feature tensor
+
+ Returns: decode result
+
+ &quot;&quot;&quot;
+
+ raw_logp = self.ctc.softmax(x.unsqueeze(0)).detach().squeeze(0).cpu()
+ xlen = torch.tensor([raw_logp.size(1)])
+
+ return self._decode_inside(raw_logp, xlen)
+</code></pre></details></div>
+<div class="api-detail" id="e378"><span class="dtag badge-f">function</span><h2><code>is_ffmpeg_installed()</code></h2><div class="dmod">funasr.utils.load_utils · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/utils/load_utils.py#L27" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Is ffmpeg installed.</p>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code>def is_ffmpeg_installed():
+ &quot;&quot;&quot;Is ffmpeg installed.&quot;&quot;&quot;
+ try:
+ output = subprocess.check_output([&quot;ffmpeg&quot;, &quot;-version&quot;], stderr=subprocess.STDOUT)
+ return &quot;ffmpeg version&quot; in output.decode(&quot;utf-8&quot;)
+ except (subprocess.CalledProcessError, FileNotFoundError):
+ return False
+</code></pre></details></div>
+<div class="api-detail" id="e379"><span class="dtag badge-f">function</span><h2><code>load_audio_text_image_video(data_or_path_or_list, fs, audio_fs, data_type, tokenizer, **kwargs)</code></h2><div class="dmod">funasr.utils.load_utils · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/utils/load_utils.py#L48" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Load audio/text/image/video data from various input formats.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>data_or_path_or_list</code> — File path, URL, numpy array, torch Tensor, bytes, or list.</li>
+<li><code>fs (int)</code> — Target sample rate (default 16000).</li>
+<li><code>audio_fs (int)</code> — Source audio sample rate.</li>
+<li><code>data_type (str)</code> — Input type (&quot;sound&quot;, &quot;text&quot;, &quot;fbank&quot;).</li>
+</ul>
+<h4>Returns:</h4>
+<p>torch.Tensor or list: Loaded and resampled audio tensor(s).</p>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code>def load_audio_text_image_video(
+ data_or_path_or_list,
+ fs: int = 16000,
+ audio_fs: int = 16000,
+ data_type=&quot;sound&quot;,
+ tokenizer=None,
+ **kwargs,
+):
+ &quot;&quot;&quot;Load audio/text/image/video data from various input formats.
+
+ Args:
+ data_or_path_or_list: File path, URL, numpy array, torch Tensor, bytes, or list.
+ fs (int): Target sample rate (default 16000).
+ audio_fs (int): Source audio sample rate.
+ data_type (str): Input type (&quot;sound&quot;, &quot;text&quot;, &quot;fbank&quot;).
+
+ Returns:
+ torch.Tensor or list: Loaded and resampled audio tensor(s).
+ &quot;&quot;&quot;
+ if isinstance(data_or_path_or_list, (list, tuple)):
+ if data_type is not None and isinstance(data_type, (list, tuple)):
+ data_types = [data_type] * len(data_or_path_or_list)
+ data_or_path_or_list_ret = [[] for d in data_type]
+ for i, (data_type_i, data_or_path_or_list_i) in enumerate(
+ zip(data_types, data_or_path_or_list)
+ ):
+ for j, (data_type_j, data_or_path_or_list_j) in enumerate(
+ zip(data_type_i, data_or_path_or_list_i)
+ ):
+ data_or_path_or_list_j = load_audio_text_image_video(
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/utils/load_utils.py#L48" target="_blank" class="src-more">View full source on GitHub →</a></details></div>
+<div class="api-detail" id="e380"><span class="dtag badge-f">function</span><h2><code>load_bytes(input)</code></h2><div class="dmod">funasr.utils.load_utils · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/utils/load_utils.py#L187" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Convert audio bytes to numpy array.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>input (bytes)</code> — Raw audio bytes.</li>
+</ul>
+<h4>Returns:</h4>
+<ul>
+<li><code>numpy.ndarray</code> — Decoded audio samples.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code>def load_bytes(input):
+ &quot;&quot;&quot;Convert audio bytes to numpy array.
+
+ Args:
+ input (bytes): Raw audio bytes.
+
+ Returns:
+ numpy.ndarray: Decoded audio samples.
+ &quot;&quot;&quot;
+ # Only run the (expensive) frame-rate validation when the payload is an
+ # actual audio container (WAV, MP3, OGG, …). Raw PCM buffers have no
+ # recognisable header and would cause pydub to spend ~200 ms before
+ # raising an exception that is then silently swallowed anyway.
+ if _is_audio_container(input):
+ try:
+ input = validate_frame_rate(input)
+ except Exception:
+ pass
+ middle_data = np.frombuffer(input, dtype=np.int16)
+ middle_data = np.asarray(middle_data)
+ if middle_data.dtype.kind not in &quot;iu&quot;:
+ raise TypeError(&quot;&#x27;middle_data&#x27; must be an array of integers&quot;)
+ dtype = np.dtype(&quot;float32&quot;)
+ if dtype.kind != &quot;f&quot;:
+ raise TypeError(&quot;&#x27;dtype&#x27; must be a floating point type&quot;)
+
+ i = np.iinfo(middle_data.dtype)
+ abs_max = 2 ** (i.bits - 1)
+ offset = i.min + abs_max
+ array = np.frombuffer((middle_data.astype(dtype) - offset) / abs_max, dtype=np.float32)
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/utils/load_utils.py#L187" target="_blank" class="src-more">View full source on GitHub →</a></details></div>
+<div class="api-detail" id="e381"><span class="dtag badge-f">function</span><h2><code>validate_frame_rate(input, fs)</code></h2><div class="dmod">funasr.utils.load_utils · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/utils/load_utils.py#L220" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Validate frame rate.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>input</code> — Input audio/text data.</li>
+<li><code>fs</code> — TODO.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code>def validate_frame_rate(
+ input,
+ fs: int = 16000,
+):
+
+ # 将文件读取为字节流
+ &quot;&quot;&quot;Validate frame rate.
+
+ Args:
+ input: Input audio/text data.
+ fs: TODO.
+ &quot;&quot;&quot;
+ byte_data = BytesIO(input)
+
+ # 使用 pydub 加载音频
+ try:
+ audio = AudioSegment.from_file(byte_data)
+ except:
+ raise RuntimeError(
+ &quot;You are decoding the pcm data, please install pydub first. via `pip install pydub`.&quot;
+ )
+
+ # 确保采样率为 16000 Hz
+ if audio.frame_rate != fs:
+ audio = audio.set_frame_rate(fs)
+
+ # 将重新采样后的音频导出为字节流
+ output = BytesIO()
+ audio.export(output, format=&quot;wav&quot;)
+ output.seek(0)
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/utils/load_utils.py#L220" target="_blank" class="src-more">View full source on GitHub →</a></details></div>
+<div class="api-detail" id="e382"><span class="dtag badge-f">function</span><h2><code>extract_fbank(data, data_len, data_type, frontend, **kwargs)</code></h2><div class="dmod">funasr.utils.load_utils · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/utils/load_utils.py#L257" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Extract filter-bank features from audio data.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>data</code> — Audio samples (list of numpy arrays or tensors).</li>
+<li><code>data_len</code> — Lengths of each sample.</li>
+<li><code>data_type (str)</code> — Input type (&quot;sound&quot;, &quot;fbank&quot;).</li>
+<li><code>frontend</code> — Frontend instance for feature extraction.</li>
+</ul>
+<h4>Returns:</h4>
+<ul>
+<li><code>tuple</code> — (features_tensor, feature_lengths, feature_times)</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code>def extract_fbank(data, data_len=None, data_type: str = &quot;sound&quot;, frontend=None, **kwargs):
+ &quot;&quot;&quot;Extract filter-bank features from audio data.
+
+ Args:
+ data: Audio samples (list of numpy arrays or tensors).
+ data_len: Lengths of each sample.
+ data_type (str): Input type (&quot;sound&quot;, &quot;fbank&quot;).
+ frontend: Frontend instance for feature extraction.
+
+ Returns:
+ tuple: (features_tensor, feature_lengths, feature_times)
+ &quot;&quot;&quot;
+ if isinstance(data, np.ndarray):
+ data = torch.from_numpy(data)
+ if len(data.shape) &lt; 2:
+ data = data[None, :] # data: [batch, N]
+ elif data.shape[0] &gt; 1:
+ data = data.mean(dim=0, keepdim=True) # convert stereo/multi-channel to mono
+ data_len = [data.shape[1]] if data_len is None else data_len
+ elif isinstance(data, torch.Tensor):
+ if len(data.shape) &lt; 2:
+ data = data[None, :] # data: [batch, N]
+ elif data.shape[0] &gt; 1:
+ data = data.mean(dim=0, keepdim=True) # convert stereo/multi-channel to mono
+ data_len = [data.shape[1]] if data_len is None else data_len
+ elif isinstance(data, (list, tuple)):
+ data_list, data_len = [], []
+ for data_i in data:
+ if isinstance(data_i, np.ndarray):
+ data_i = torch.from_numpy(data_i)
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/utils/load_utils.py#L257" target="_blank" class="src-more">View full source on GitHub →</a></details></div>
+<div class="api-detail" id="e383"><span class="dtag badge-f">function</span><h2><code>statistic_model_parameters(model, prefix)</code></h2><div class="dmod">funasr.utils.misc · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/utils/misc.py#L10" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Statistic model parameters.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>model</code> — Model instance or model name.</li>
+<li><code>prefix</code> — TODO.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code>def statistic_model_parameters(model, prefix=None):
+ &quot;&quot;&quot;Statistic model parameters.
+
+ Args:
+ model: Model instance or model name.
+ prefix: TODO.
+ &quot;&quot;&quot;
+ var_dict = model.state_dict()
+ numel = 0
+ for i, key in enumerate(
+ sorted(list([x for x in var_dict.keys() if &quot;num_batches_tracked&quot; not in x]))
+ ):
+ if prefix is None or key.startswith(prefix):
+ numel += var_dict[key].numel()
+ return numel
+</code></pre></details></div>
+<div class="api-detail" id="e384"><span class="dtag badge-f">function</span><h2><code>int2vec(x, vec_dim, dtype)</code></h2><div class="dmod">funasr.utils.misc · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/utils/misc.py#L27" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Int2vec.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>x</code> — TODO.</li>
+<li><code>vec_dim</code> — Size/dimension parameter.</li>
+<li><code>dtype</code> — TODO.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code>def int2vec(x, vec_dim=8, dtype=np.int32):
+ &quot;&quot;&quot;Int2vec.
+
+ Args:
+ x: TODO.
+ vec_dim: Size/dimension parameter.
+ dtype: TODO.
+ &quot;&quot;&quot;
+ b = (&quot;{:0&quot; + str(vec_dim) + &quot;b}&quot;).format(x)
+ # little-endian order: lower bit first
+ return (np.array(list(b)[::-1]) == &quot;1&quot;).astype(dtype)
+</code></pre></details></div>
+<div class="api-detail" id="e385"><span class="dtag badge-f">function</span><h2><code>seq2arr(seq, vec_dim)</code></h2><div class="dmod">funasr.utils.misc · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/utils/misc.py#L40" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Seq2arr.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>seq</code> — TODO.</li>
+<li><code>vec_dim</code> — Size/dimension parameter.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code>def seq2arr(seq, vec_dim=8):
+ &quot;&quot;&quot;Seq2arr.
+
+ Args:
+ seq: TODO.
+ vec_dim: Size/dimension parameter.
+ &quot;&quot;&quot;
+ return np.row_stack([int2vec(int(x), vec_dim) for x in seq])
+</code></pre></details></div>
+<div class="api-detail" id="e386"><span class="dtag badge-f">function</span><h2><code>load_scp_as_dict(scp_path, value_type, kv_sep)</code></h2><div class="dmod">funasr.utils.misc · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/utils/misc.py#L50" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Load scp as dict.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>scp_path</code> — TODO.</li>
+<li><code>value_type</code> — TODO.</li>
+<li><code>kv_sep</code> — TODO.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code>def load_scp_as_dict(scp_path, value_type=&quot;str&quot;, kv_sep=&quot; &quot;):
+ &quot;&quot;&quot;Load scp as dict.
+
+ Args:
+ scp_path: TODO.
+ value_type: TODO.
+ kv_sep: TODO.
+ &quot;&quot;&quot;
+ with io.open(scp_path, &quot;r&quot;, encoding=&quot;utf-8&quot;) as f:
+ ret_dict = OrderedDict()
+ for one_line in f.readlines():
+ one_line = one_line.strip()
+ pos = one_line.find(kv_sep)
+ key, value = one_line[:pos], one_line[pos + 1 :]
+ if value_type == &quot;list&quot;:
+ value = value.split(&quot; &quot;)
+ ret_dict[key] = value
+ return ret_dict
+</code></pre></details></div>
+<div class="api-detail" id="e387"><span class="dtag badge-f">function</span><h2><code>load_scp_as_list(scp_path, value_type, kv_sep)</code></h2><div class="dmod">funasr.utils.misc · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/utils/misc.py#L70" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Load scp as list.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>scp_path</code> — TODO.</li>
+<li><code>value_type</code> — TODO.</li>
+<li><code>kv_sep</code> — TODO.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code>def load_scp_as_list(scp_path, value_type=&quot;str&quot;, kv_sep=&quot; &quot;):
+ &quot;&quot;&quot;Load scp as list.
+
+ Args:
+ scp_path: TODO.
+ value_type: TODO.
+ kv_sep: TODO.
+ &quot;&quot;&quot;
+ with io.open(scp_path, &quot;r&quot;, encoding=&quot;utf8&quot;) as f:
+ ret_dict = []
+ for one_line in f.readlines():
+ one_line = one_line.strip()
+ pos = one_line.find(kv_sep)
+ key, value = one_line[:pos], one_line[pos + 1 :]
+ if value_type == &quot;list&quot;:
+ value = value.split(&quot; &quot;)
+ ret_dict.append((key, value))
+ return ret_dict
+</code></pre></details></div>
+<div class="api-detail" id="e388"><span class="dtag badge-f">function</span><h2><code>deep_update(original, update)</code></h2><div class="dmod">funasr.utils.misc · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/utils/misc.py#L90" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Recursively merge update dict into original dict (in-place).</p>
+<p>For nested dicts, merges recursively. For other types, overwrites.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>original (dict)</code> — Target dict to be updated in-place.</li>
+<li><code>update (dict)</code> — Source dict with new values.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code>def deep_update(original, update):
+
+ &quot;&quot;&quot;Recursively merge update dict into original dict (in-place).
+
+ For nested dicts, merges recursively. For other types, overwrites.
+
+ Args:
+ original (dict): Target dict to be updated in-place.
+ update (dict): Source dict with new values.
+ &quot;&quot;&quot;
+ for key, value in update.items():
+ if isinstance(value, dict) and key in original:
+ if len(value) == 0:
+ original[key] = value
+ deep_update(original[key], value)
+ else:
+ original[key] = value
+</code></pre></details></div>
+<div class="api-detail" id="e389"><span class="dtag badge-f">function</span><h2><code>prepare_model_dir(**kwargs)</code></h2><div class="dmod">funasr.utils.misc · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/utils/misc.py#L109" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Prepare model dir.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>**kwargs</code> — Additional keyword arguments.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code>def prepare_model_dir(**kwargs):
+
+ &quot;&quot;&quot;Prepare model dir.
+
+ Args:
+ **kwargs: Additional keyword arguments.
+ &quot;&quot;&quot;
+ os.makedirs(kwargs.get(&quot;output_dir&quot;, &quot;./&quot;), exist_ok=True)
+
+ yaml_file = os.path.join(kwargs.get(&quot;output_dir&quot;, &quot;./&quot;), &quot;config.yaml&quot;)
+ OmegaConf.save(config=kwargs, f=yaml_file)
+ logging.info(f&quot;kwargs: {kwargs}&quot;)
+ logging.info(&quot;config.yaml is saved to: %s&quot;, yaml_file)
+
+ model_path = kwargs.get(&quot;model_path&quot;, None)
+ if model_path is not None:
+ config_json = os.path.join(model_path, &quot;configuration.json&quot;)
+ if os.path.exists(config_json):
+ shutil.copy(
+ config_json, os.path.join(kwargs.get(&quot;output_dir&quot;, &quot;./&quot;), &quot;configuration.json&quot;)
+ )
+</code></pre></details></div>
+<div class="api-detail" id="e390"><span class="dtag badge-f">function</span><h2><code>extract_filename_without_extension(file_path)</code></h2><div class="dmod">funasr.utils.misc · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/utils/misc.py#L132" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>从给定的文件路径中提取文件名(不包含路径和扩展名)</p>
+<p>:param file_path: 完整的文件路径</p>
+<p>:return: 文件名(不含路径和扩展名)</p>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code>def extract_filename_without_extension(file_path):
+ &quot;&quot;&quot;
+ 从给定的文件路径中提取文件名(不包含路径和扩展名)
+ :param file_path: 完整的文件路径
+ :return: 文件名(不含路径和扩展名)
+ &quot;&quot;&quot;
+ # 首先,使用os.path.basename获取路径中的文件名部分(含扩展名)
+ filename_with_extension = os.path.basename(file_path)
+ # 然后,使用os.path.splitext分离文件名和扩展名
+ filename, extension = os.path.splitext(filename_with_extension)
+ # 返回不包含扩展名的文件名
+ return filename
+</code></pre></details></div>
+<div class="api-detail" id="e391"><span class="dtag badge-f">function</span><h2><code>smart_remove(path)</code></h2><div class="dmod">funasr.utils.misc · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/utils/misc.py#L146" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Intelligently removes files, empty directories, and non-empty directories recursively.</p>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code>def smart_remove(path):
+ &quot;&quot;&quot;Intelligently removes files, empty directories, and non-empty directories recursively.&quot;&quot;&quot;
+ # Check if the provided path exists
+ if not os.path.exists(path):
+ print(f&quot;{path} does not exist.&quot;)
+ return
+
+ # If the path is a file, delete it
+ if os.path.isfile(path):
+ os.remove(path)
+ print(f&quot;File {path} has been deleted.&quot;)
+ # If the path is a directory
+ elif os.path.isdir(path):
+ try:
+ # Attempt to remove an empty directory
+ os.rmdir(path)
+ print(f&quot;Empty directory {path} has been deleted.&quot;)
+ except OSError:
+ # If the directory is not empty, remove it along with all its contents
+ shutil.rmtree(path)
+ print(f&quot;Non-empty directory {path} has been recursively deleted.&quot;)
+</code></pre></details></div>
+<div class="api-detail" id="e392"><span class="dtag badge-f">function</span><h2><code>isChinese(ch)</code></h2><div class="dmod">funasr.utils.postprocess_utils · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/utils/postprocess_utils.py#L8" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Ischinese.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>ch</code> — TODO.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code>def isChinese(ch: str):
+ &quot;&quot;&quot;Ischinese.
+
+ Args:
+ ch: TODO.
+ &quot;&quot;&quot;
+ if &quot;\u4e00&quot; &lt;= ch &lt;= &quot;\u9fff&quot; or &quot;\u0030&quot; &lt;= ch &lt;= &quot;\u0039&quot; or ch == &quot;@&quot;:
+ return True
+ return False
+</code></pre></details></div>
+<div class="api-detail" id="e393"><span class="dtag badge-f">function</span><h2><code>isAllChinese(word)</code></h2><div class="dmod">funasr.utils.postprocess_utils · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/utils/postprocess_utils.py#L19" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Isallchinese.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>word</code> — TODO.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code>def isAllChinese(word: Union[List[Any], str]):
+ &quot;&quot;&quot;Isallchinese.
+
+ Args:
+ word: TODO.
+ &quot;&quot;&quot;
+ word_lists = []
+ for i in word:
+ cur = i.replace(&quot; &quot;, &quot;&quot;)
+ cur = cur.replace(&quot;&lt;/s&gt;&quot;, &quot;&quot;)
+ cur = cur.replace(&quot;&lt;s&gt;&quot;, &quot;&quot;)
+ cur = cur.replace(&quot;&lt;unk&gt;&quot;, &quot;&quot;)
+ cur = cur.replace(&quot;&lt;OOV&gt;&quot;, &quot;&quot;)
+ word_lists.append(cur)
+
+ if len(word_lists) == 0:
+ return False
+
+ for ch in word_lists:
+ if isChinese(ch) is False:
+ return False
+ return True
+</code></pre></details></div>
+<div class="api-detail" id="e394"><span class="dtag badge-f">function</span><h2><code>isAllAlpha(word)</code></h2><div class="dmod">funasr.utils.postprocess_utils · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/utils/postprocess_utils.py#L43" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Isallalpha.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>word</code> — TODO.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code>def isAllAlpha(word: Union[List[Any], str]):
+ &quot;&quot;&quot;Isallalpha.
+
+ Args:
+ word: TODO.
+ &quot;&quot;&quot;
+ word_lists = []
+ for i in word:
+ cur = i.replace(&quot; &quot;, &quot;&quot;)
+ cur = cur.replace(&quot;&lt;/s&gt;&quot;, &quot;&quot;)
+ cur = cur.replace(&quot;&lt;s&gt;&quot;, &quot;&quot;)
+ cur = cur.replace(&quot;&lt;unk&gt;&quot;, &quot;&quot;)
+ cur = cur.replace(&quot;&lt;OOV&gt;&quot;, &quot;&quot;)
+ word_lists.append(cur)
+
+ if len(word_lists) == 0:
+ return False
+
+ for ch in word_lists:
+ if ch.isalpha() is False and ch != &quot;&#x27;&quot;:
+ return False
+ elif ch.isalpha() is True and isChinese(ch) is True:
+ return False
+
+ return True
+</code></pre></details></div>
+<div class="api-detail" id="e395"><span class="dtag badge-f">function</span><h2><code>abbr_dispose(words, time_stamp)</code></h2><div class="dmod">funasr.utils.postprocess_utils · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/utils/postprocess_utils.py#L71" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Abbr dispose.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>words</code> — TODO.</li>
+<li><code>time_stamp</code> — TODO.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code>def abbr_dispose(words: List[Any], time_stamp: List[List] = None) -&gt; List[Any]:
+ &quot;&quot;&quot;Abbr dispose.
+
+ Args:
+ words: TODO.
+ time_stamp: TODO.
+ &quot;&quot;&quot;
+ words_size = len(words)
+ word_lists = []
+ abbr_begin = []
+ abbr_end = []
+ last_num = -1
+ ts_lists = []
+ ts_nums = []
+ ts_index = 0
+ for num in range(words_size):
+ if num &lt;= last_num:
+ continue
+
+ if len(words[num]) == 1 and words[num].encode(&quot;utf-8&quot;).isalpha():
+ if (
+ num + 1 &lt; words_size
+ and words[num + 1] == &quot; &quot;
+ and num + 2 &lt; words_size
+ and len(words[num + 2]) == 1
+ and words[num + 2].encode(&quot;utf-8&quot;).isalpha()
+ ):
+ # found the begin of abbr
+ abbr_begin.append(num)
+ num += 2
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/utils/postprocess_utils.py#L71" target="_blank" class="src-more">View full source on GitHub →</a></details></div>
+<div class="api-detail" id="e396"><span class="dtag badge-f">function</span><h2><code>sentence_postprocess(words, time_stamp)</code></h2><div class="dmod">funasr.utils.postprocess_utils · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/utils/postprocess_utils.py#L165" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Sentence postprocess.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>words</code> — TODO.</li>
+<li><code>time_stamp</code> — TODO.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code>def sentence_postprocess(words: List[Any], time_stamp: List[List] = None):
+ &quot;&quot;&quot;Sentence postprocess.
+
+ Args:
+ words: TODO.
+ time_stamp: TODO.
+ &quot;&quot;&quot;
+ middle_lists = []
+ word_lists = []
+ word_item = &quot;&quot;
+ ts_lists = []
+
+ # wash words lists
+ for i in words:
+ word = &quot;&quot;
+ if isinstance(i, str):
+ word = i
+ else:
+ word = i.decode(&quot;utf-8&quot;)
+
+ if word in [&quot;&lt;s&gt;&quot;, &quot;&lt;/s&gt;&quot;, &quot;&lt;unk&gt;&quot;, &quot;&lt;OOV&gt;&quot;]:
+ continue
+ else:
+ middle_lists.append(word)
+
+ # all chinese characters
+ if isAllChinese(middle_lists):
+ for i, ch in enumerate(middle_lists):
+ word_lists.append(ch.replace(&quot; &quot;, &quot;&quot;))
+ if time_stamp is not None:
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/utils/postprocess_utils.py#L165" target="_blank" class="src-more">View full source on GitHub →</a></details></div>
+<div class="api-detail" id="e397"><span class="dtag badge-f">function</span><h2><code>sentence_postprocess_sentencepiece(words)</code></h2><div class="dmod">funasr.utils.postprocess_utils · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/utils/postprocess_utils.py#L281" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Sentence postprocess sentencepiece.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>words</code> — TODO.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code>def sentence_postprocess_sentencepiece(words):
+ &quot;&quot;&quot;Sentence postprocess sentencepiece.
+
+ Args:
+ words: TODO.
+ &quot;&quot;&quot;
+ middle_lists = []
+ word_lists = []
+ word_item = &quot;&quot;
+
+ # wash words lists
+ for i in words:
+ word = &quot;&quot;
+ if isinstance(i, str):
+ word = i
+ else:
+ word = i.decode(&quot;utf-8&quot;)
+
+ if word in [&quot;&lt;s&gt;&quot;, &quot;&lt;/s&gt;&quot;, &quot;&lt;unk&gt;&quot;, &quot;&lt;OOV&gt;&quot;]:
+ continue
+ else:
+ middle_lists.append(word)
+
+ # all alpha characters
+ for i, ch in enumerate(middle_lists):
+ word = &quot;&quot;
+ if &quot;\u2581&quot; in ch and i == 0:
+ word_item = &quot;&quot;
+ word = ch.replace(&quot;\u2581&quot;, &quot;&quot;)
+ word_item += word
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/utils/postprocess_utils.py#L281" target="_blank" class="src-more">View full source on GitHub →</a></details></div>
+<div class="api-detail" id="e398"><span class="dtag badge-f">function</span><h2><code>format_str_v2(s)</code></h2><div class="dmod">funasr.utils.postprocess_utils · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/utils/postprocess_utils.py#L411" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Format str v2.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>s</code> — TODO.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code>def format_str_v2(s):
+ &quot;&quot;&quot;Format str v2.
+
+ Args:
+ s: TODO.
+ &quot;&quot;&quot;
+ sptk_dict = {}
+ for sptk in emoji_dict:
+ sptk_dict[sptk] = s.count(sptk)
+ s = s.replace(sptk, &quot;&quot;)
+ emo = &quot;&lt;|NEUTRAL|&gt;&quot;
+ for e in emo_dict:
+ if sptk_dict[e] &gt; sptk_dict[emo]:
+ emo = e
+ for e in event_dict:
+ if sptk_dict[e] &gt; 0:
+ s = event_dict[e] + s
+ s = s + emo_dict[emo]
+
+ for emoji in emo_set.union(event_set):
+ s = s.replace(&quot; &quot; + emoji, emoji)
+ s = s.replace(emoji + &quot; &quot;, emoji)
+ return s.strip()
+</code></pre></details></div>
+<div class="api-detail" id="e399"><span class="dtag badge-f">function</span><h2><code>rich_transcription_postprocess(s)</code></h2><div class="dmod">funasr.utils.postprocess_utils · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/utils/postprocess_utils.py#L436" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Rich transcription postprocess.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>s</code> — TODO.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code>def rich_transcription_postprocess(s):
+ &quot;&quot;&quot;Rich transcription postprocess.
+
+ Args:
+ s: TODO.
+ &quot;&quot;&quot;
+ def get_emo(s):
+ &quot;&quot;&quot;Get emo.
+
+ Args:
+ s: TODO.
+ &quot;&quot;&quot;
+ return s[-1] if s[-1] in emo_set else None
+
+ def get_event(s):
+ &quot;&quot;&quot;Get event.
+
+ Args:
+ s: TODO.
+ &quot;&quot;&quot;
+ return s[0] if s[0] in event_set else None
+
+ s = s.replace(&quot;&lt;|nospeech|&gt;&lt;|Event_UNK|&gt;&quot;, &quot;❓&quot;)
+ for lang in lang_dict:
+ s = s.replace(lang, &quot;&lt;|lang|&gt;&quot;)
+ s_list = [format_str_v2(s_i).strip(&quot; &quot;) for s_i in s.split(&quot;&lt;|lang|&gt;&quot;)]
+ new_s = &quot; &quot; + s_list[0]
+ cur_ent_event = get_event(new_s)
+ for i in range(1, len(s_list)):
+ if len(s_list[i]) == 0:
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/utils/postprocess_utils.py#L436" target="_blank" class="src-more">View full source on GitHub →</a></details></div>
+<div class="api-detail" id="e400"><span class="dtag badge-f">function</span><h2><code>check_audio_list(audio)</code></h2><div class="dmod">funasr.utils.speaker_utils · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/utils/speaker_utils.py#L18" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Check audio list.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>audio</code> — TODO.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code>def check_audio_list(audio: list):
+ &quot;&quot;&quot;Check audio list.
+
+ Args:
+ audio: TODO.
+ &quot;&quot;&quot;
+ audio_dur = 0
+ for i in range(len(audio)):
+ seg = audio[i]
+ assert seg[1] &gt;= seg[0], &quot;modelscope error: Wrong time stamps.&quot;
+ assert isinstance(seg[2], np.ndarray), &quot;modelscope error: Wrong data type.&quot;
+ assert (
+ int(seg[1] * 16000) - int(seg[0] * 16000) == seg[2].shape[0]
+ ), &quot;modelscope error: audio data in list is inconsistent with time length.&quot;
+ if i &gt; 0:
+ assert seg[0] &gt;= audio[i - 1][1], &quot;modelscope error: Wrong time stamps.&quot;
+ audio_dur += seg[1] - seg[0]
+ return audio_dur
+</code></pre></details></div>
+<div class="api-detail" id="e401"><span class="dtag badge-f">function</span><h2><code>sv_preprocess(inputs)</code></h2><div class="dmod">funasr.utils.speaker_utils · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/utils/speaker_utils.py#L39" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Sv preprocess.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>inputs</code> — TODO.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code>def sv_preprocess(inputs: Union[np.ndarray, list]):
+ &quot;&quot;&quot;Sv preprocess.
+
+ Args:
+ inputs: TODO.
+ &quot;&quot;&quot;
+ output = []
+ for i in range(len(inputs)):
+ if isinstance(inputs[i], str):
+ file_bytes = File.read(inputs[i])
+ data, fs = sf.load(io.BytesIO(file_bytes), dtype=&quot;float32&quot;)
+ if len(data.shape) == 2:
+ data = data[:, 0]
+ data = torch.from_numpy(data).unsqueeze(0)
+ data = data.squeeze(0)
+ elif isinstance(inputs[i], np.ndarray):
+ assert len(inputs[i].shape) == 1, &quot;modelscope error: Input array should be [N, T]&quot;
+ data = inputs[i]
+ if data.dtype in [&quot;int16&quot;, &quot;int32&quot;, &quot;int64&quot;]:
+ data = (data / (1 &lt;&lt; 15)).astype(&quot;float32&quot;)
+ else:
+ data = data.astype(&quot;float32&quot;)
+ data = torch.from_numpy(data)
+ else:
+ raise ValueError(
+ &quot;modelscope error: The input type is restricted to audio address and nump array.&quot;
+ )
+ output.append(data)
+ return output
+</code></pre></details></div>
+<div class="api-detail" id="e402"><span class="dtag badge-f">function</span><h2><code>sv_chunk(vad_segments, fs)</code></h2><div class="dmod">funasr.utils.speaker_utils · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/utils/speaker_utils.py#L70" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Sv chunk.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>vad_segments</code> — TODO.</li>
+<li><code>fs</code> — TODO.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code>def sv_chunk(vad_segments: list, fs=16000) -&gt; list:
+ &quot;&quot;&quot;Sv chunk.
+
+ Args:
+ vad_segments: TODO.
+ fs: TODO.
+ &quot;&quot;&quot;
+ config = {
+ &quot;seg_dur&quot;: 1.5,
+ &quot;seg_shift&quot;: 0.75,
+ }
+
+ def seg_chunk(seg_data):
+ &quot;&quot;&quot;Seg chunk.
+
+ Args:
+ seg_data: TODO.
+ &quot;&quot;&quot;
+ seg_st = seg_data[0]
+ data = seg_data[2]
+ chunk_len = int(config[&quot;seg_dur&quot;] * fs)
+ chunk_shift = int(config[&quot;seg_shift&quot;] * fs)
+ last_chunk_ed = 0
+ seg_res = []
+ for chunk_st in range(0, data.shape[0], chunk_shift):
+ chunk_ed = min(chunk_st + chunk_len, data.shape[0])
+ if chunk_ed &lt;= last_chunk_ed:
+ break
+ last_chunk_ed = chunk_ed
+ chunk_st = max(0, chunk_ed - chunk_len)
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/utils/speaker_utils.py#L70" target="_blank" class="src-more">View full source on GitHub →</a></details></div>
+<div class="api-detail" id="e403"><span class="dtag badge-f">function</span><h2><code>extract_feature(audio)</code></h2><div class="dmod">funasr.utils.speaker_utils · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/utils/speaker_utils.py#L113" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Extract feature.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>audio</code> — TODO.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code>def extract_feature(audio):
+ &quot;&quot;&quot;Extract feature.
+
+ Args:
+ audio: TODO.
+ &quot;&quot;&quot;
+ features = []
+ for au in audio:
+ feature = Kaldi.fbank(au.unsqueeze(0), num_mel_bins=80)
+ feature = feature - feature.mean(dim=0, keepdim=True)
+ features.append(feature.unsqueeze(0))
+ features = torch.cat(features)
+ return features
+</code></pre></details></div>
+<div class="api-detail" id="e404"><span class="dtag badge-f">function</span><h2><code>postprocess(segments, vad_segments, labels, embeddings)</code></h2><div class="dmod">funasr.utils.speaker_utils · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/utils/speaker_utils.py#L128" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Postprocess.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>segments</code> — TODO.</li>
+<li><code>vad_segments</code> — TODO.</li>
+<li><code>labels</code> — TODO.</li>
+<li><code>embeddings</code> — TODO.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code>def postprocess(
+ segments: list, vad_segments: list, labels: np.ndarray, embeddings: np.ndarray
+) -&gt; list:
+ &quot;&quot;&quot;Postprocess.
+
+ Args:
+ segments: TODO.
+ vad_segments: TODO.
+ labels: TODO.
+ embeddings: TODO.
+ &quot;&quot;&quot;
+ assert len(segments) == len(labels)
+ labels = correct_labels(labels)
+ distribute_res = []
+ for i in range(len(segments)):
+ distribute_res.append([segments[i][0], segments[i][1], labels[i]])
+ # merge the same speakers chronologically
+ distribute_res = merge_seque(distribute_res)
+
+ # accquire speaker center
+ spk_embs = []
+ for i in range(labels.max() + 1):
+ spk_emb = embeddings[labels == i].mean(0)
+ spk_embs.append(spk_emb)
+ spk_embs = np.stack(spk_embs)
+
+ def is_overlapped(t1, t2):
+ &quot;&quot;&quot;Is overlapped.
+
+ Args:
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/utils/speaker_utils.py#L128" target="_blank" class="src-more">View full source on GitHub →</a></details></div>
+<div class="api-detail" id="e405"><span class="dtag badge-f">function</span><h2><code>correct_labels(labels)</code></h2><div class="dmod">funasr.utils.speaker_utils · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/utils/speaker_utils.py#L178" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Correct labels.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>labels</code> — TODO.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code>def correct_labels(labels):
+ &quot;&quot;&quot;Correct labels.
+
+ Args:
+ labels: TODO.
+ &quot;&quot;&quot;
+ labels_id = 0
+ id2id = {}
+ new_labels = []
+ for i in labels:
+ if i not in id2id:
+ id2id[i] = labels_id
+ labels_id += 1
+ new_labels.append(id2id[i])
+ return np.array(new_labels)
+</code></pre></details></div>
+<div class="api-detail" id="e406"><span class="dtag badge-f">function</span><h2><code>merge_seque(distribute_res)</code></h2><div class="dmod">funasr.utils.speaker_utils · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/utils/speaker_utils.py#L195" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Merge seque.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>distribute_res</code> — TODO.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code>def merge_seque(distribute_res):
+ &quot;&quot;&quot;Merge seque.
+
+ Args:
+ distribute_res: TODO.
+ &quot;&quot;&quot;
+ res = [distribute_res[0]]
+ for i in range(1, len(distribute_res)):
+ if distribute_res[i][2] != res[-1][2] or distribute_res[i][0] &gt; res[-1][1]:
+ res.append(distribute_res[i])
+ else:
+ res[-1][1] = distribute_res[i][1]
+ return res
+</code></pre></details></div>
+<div class="api-detail" id="e407"><span class="dtag badge-f">function</span><h2><code>smooth(res, mindur)</code></h2><div class="dmod">funasr.utils.speaker_utils · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/utils/speaker_utils.py#L210" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Smooth.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>res</code> — TODO.</li>
+<li><code>mindur</code> — TODO.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code>def smooth(res, mindur=1):
+ # short segments are assigned to nearest speakers.
+ &quot;&quot;&quot;Smooth.
+
+ Args:
+ res: TODO.
+ mindur: TODO.
+ &quot;&quot;&quot;
+ for i in range(len(res)):
+ res[i][0] = round(res[i][0], 2)
+ res[i][1] = round(res[i][1], 2)
+ if res[i][1] - res[i][0] &lt; mindur:
+ if i == 0:
+ res[i][2] = res[i + 1][2]
+ elif i == len(res) - 1:
+ res[i][2] = res[i - 1][2]
+ elif res[i][0] - res[i - 1][1] &lt;= res[i + 1][0] - res[i][1]:
+ res[i][2] = res[i - 1][2]
+ else:
+ res[i][2] = res[i + 1][2]
+ # merge the speakers
+ res = merge_seque(res)
+
+ return res
+</code></pre></details></div>
+<div class="api-detail" id="e408"><span class="dtag badge-f">function</span><h2><code>distribute_spk(sentence_list, sd_time_list)</code></h2><div class="dmod">funasr.utils.speaker_utils · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/utils/speaker_utils.py#L236" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Distribute spk.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>sentence_list</code> — TODO.</li>
+<li><code>sd_time_list</code> — TODO.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code>def distribute_spk(sentence_list, sd_time_list):
+ &quot;&quot;&quot;Distribute spk.
+
+ Args:
+ sentence_list: TODO.
+ sd_time_list: TODO.
+ &quot;&quot;&quot;
+ sd_sentence_list = []
+ for d in sentence_list:
+ sentence_start = d[&quot;ts_list&quot;][0][0]
+ sentence_end = d[&quot;ts_list&quot;][-1][1]
+ sentence_spk = 0
+ max_overlap = 0
+ for sd_time in sd_time_list:
+ spk_st, spk_ed, spk = sd_time
+ spk_st = spk_st * 1000
+ spk_ed = spk_ed * 1000
+ overlap = max(min(sentence_end, spk_ed) - max(sentence_start, spk_st), 0)
+ if overlap &gt; max_overlap:
+ max_overlap = overlap
+ sentence_spk = spk
+ d[&quot;spk&quot;] = sentence_spk
+ sd_sentence_list.append(d)
+ return sd_sentence_list
+</code></pre></details></div>
+<div class="api-detail" id="e409"><span class="dtag badge-f">function</span><h2><code>cif_wo_hidden(alphas, threshold)</code></h2><div class="dmod">funasr.utils.timestamp_tools · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/utils/timestamp_tools.py#L11" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Cif wo hidden.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>alphas</code> — TODO.</li>
+<li><code>threshold</code> — TODO.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code>def cif_wo_hidden(alphas, threshold):
+ &quot;&quot;&quot;Cif wo hidden.
+
+ Args:
+ alphas: TODO.
+ threshold: TODO.
+ &quot;&quot;&quot;
+ batch_size, len_time = alphas.size()
+ # loop varss
+ integrate = torch.zeros([batch_size], device=alphas.device)
+ # intermediate vars along time
+ list_fires = []
+ for t in range(len_time):
+ alpha = alphas[:, t]
+ integrate += alpha
+ list_fires.append(integrate)
+ fire_place = integrate &gt;= threshold
+ integrate = torch.where(
+ fire_place,
+ integrate - torch.ones([batch_size], device=alphas.device) * threshold,
+ integrate,
+ )
+ fires = torch.stack(list_fires, 1)
+ return fires
+</code></pre></details></div>
+<div class="api-detail" id="e410"><span class="dtag badge-f">function</span><h2><code>ts_prediction_lfr6_standard(us_alphas, us_peaks, char_list, vad_offset, force_time_shift, sil_in_str, upsample_rate)</code></h2><div class="dmod">funasr.utils.timestamp_tools · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/utils/timestamp_tools.py#L37" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Ts prediction lfr6 standard.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>us_alphas</code> — TODO.</li>
+<li><code>us_peaks</code> — TODO.</li>
+<li><code>char_list</code> — TODO.</li>
+<li><code>vad_offset</code> — TODO.</li>
+<li><code>force_time_shift</code> — TODO.</li>
+<li><code>sil_in_str</code> — TODO.</li>
+<li><code>upsample_rate</code> — TODO.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code>def ts_prediction_lfr6_standard(
+ us_alphas, us_peaks, char_list, vad_offset=0.0, force_time_shift=-1.5, sil_in_str=True, upsample_rate=3,
+):
+ &quot;&quot;&quot;Ts prediction lfr6 standard.
+
+ Args:
+ us_alphas: TODO.
+ us_peaks: TODO.
+ char_list: TODO.
+ vad_offset: TODO.
+ force_time_shift: TODO.
+ sil_in_str: TODO.
+ upsample_rate: TODO.
+ &quot;&quot;&quot;
+ if not len(char_list):
+ return &quot;&quot;, []
+ START_END_THRESHOLD = 5
+ MAX_TOKEN_DURATION = 12 # 3 times upsampled
+ TIME_RATE=10.0 * 6 / 1000 / upsample_rate
+ if len(us_alphas.shape) == 2:
+ alphas, peaks = us_alphas[0], us_peaks[0] # support inference batch_size=1 only
+ else:
+ alphas, peaks = us_alphas, us_peaks
+ if char_list[-1] == &quot;&lt;/s&gt;&quot;:
+ char_list = char_list[:-1]
+ fire_place = (
+ torch.where(peaks &gt;= 1.0 - 1e-4)[0].cpu().numpy() + force_time_shift
+ ) # total offset
+ if len(fire_place) != len(char_list) + 1:
+ alphas /= alphas.sum() / (len(char_list) + 1)
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/utils/timestamp_tools.py#L37" target="_blank" class="src-more">View full source on GitHub →</a></details></div>
+<div class="api-detail" id="e411"><span class="dtag badge-f">function</span><h2><code>timestamp_sentence(punc_id_list, timestamp_postprocessed, text_postprocessed, return_raw_text)</code></h2><div class="dmod">funasr.utils.timestamp_tools · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/utils/timestamp_tools.py#L125" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Split recognized text into sentences using punctuation, with timestamps.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>punc_id_list (Tensor/list)</code> — Punctuation IDs from CT-Transformer.</li>
+<li><code>Values</code> — 1=none, 2=comma, 3=period, 4=question.</li>
+<li><code>timestamp_postprocessed (list)</code> — Per-character timestamps [[start_ms, end_ms], ...].</li>
+<li><code>text_postprocessed (str)</code> — Space-separated recognized text.</li>
+<li><code>return_raw_text (bool)</code> — Include raw_text in output.</li>
+</ul>
+<h4>Returns:</h4>
+<p>list[dict]: Sentences with keys: text, start, end, timestamp, [raw_text].</p>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code>def timestamp_sentence(
+ punc_id_list, timestamp_postprocessed, text_postprocessed, return_raw_text=False
+):
+
+ &quot;&quot;&quot;Split recognized text into sentences using punctuation, with timestamps.
+
+ Args:
+ punc_id_list (Tensor/list): Punctuation IDs from CT-Transformer.
+ Values: 1=none, 2=comma, 3=period, 4=question.
+ timestamp_postprocessed (list): Per-character timestamps [[start_ms, end_ms], ...].
+ text_postprocessed (str): Space-separated recognized text.
+ return_raw_text (bool): Include raw_text in output.
+
+ Returns:
+ list[dict]: Sentences with keys: text, start, end, timestamp, [raw_text].
+ &quot;&quot;&quot;
+ punc_list = [&quot;,&quot;, &quot;。&quot;, &quot;?&quot;, &quot;、&quot;]
+ res = []
+ if text_postprocessed is None:
+ return res
+ if timestamp_postprocessed is None:
+ return res
+ if len(timestamp_postprocessed) == 0:
+ return res
+ if len(text_postprocessed) == 0:
+ return res
+
+ if punc_id_list is None or len(punc_id_list) == 0:
+ res.append(
+ {
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/utils/timestamp_tools.py#L125" target="_blank" class="src-more">View full source on GitHub →</a></details></div>
+<div class="api-detail" id="e412"><span class="dtag badge-f">function</span><h2><code>timestamp_sentence_en(punc_id_list, timestamp_postprocessed, text_postprocessed, return_raw_text)</code></h2><div class="dmod">funasr.utils.timestamp_tools · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/utils/timestamp_tools.py#L223" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Timestamp sentence en.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>punc_id_list</code> — TODO.</li>
+<li><code>timestamp_postprocessed</code> — TODO.</li>
+<li><code>text_postprocessed</code> — TODO.</li>
+<li><code>return_raw_text</code> — TODO.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code>def timestamp_sentence_en(
+ punc_id_list, timestamp_postprocessed, text_postprocessed, return_raw_text=False
+):
+ &quot;&quot;&quot;Timestamp sentence en.
+
+ Args:
+ punc_id_list: TODO.
+ timestamp_postprocessed: TODO.
+ text_postprocessed: TODO.
+ return_raw_text: TODO.
+ &quot;&quot;&quot;
+ punc_list = [&quot;,&quot;, &quot;.&quot;, &quot;?&quot;, &quot;,&quot;]
+ res = []
+ if text_postprocessed is None:
+ return res
+ if timestamp_postprocessed is None:
+ return res
+ if len(timestamp_postprocessed) == 0:
+ return res
+ if len(text_postprocessed) == 0:
+ return res
+
+ if punc_id_list is None or len(punc_id_list) == 0:
+ res.append(
+ {
+ &quot;text&quot;: text_postprocessed.split(),
+ &quot;start&quot;: timestamp_postprocessed[0][0],
+ &quot;end&quot;: timestamp_postprocessed[-1][1],
+ &quot;timestamp&quot;: timestamp_postprocessed,
+ }
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/utils/timestamp_tools.py#L223" target="_blank" class="src-more">View full source on GitHub →</a></details></div>
+<div class="api-detail" id="e413"><span class="dtag badge-c">class</span><h2>MakePadMask</h2><div class="dmod">funasr.utils.torch_function · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/utils/torch_function.py#L9" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p class="muted">No documentation yet.</p></div><details class="src-block"><summary>📄 Source code</summary><pre><code>class MakePadMask(nn.Module):
+ def __init__(self, max_seq_len=512, flip=True):
+ &quot;&quot;&quot;Initialize MakePadMask.
+
+ Args:
+ max_seq_len: TODO.
+ flip: TODO.
+ &quot;&quot;&quot;
+ super().__init__()
+ if flip:
+ self.mask_pad = torch.Tensor(1 - np.tri(max_seq_len)).type(torch.bool)
+ else:
+ self.mask_pad = torch.Tensor(np.tri(max_seq_len)).type(torch.bool)
+
+ def forward(self, lengths, xs=None, length_dim=-1, maxlen=None):
+ &quot;&quot;&quot;Make mask tensor containing indices of padded part.
+ This implementation creates the same mask tensor with original make_pad_mask,
+ which can be converted into onnx format.
+ Dimension length of xs should be 2 or 3.
+ &quot;&quot;&quot;
+ if length_dim == 0:
+ raise ValueError(&quot;length_dim cannot be 0: {}&quot;.format(length_dim))
+
+ if xs is not None and len(xs.shape) == 3:
+ if length_dim == 1:
+ lengths = lengths.unsqueeze(1).expand(*xs.transpose(1, 2).shape[:2])
+ else:
+ lengths = lengths.unsqueeze(1).expand(*xs.shape[:2])
+
+ if maxlen is not None:
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/utils/torch_function.py#L9" target="_blank" class="src-more">View full source on GitHub →</a></details><h3 class="mt">Methods</h3><div class="mblk"><code>.forward(lengths, xs, length_dim, maxlen)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/utils/torch_function.py#L23" target="_blank" class="gh-link">L23</a><div class="mdoc"><p>Make mask tensor containing indices of padded part.</p>
+<p>This implementation creates the same mask tensor with original make_pad_mask,</p>
+<p>which can be converted into onnx format.</p>
+<p>Dimension length of xs should be 2 or 3.</p>
+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def forward(self, lengths, xs=None, length_dim=-1, maxlen=None):
+ &quot;&quot;&quot;Make mask tensor containing indices of padded part.
+ This implementation creates the same mask tensor with original make_pad_mask,
+ which can be converted into onnx format.
+ Dimension length of xs should be 2 or 3.
+ &quot;&quot;&quot;
+ if length_dim == 0:
+ raise ValueError(&quot;length_dim cannot be 0: {}&quot;.format(length_dim))
+
+ if xs is not None and len(xs.shape) == 3:
+ if length_dim == 1:
+ lengths = lengths.unsqueeze(1).expand(*xs.transpose(1, 2).shape[:2])
+ else:
+ lengths = lengths.unsqueeze(1).expand(*xs.shape[:2])
+
+ if maxlen is not None:
+ m = maxlen
+ elif xs is not None:
+ m = xs.shape[-1]
+ else:
+ m = torch.max(lengths)
+
+ mask = self.mask_pad[lengths - 1][..., :m].type(torch.float32)
+
+ if length_dim == 1:
+ return mask.transpose(1, 2)
+ else:
+ return mask
+</code></pre></details></div></div>
+<div class="api-detail" id="e414"><span class="dtag badge-m">method</span><h2><code>MakePadMask.forward(lengths, xs, length_dim, maxlen)</code></h2><div class="dmod">funasr.utils.torch_function.MakePadMask · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/utils/torch_function.py#L23" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Make mask tensor containing indices of padded part.</p>
+<p>This implementation creates the same mask tensor with original make_pad_mask,</p>
+<p>which can be converted into onnx format.</p>
+<p>Dimension length of xs should be 2 or 3.</p>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def forward(self, lengths, xs=None, length_dim=-1, maxlen=None):
+ &quot;&quot;&quot;Make mask tensor containing indices of padded part.
+ This implementation creates the same mask tensor with original make_pad_mask,
+ which can be converted into onnx format.
+ Dimension length of xs should be 2 or 3.
+ &quot;&quot;&quot;
+ if length_dim == 0:
+ raise ValueError(&quot;length_dim cannot be 0: {}&quot;.format(length_dim))
+
+ if xs is not None and len(xs.shape) == 3:
+ if length_dim == 1:
+ lengths = lengths.unsqueeze(1).expand(*xs.transpose(1, 2).shape[:2])
+ else:
+ lengths = lengths.unsqueeze(1).expand(*xs.shape[:2])
+
+ if maxlen is not None:
+ m = maxlen
+ elif xs is not None:
+ m = xs.shape[-1]
+ else:
+ m = torch.max(lengths)
+
+ mask = self.mask_pad[lengths - 1][..., :m].type(torch.float32)
+
+ if length_dim == 1:
+ return mask.transpose(1, 2)
+ else:
+ return mask
+</code></pre></details></div>
+<div class="api-detail" id="e415"><span class="dtag badge-c">class</span><h2>sequence_mask</h2><div class="dmod">funasr.utils.torch_function · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/utils/torch_function.py#L53" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p class="muted">No documentation yet.</p></div><details class="src-block"><summary>📄 Source code</summary><pre><code>class sequence_mask(nn.Module):
+ def __init__(self, max_seq_len=512, flip=True):
+ &quot;&quot;&quot;Initialize sequence_mask.
+
+ Args:
+ max_seq_len: TODO.
+ flip: TODO.
+ &quot;&quot;&quot;
+ super().__init__()
+
+ def forward(self, lengths, max_seq_len=None, dtype=torch.float32, device=None):
+ &quot;&quot;&quot;Forward pass for training.
+
+ Args:
+ lengths: TODO.
+ max_seq_len: TODO.
+ dtype: TODO.
+ device: Target device (&quot;cuda:0&quot;, &quot;cpu&quot;, etc.).
+ &quot;&quot;&quot;
+ if max_seq_len is None:
+ max_seq_len = lengths.max()
+ row_vector = torch.arange(0, max_seq_len, 1).to(lengths.device)
+ matrix = torch.unsqueeze(lengths, dim=-1)
+ mask = row_vector &lt; matrix
+
+ return mask.type(dtype).to(device) if device is not None else mask.type(dtype)
+</code></pre></details><h3 class="mt">Methods</h3><div class="mblk"><code>.forward(lengths, max_seq_len, dtype, device)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/utils/torch_function.py#L63" target="_blank" class="gh-link">L63</a><div class="mdoc"><p>Forward pass for training.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>lengths</code> — TODO.</li>
+<li><code>max_seq_len</code> — TODO.</li>
+<li><code>dtype</code> — TODO.</li>
+<li><code>device</code> — Target device (&quot;cuda:0&quot;, &quot;cpu&quot;, etc.).</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def forward(self, lengths, max_seq_len=None, dtype=torch.float32, device=None):
+ &quot;&quot;&quot;Forward pass for training.
+
+ Args:
+ lengths: TODO.
+ max_seq_len: TODO.
+ dtype: TODO.
+ device: Target device (&quot;cuda:0&quot;, &quot;cpu&quot;, etc.).
+ &quot;&quot;&quot;
+ if max_seq_len is None:
+ max_seq_len = lengths.max()
+ row_vector = torch.arange(0, max_seq_len, 1).to(lengths.device)
+ matrix = torch.unsqueeze(lengths, dim=-1)
+ mask = row_vector &lt; matrix
+
+ return mask.type(dtype).to(device) if device is not None else mask.type(dtype)
+</code></pre></details></div></div>
+<div class="api-detail" id="e416"><span class="dtag badge-m">method</span><h2><code>sequence_mask.forward(lengths, max_seq_len, dtype, device)</code></h2><div class="dmod">funasr.utils.torch_function.sequence_mask · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/utils/torch_function.py#L63" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Forward pass for training.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>lengths</code> — TODO.</li>
+<li><code>max_seq_len</code> — TODO.</li>
+<li><code>dtype</code> — TODO.</li>
+<li><code>device</code> — Target device (&quot;cuda:0&quot;, &quot;cpu&quot;, etc.).</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def forward(self, lengths, max_seq_len=None, dtype=torch.float32, device=None):
+ &quot;&quot;&quot;Forward pass for training.
+
+ Args:
+ lengths: TODO.
+ max_seq_len: TODO.
+ dtype: TODO.
+ device: Target device (&quot;cuda:0&quot;, &quot;cpu&quot;, etc.).
+ &quot;&quot;&quot;
+ if max_seq_len is None:
+ max_seq_len = lengths.max()
+ row_vector = torch.arange(0, max_seq_len, 1).to(lengths.device)
+ matrix = torch.unsqueeze(lengths, dim=-1)
+ mask = row_vector &lt; matrix
+
+ return mask.type(dtype).to(device) if device is not None else mask.type(dtype)
+</code></pre></details></div>
+<div class="api-detail" id="e417"><span class="dtag badge-f">function</span><h2><code>normalize(input, p, dim, out)</code></h2><div class="dmod">funasr.utils.torch_function · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/utils/torch_function.py#L81" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Normalize.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>input</code> — Input audio/text data.</li>
+<li><code>p</code> — TODO.</li>
+<li><code>dim</code> — TODO.</li>
+<li><code>out</code> — TODO.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code>def normalize(
+ input: torch.Tensor, p: float = 2.0, dim: int = 1, out: Optional[torch.Tensor] = None
+) -&gt; torch.Tensor:
+ &quot;&quot;&quot;Normalize.
+
+ Args:
+ input: Input audio/text data.
+ p: TODO.
+ dim: TODO.
+ out: TODO.
+ &quot;&quot;&quot;
+ if out is None:
+ denom = input.norm(p, dim, keepdim=True).expand_as(input)
+ return input / denom
+ else:
+ denom = input.norm(p, dim, keepdim=True).expand_as(input)
+ return torch.div(input, denom, out=out)
+</code></pre></details></div>
+<div class="api-detail" id="e418"><span class="dtag badge-f">function</span><h2><code>subsequent_mask(size)</code></h2><div class="dmod">funasr.utils.torch_function · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/utils/torch_function.py#L100" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Subsequent mask.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>size</code> — TODO.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code>def subsequent_mask(size: torch.Tensor):
+ &quot;&quot;&quot;Subsequent mask.
+
+ Args:
+ size: TODO.
+ &quot;&quot;&quot;
+ return torch.ones(size, size).tril()
+</code></pre></details></div>
+<div class="api-detail" id="e419"><span class="dtag badge-f">function</span><h2><code>MakePadMask_test()</code></h2><div class="dmod">funasr.utils.torch_function · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/utils/torch_function.py#L109" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Makepadmask test.</p>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code>def MakePadMask_test():
+ &quot;&quot;&quot;Makepadmask test.&quot;&quot;&quot;
+ feats_length = torch.tensor([10]).type(torch.long)
+ mask_fn = MakePadMask()
+ mask = mask_fn(feats_length)
+ print(mask)
+</code></pre></details></div>
+<div class="api-detail" id="e420"><span class="dtag badge-f">function</span><h2><code>str2bool(value)</code></h2><div class="dmod">funasr.utils.type_utils · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/utils/type_utils.py#L9" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Str2bool.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>value</code> — TODO.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code>def str2bool(value: str) -&gt; bool:
+ &quot;&quot;&quot;Str2bool.
+
+ Args:
+ value: TODO.
+ &quot;&quot;&quot;
+ return bool(strtobool(value))
+</code></pre></details></div>
+<div class="api-detail" id="e421"><span class="dtag badge-f">function</span><h2><code>remove_parenthesis(value)</code></h2><div class="dmod">funasr.utils.type_utils · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/utils/type_utils.py#L18" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Remove parenthesis.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>value</code> — TODO.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code>def remove_parenthesis(value: str):
+ &quot;&quot;&quot;Remove parenthesis.
+
+ Args:
+ value: TODO.
+ &quot;&quot;&quot;
+ value = value.strip()
+ if value.startswith(&quot;(&quot;) and value.endswith(&quot;)&quot;):
+ value = value[1:-1]
+ elif value.startswith(&quot;[&quot;) and value.endswith(&quot;]&quot;):
+ value = value[1:-1]
+ return value
+</code></pre></details></div>
+<div class="api-detail" id="e422"><span class="dtag badge-f">function</span><h2><code>remove_quotes(value)</code></h2><div class="dmod">funasr.utils.type_utils · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/utils/type_utils.py#L32" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Remove quotes.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>value</code> — TODO.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code>def remove_quotes(value: str):
+ &quot;&quot;&quot;Remove quotes.
+
+ Args:
+ value: TODO.
+ &quot;&quot;&quot;
+ value = value.strip()
+ if value.startswith(&#x27;&quot;&#x27;) and value.endswith(&#x27;&quot;&#x27;):
+ value = value[1:-1]
+ elif value.startswith(&quot;&#x27;&quot;) and value.endswith(&quot;&#x27;&quot;):
+ value = value[1:-1]
+ return value
+</code></pre></details></div>
+<div class="api-detail" id="e423"><span class="dtag badge-f">function</span><h2><code>int_or_none(value)</code></h2><div class="dmod">funasr.utils.type_utils · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/utils/type_utils.py#L46" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>int_or_none.</p>
+<h4>Examples:</h4>
+<p>&gt;&gt;&gt; import argparse</p>
+<p>&gt;&gt;&gt; parser = argparse.ArgumentParser()</p>
+<p>&gt;&gt;&gt; _ = parser.add_argument(&#x27;--foo&#x27;, type=int_or_none)</p>
+<p>&gt;&gt;&gt; parser.parse_args([&#x27;--foo&#x27;, &#x27;456&#x27;])</p>
+<p>Namespace(foo=456)</p>
+<p>&gt;&gt;&gt; parser.parse_args([&#x27;--foo&#x27;, &#x27;none&#x27;])</p>
+<p>Namespace(foo=None)</p>
+<p>&gt;&gt;&gt; parser.parse_args([&#x27;--foo&#x27;, &#x27;null&#x27;])</p>
+<p>Namespace(foo=None)</p>
+<p>&gt;&gt;&gt; parser.parse_args([&#x27;--foo&#x27;, &#x27;nil&#x27;])</p>
+<p>Namespace(foo=None)</p>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code>def int_or_none(value: str) -&gt; Optional[int]:
+ &quot;&quot;&quot;int_or_none.
+
+ Examples:
+ &gt;&gt;&gt; import argparse
+ &gt;&gt;&gt; parser = argparse.ArgumentParser()
+ &gt;&gt;&gt; _ = parser.add_argument(&#x27;--foo&#x27;, type=int_or_none)
+ &gt;&gt;&gt; parser.parse_args([&#x27;--foo&#x27;, &#x27;456&#x27;])
+ Namespace(foo=456)
+ &gt;&gt;&gt; parser.parse_args([&#x27;--foo&#x27;, &#x27;none&#x27;])
+ Namespace(foo=None)
+ &gt;&gt;&gt; parser.parse_args([&#x27;--foo&#x27;, &#x27;null&#x27;])
+ Namespace(foo=None)
+ &gt;&gt;&gt; parser.parse_args([&#x27;--foo&#x27;, &#x27;nil&#x27;])
+ Namespace(foo=None)
+
+ &quot;&quot;&quot;
+ if value.strip().lower() in (&quot;none&quot;, &quot;null&quot;, &quot;nil&quot;):
+ return None
+ return int(value)
+</code></pre></details></div>
+<div class="api-detail" id="e424"><span class="dtag badge-f">function</span><h2><code>float_or_none(value)</code></h2><div class="dmod">funasr.utils.type_utils · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/utils/type_utils.py#L68" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>float_or_none.</p>
+<h4>Examples:</h4>
+<p>&gt;&gt;&gt; import argparse</p>
+<p>&gt;&gt;&gt; parser = argparse.ArgumentParser()</p>
+<p>&gt;&gt;&gt; _ = parser.add_argument(&#x27;--foo&#x27;, type=float_or_none)</p>
+<p>&gt;&gt;&gt; parser.parse_args([&#x27;--foo&#x27;, &#x27;4.5&#x27;])</p>
+<p>Namespace(foo=4.5)</p>
+<p>&gt;&gt;&gt; parser.parse_args([&#x27;--foo&#x27;, &#x27;none&#x27;])</p>
+<p>Namespace(foo=None)</p>
+<p>&gt;&gt;&gt; parser.parse_args([&#x27;--foo&#x27;, &#x27;null&#x27;])</p>
+<p>Namespace(foo=None)</p>
+<p>&gt;&gt;&gt; parser.parse_args([&#x27;--foo&#x27;, &#x27;nil&#x27;])</p>
+<p>Namespace(foo=None)</p>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code>def float_or_none(value: str) -&gt; Optional[float]:
+ &quot;&quot;&quot;float_or_none.
+
+ Examples:
+ &gt;&gt;&gt; import argparse
+ &gt;&gt;&gt; parser = argparse.ArgumentParser()
+ &gt;&gt;&gt; _ = parser.add_argument(&#x27;--foo&#x27;, type=float_or_none)
+ &gt;&gt;&gt; parser.parse_args([&#x27;--foo&#x27;, &#x27;4.5&#x27;])
+ Namespace(foo=4.5)
+ &gt;&gt;&gt; parser.parse_args([&#x27;--foo&#x27;, &#x27;none&#x27;])
+ Namespace(foo=None)
+ &gt;&gt;&gt; parser.parse_args([&#x27;--foo&#x27;, &#x27;null&#x27;])
+ Namespace(foo=None)
+ &gt;&gt;&gt; parser.parse_args([&#x27;--foo&#x27;, &#x27;nil&#x27;])
+ Namespace(foo=None)
+
+ &quot;&quot;&quot;
+ if value.strip().lower() in (&quot;none&quot;, &quot;null&quot;, &quot;nil&quot;):
+ return None
+ return float(value)
+</code></pre></details></div>
+<div class="api-detail" id="e425"><span class="dtag badge-f">function</span><h2><code>humanfriendly_parse_size_or_none(value)</code></h2><div class="dmod">funasr.utils.type_utils · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/utils/type_utils.py#L90" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Humanfriendly parse size or none.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>value</code> — TODO.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code>def humanfriendly_parse_size_or_none(value) -&gt; Optional[float]:
+ &quot;&quot;&quot;Humanfriendly parse size or none.
+
+ Args:
+ value: TODO.
+ &quot;&quot;&quot;
+ if value.strip().lower() in (&quot;none&quot;, &quot;null&quot;, &quot;nil&quot;):
+ return None
+ return humanfriendly.parse_size(value)
+</code></pre></details></div>
+<div class="api-detail" id="e426"><span class="dtag badge-f">function</span><h2><code>str_or_int(value)</code></h2><div class="dmod">funasr.utils.type_utils · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/utils/type_utils.py#L101" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Str or int.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>value</code> — TODO.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code>def str_or_int(value: str) -&gt; Union[str, int]:
+ &quot;&quot;&quot;Str or int.
+
+ Args:
+ value: TODO.
+ &quot;&quot;&quot;
+ try:
+ return int(value)
+ except ValueError:
+ return value
+</code></pre></details></div>
+<div class="api-detail" id="e427"><span class="dtag badge-f">function</span><h2><code>str_or_none(value)</code></h2><div class="dmod">funasr.utils.type_utils · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/utils/type_utils.py#L113" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>str_or_none.</p>
+<h4>Examples:</h4>
+<p>&gt;&gt;&gt; import argparse</p>
+<p>&gt;&gt;&gt; parser = argparse.ArgumentParser()</p>
+<p>&gt;&gt;&gt; _ = parser.add_argument(&#x27;--foo&#x27;, type=str_or_none)</p>
+<p>&gt;&gt;&gt; parser.parse_args([&#x27;--foo&#x27;, &#x27;aaa&#x27;])</p>
+<p>Namespace(foo=&#x27;aaa&#x27;)</p>
+<p>&gt;&gt;&gt; parser.parse_args([&#x27;--foo&#x27;, &#x27;none&#x27;])</p>
+<p>Namespace(foo=None)</p>
+<p>&gt;&gt;&gt; parser.parse_args([&#x27;--foo&#x27;, &#x27;null&#x27;])</p>
+<p>Namespace(foo=None)</p>
+<p>&gt;&gt;&gt; parser.parse_args([&#x27;--foo&#x27;, &#x27;nil&#x27;])</p>
+<p>Namespace(foo=None)</p>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code>def str_or_none(value: str) -&gt; Optional[str]:
+ &quot;&quot;&quot;str_or_none.
+
+ Examples:
+ &gt;&gt;&gt; import argparse
+ &gt;&gt;&gt; parser = argparse.ArgumentParser()
+ &gt;&gt;&gt; _ = parser.add_argument(&#x27;--foo&#x27;, type=str_or_none)
+ &gt;&gt;&gt; parser.parse_args([&#x27;--foo&#x27;, &#x27;aaa&#x27;])
+ Namespace(foo=&#x27;aaa&#x27;)
+ &gt;&gt;&gt; parser.parse_args([&#x27;--foo&#x27;, &#x27;none&#x27;])
+ Namespace(foo=None)
+ &gt;&gt;&gt; parser.parse_args([&#x27;--foo&#x27;, &#x27;null&#x27;])
+ Namespace(foo=None)
+ &gt;&gt;&gt; parser.parse_args([&#x27;--foo&#x27;, &#x27;nil&#x27;])
+ Namespace(foo=None)
+
+ &quot;&quot;&quot;
+ if value.strip().lower() in (&quot;none&quot;, &quot;null&quot;, &quot;nil&quot;):
+ return None
+ return value
+</code></pre></details></div>
+<div class="api-detail" id="e428"><span class="dtag badge-f">function</span><h2><code>str2pair_str(value)</code></h2><div class="dmod">funasr.utils.type_utils · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/utils/type_utils.py#L135" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>str2pair_str.</p>
+<h4>Examples:</h4>
+<p>&gt;&gt;&gt; import argparse</p>
+<p>&gt;&gt;&gt; str2pair_str(&#x27;abc,def &#x27;)</p>
+<p>(&#x27;abc&#x27;, &#x27;def&#x27;)</p>
+<p>&gt;&gt;&gt; parser = argparse.ArgumentParser()</p>
+<p>&gt;&gt;&gt; _ = parser.add_argument(&#x27;--foo&#x27;, type=str2pair_str)</p>
+<p>&gt;&gt;&gt; parser.parse_args([&#x27;--foo&#x27;, &#x27;abc,def&#x27;])</p>
+<p>Namespace(foo=(&#x27;abc&#x27;, &#x27;def&#x27;))</p>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code>def str2pair_str(value: str) -&gt; Tuple[str, str]:
+ &quot;&quot;&quot;str2pair_str.
+
+ Examples:
+ &gt;&gt;&gt; import argparse
+ &gt;&gt;&gt; str2pair_str(&#x27;abc,def &#x27;)
+ (&#x27;abc&#x27;, &#x27;def&#x27;)
+ &gt;&gt;&gt; parser = argparse.ArgumentParser()
+ &gt;&gt;&gt; _ = parser.add_argument(&#x27;--foo&#x27;, type=str2pair_str)
+ &gt;&gt;&gt; parser.parse_args([&#x27;--foo&#x27;, &#x27;abc,def&#x27;])
+ Namespace(foo=(&#x27;abc&#x27;, &#x27;def&#x27;))
+
+ &quot;&quot;&quot;
+ value = remove_parenthesis(value)
+ a, b = value.split(&quot;,&quot;)
+
+ # Workaround for configargparse issues:
+ # If the list values are given from yaml file,
+ # the value givent to type() is shaped as python-list,
+ # e.g. [&#x27;a&#x27;, &#x27;b&#x27;, &#x27;c&#x27;],
+ # so we need to remove double quotes from it.
+ return remove_quotes(a), remove_quotes(b)
+</code></pre></details></div>
+<div class="api-detail" id="e429"><span class="dtag badge-f">function</span><h2><code>str2triple_str(value)</code></h2><div class="dmod">funasr.utils.type_utils · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/utils/type_utils.py#L159" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>str2triple_str.</p>
+<h4>Examples:</h4>
+<p>&gt;&gt;&gt; str2triple_str(&#x27;abc,def ,ghi&#x27;)</p>
+<p>(&#x27;abc&#x27;, &#x27;def&#x27;, &#x27;ghi&#x27;)</p>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code>def str2triple_str(value: str) -&gt; Tuple[str, str, str]:
+ &quot;&quot;&quot;str2triple_str.
+
+ Examples:
+ &gt;&gt;&gt; str2triple_str(&#x27;abc,def ,ghi&#x27;)
+ (&#x27;abc&#x27;, &#x27;def&#x27;, &#x27;ghi&#x27;)
+ &quot;&quot;&quot;
+ value = remove_parenthesis(value)
+ a, b, c = value.split(&quot;,&quot;)
+
+ # Workaround for configargparse issues:
+ # If the list values are given from yaml file,
+ # the value givent to type() is shaped as python-list,
+ # e.g. [&#x27;a&#x27;, &#x27;b&#x27;, &#x27;c&#x27;],
+ # so we need to remove quotes from it.
+ return remove_quotes(a), remove_quotes(b), remove_quotes(c)
+</code></pre></details></div>
+<div class="api-detail" id="e430"><span class="dtag badge-f">function</span><h2><code>slice_padding_fbank(speech, speech_lengths, vad_segments)</code></h2><div class="dmod">funasr.utils.vad_utils · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/utils/vad_utils.py#L5" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Slice padding fbank.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>speech</code> — Speech audio tensor, shape (batch, time).</li>
+<li><code>speech_lengths</code> — Length of each speech sample.</li>
+<li><code>vad_segments</code> — TODO.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code>def slice_padding_fbank(speech, speech_lengths, vad_segments):
+ &quot;&quot;&quot;Slice padding fbank.
+
+ Args:
+ speech: Speech audio tensor, shape (batch, time).
+ speech_lengths: Length of each speech sample.
+ vad_segments: TODO.
+ &quot;&quot;&quot;
+ speech_list = []
+ speech_lengths_list = []
+ for i, segment in enumerate(vad_segments):
+
+ bed_idx = int(segment[0][0] * 16)
+ end_idx = min(int(segment[0][1] * 16), speech_lengths[0])
+ speech_i = speech[0, bed_idx:end_idx]
+ speech_lengths_i = end_idx - bed_idx
+ speech_list.append(speech_i)
+ speech_lengths_list.append(speech_lengths_i)
+ feats_pad = pad_sequence(speech_list, batch_first=True, padding_value=0.0)
+ speech_lengths_pad = torch.Tensor(speech_lengths_list).int()
+ return feats_pad, speech_lengths_pad
+</code></pre></details></div>
+<div class="api-detail" id="e431"><span class="dtag badge-f">function</span><h2><code>slice_padding_audio_samples(speech, speech_lengths, vad_segments)</code></h2><div class="dmod">funasr.utils.vad_utils · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/utils/vad_utils.py#L28" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Slice audio into VAD segments with proper padding.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>speech (Tensor)</code> — Full audio tensor.</li>
+<li><code>speech_lengths (int)</code> — Total audio length.</li>
+<li><code>vad_segments (list)</code> — List of (segment_info, original_index) tuples,</li>
+<li class="sub">where segment_info is [start_ms, end_ms].</li>
+</ul>
+<h4>Returns:</h4>
+<ul>
+<li><code>tuple</code> — (speech_list, speech_lengths_list) - lists of numpy arrays and their lengths.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code>def slice_padding_audio_samples(speech, speech_lengths, vad_segments):
+
+ &quot;&quot;&quot;Slice audio into VAD segments with proper padding.
+
+ Args:
+ speech (Tensor): Full audio tensor.
+ speech_lengths (int): Total audio length.
+ vad_segments (list): List of (segment_info, original_index) tuples,
+ where segment_info is [start_ms, end_ms].
+
+ Returns:
+ tuple: (speech_list, speech_lengths_list) - lists of numpy arrays and their lengths.
+ &quot;&quot;&quot;
+ speech_list = []
+ speech_lengths_list = []
+ for i, segment in enumerate(vad_segments):
+ bed_idx = int(segment[0][0] * 16)
+ end_idx = min(int(segment[0][1] * 16), speech_lengths)
+ speech_i = speech[bed_idx:end_idx]
+ speech_lengths_i = end_idx - bed_idx
+ speech_list.append(speech_i)
+ speech_lengths_list.append(speech_lengths_i)
+
+ return speech_list, speech_lengths_list
+</code></pre></details></div>
+<div class="api-detail" id="e432"><span class="dtag badge-f">function</span><h2><code>merge_vad(vad_result, max_length, min_length)</code></h2><div class="dmod">funasr.utils.vad_utils · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/utils/vad_utils.py#L54" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Merge short VAD segments to reduce fragmentation.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>vad_result (list)</code> — VAD segments [[start_ms, end_ms], ...].</li>
+<li><code>max_length (int)</code> — Maximum merged segment length in ms (default 15000).</li>
+<li><code>min_length (int)</code> — Minimum segment length; shorter ones get merged (default 0).</li>
+</ul>
+<h4>Returns:</h4>
+<ul>
+<li><code>list</code> — Merged VAD segments [[start_ms, end_ms], ...].</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code>def merge_vad(vad_result, max_length=15000, min_length=0):
+
+ &quot;&quot;&quot;Merge short VAD segments to reduce fragmentation.
+
+ Args:
+ vad_result (list): VAD segments [[start_ms, end_ms], ...].
+ max_length (int): Maximum merged segment length in ms (default 15000).
+ min_length (int): Minimum segment length; shorter ones get merged (default 0).
+
+ Returns:
+ list: Merged VAD segments [[start_ms, end_ms], ...].
+ &quot;&quot;&quot;
+ new_result = []
+ if len(vad_result) &lt;= 1:
+ return vad_result
+ time_step = [t[0] for t in vad_result] + [t[1] for t in vad_result]
+ time_step = sorted(list(set(time_step)))
+ if len(time_step) == 0:
+ return []
+ bg = 0
+ for i in range(len(time_step) - 1):
+ time = time_step[i]
+ if time_step[i + 1] - bg &lt; max_length:
+ continue
+ if time - bg &gt; min_length:
+ new_result.append([bg, time])
+ # if time - bg &lt; max_length * 1.5:
+ # new_result.append([bg, time])
+ # else:
+ # split_num = int(time - bg) // max_length + 1
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/utils/vad_utils.py#L54" target="_blank" class="src-more">View full source on GitHub →</a></details></div>
+<div class="api-detail" id="e433"><span class="dtag badge-f">function</span><h2><code>get_pypi_version(package_name)</code></h2><div class="dmod">funasr.utils.version_checker · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/utils/version_checker.py#L5" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Get pypi version.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>package_name</code> — TODO.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code>def get_pypi_version(package_name):
+ &quot;&quot;&quot;Get pypi version.
+
+ Args:
+ package_name: TODO.
+ &quot;&quot;&quot;
+ import requests
+
+ url = f&quot;https://pypi.org/pypi/{package_name}/json&quot;
+ response = requests.get(url)
+ if response.status_code == 200:
+ data = response.json()
+ return version.parse(data[&quot;info&quot;][&quot;version&quot;])
+ else:
+ raise Exception(&quot;Failed to retrieve version information from PyPI.&quot;)
+</code></pre></details></div>
+<div class="api-detail" id="e434"><span class="dtag badge-f">function</span><h2><code>check_for_update(disable)</code></h2><div class="dmod">funasr.utils.version_checker · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/utils/version_checker.py#L22" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Check for update.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>disable</code> — TODO.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code>def check_for_update(disable=False):
+ &quot;&quot;&quot;Check for update.
+
+ Args:
+ disable: TODO.
+ &quot;&quot;&quot;
+ current_version = version.parse(__version__)
+ print(f&quot;funasr version: {current_version}.&quot;)
+ if disable:
+ return
+
+ print(
+ &quot;Check update of funasr, and it would cost few times. You may disable it by set `disable_update=True` in AutoModel&quot;
+ )
+ pypi_version = get_pypi_version(&quot;funasr&quot;)
+
+ if current_version &lt; pypi_version:
+ print(f&quot;New version is available: {pypi_version}.&quot;)
+ print(&#x27;Please use the command &quot;pip install -U funasr&quot; to upgrade.&#x27;)
+ else:
+ print(f&quot;You are using the latest version of funasr-{current_version}&quot;)
+</code></pre></details></div>
+<div class="api-detail" id="e435"><span class="dtag badge-f">function</span><h2><code>main_hydra(kwargs)</code></h2><div class="dmod">funasr.bin.compute_audio_cmvn · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/bin/compute_audio_cmvn.py#L15" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Main hydra.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>kwargs</code> — Additional keyword arguments.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code>def main_hydra(kwargs: DictConfig):
+ &quot;&quot;&quot;Main hydra.
+
+ Args:
+ kwargs: Additional keyword arguments.
+ &quot;&quot;&quot;
+ if kwargs.get(&quot;debug&quot;, False):
+ import pdb
+
+ pdb.set_trace()
+
+ assert &quot;model&quot; in kwargs
+ if &quot;model_conf&quot; not in kwargs:
+ logging.info(&quot;download models from model hub: {}&quot;.format(kwargs.get(&quot;hub&quot;, &quot;ms&quot;)))
+ kwargs = download_model(is_training=kwargs.get(&quot;is_training&quot;, True), **kwargs)
+
+ main(**kwargs)
+</code></pre></details></div>
+<div class="api-detail" id="e436"><span class="dtag badge-f">function</span><h2><code>main(**kwargs)</code></h2><div class="dmod">funasr.bin.compute_audio_cmvn · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/bin/compute_audio_cmvn.py#L34" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Main.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>**kwargs</code> — Additional keyword arguments.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code>def main(**kwargs):
+ &quot;&quot;&quot;Main.
+
+ Args:
+ **kwargs: Additional keyword arguments.
+ &quot;&quot;&quot;
+ print(kwargs)
+ # set random seed
+ # tables.print()
+ set_all_random_seed(kwargs.get(&quot;seed&quot;, 0))
+ torch.backends.cudnn.enabled = kwargs.get(&quot;cudnn_enabled&quot;, torch.backends.cudnn.enabled)
+ torch.backends.cudnn.benchmark = kwargs.get(&quot;cudnn_benchmark&quot;, torch.backends.cudnn.benchmark)
+ torch.backends.cudnn.deterministic = kwargs.get(&quot;cudnn_deterministic&quot;, True)
+
+ tokenizer = kwargs.get(&quot;tokenizer&quot;, None)
+
+ # build frontend if frontend is none None
+ frontend = kwargs.get(&quot;frontend&quot;, None)
+ if frontend is not None:
+ frontend_class = tables.frontend_classes.get(frontend)
+ frontend = frontend_class(**kwargs[&quot;frontend_conf&quot;])
+ kwargs[&quot;frontend&quot;] = frontend
+ kwargs[&quot;input_size&quot;] = frontend.output_size()
+
+ # dataset
+ dataset_class = tables.dataset_classes.get(kwargs.get(&quot;dataset&quot;, &quot;AudioDataset&quot;))
+ dataset_train = dataset_class(
+ kwargs.get(&quot;train_data_set_list&quot;),
+ frontend=frontend,
+ tokenizer=None,
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/bin/compute_audio_cmvn.py#L34" target="_blank" class="src-more">View full source on GitHub →</a></details></div>
+<div class="api-detail" id="e437"><span class="dtag badge-f">function</span><h2><code>main_hydra(cfg)</code></h2><div class="dmod">funasr.bin.export · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/bin/export.py#L10" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Main hydra.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>cfg</code> — Configuration overrides.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code>def main_hydra(cfg: DictConfig):
+ &quot;&quot;&quot;Main hydra.
+
+ Args:
+ cfg: Configuration overrides.
+ &quot;&quot;&quot;
+ def to_plain_list(cfg_item):
+ &quot;&quot;&quot;To plain list.
+
+ Args:
+ cfg_item: TODO.
+ &quot;&quot;&quot;
+ if isinstance(cfg_item, ListConfig):
+ return OmegaConf.to_container(cfg_item, resolve=True)
+ elif isinstance(cfg_item, DictConfig):
+ return {k: to_plain_list(v) for k, v in cfg_item.items()}
+ else:
+ return cfg_item
+
+ kwargs = to_plain_list(cfg)
+
+ if kwargs.get(&quot;debug&quot;, False):
+ import pdb
+
+ pdb.set_trace()
+
+ if &quot;device&quot; not in kwargs:
+ kwargs[&quot;device&quot;] = &quot;cpu&quot;
+ model = AutoModel(**kwargs)
+
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/bin/export.py#L10" target="_blank" class="src-more">View full source on GitHub →</a></details></div>
+<div class="api-detail" id="e438"><span class="dtag badge-f">function</span><h2><code>main_hydra(cfg)</code></h2><div class="dmod">funasr.bin.inference · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/bin/inference.py#L9" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Main hydra.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>cfg</code> — Configuration overrides.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code>def main_hydra(cfg: DictConfig):
+ &quot;&quot;&quot;Main hydra.
+
+ Args:
+ cfg: Configuration overrides.
+ &quot;&quot;&quot;
+ def to_plain_list(cfg_item):
+ &quot;&quot;&quot;To plain list.
+
+ Args:
+ cfg_item: TODO.
+ &quot;&quot;&quot;
+ if isinstance(cfg_item, ListConfig):
+ return OmegaConf.to_container(cfg_item, resolve=True)
+ elif isinstance(cfg_item, DictConfig):
+ return {k: to_plain_list(v) for k, v in cfg_item.items()}
+ else:
+ return cfg_item
+
+ kwargs = to_plain_list(cfg)
+
+ if kwargs.get(&quot;debug&quot;, False):
+ import pdb
+
+ pdb.set_trace()
+ model = AutoModel(**kwargs)
+ res = model.generate(input=kwargs[&quot;input&quot;])
+ print(res)
+</code></pre></details></div>
+<div class="api-detail" id="e439"><span class="dtag badge-f">function</span><h2><code>field2slice(field)</code></h2><div class="dmod">funasr.bin.tokenize_text · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/bin/tokenize_text.py#L19" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Convert field string to slice</p>
+<p>Note that field string accepts 1-based integer.</p>
+<h4>Examples:</h4>
+<p>&gt;&gt;&gt; field2slice(&quot;1-&quot;)</p>
+<p>slice(0, None, None)</p>
+<p>&gt;&gt;&gt; field2slice(&quot;1-3&quot;)</p>
+<p>slice(0, 3, None)</p>
+<p>&gt;&gt;&gt; field2slice(&quot;-3&quot;)</p>
+<p>slice(None, 3, None)</p>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code>def field2slice(field: Optional[str]) -&gt; slice:
+ &quot;&quot;&quot;Convert field string to slice
+
+ Note that field string accepts 1-based integer.
+
+ Examples:
+ &gt;&gt;&gt; field2slice(&quot;1-&quot;)
+ slice(0, None, None)
+ &gt;&gt;&gt; field2slice(&quot;1-3&quot;)
+ slice(0, 3, None)
+ &gt;&gt;&gt; field2slice(&quot;-3&quot;)
+ slice(None, 3, None)
+ &quot;&quot;&quot;
+ field = field.strip()
+ try:
+ if &quot;-&quot; in field:
+ # e.g. &quot;2-&quot; or &quot;2-5&quot; or &quot;-7&quot;
+ s1, s2 = field.split(&quot;-&quot;, maxsplit=1)
+ if s1.strip() == &quot;&quot;:
+ s1 = None
+ else:
+ s1 = int(s1)
+ if s1 == 0:
+ raise ValueError(&quot;1-based string&quot;)
+ if s2.strip() == &quot;&quot;:
+ s2 = None
+ else:
+ s2 = int(s2)
+ else:
+ # e.g. &quot;2&quot;
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/bin/tokenize_text.py#L19" target="_blank" class="src-more">View full source on GitHub →</a></details></div>
+<div class="api-detail" id="e440"><span class="dtag badge-f">function</span><h2><code>tokenize(input, output, field, delimiter, token_type, space_symbol, non_linguistic_symbols, bpemodel, log_level, write_vocabulary, vocabulary_size, remove_non_linguistic_symbols, cutoff, add_symbol, cleaner, g2p)</code></h2><div class="dmod">funasr.bin.tokenize_text · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/bin/tokenize_text.py#L65" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Tokenize.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>input</code> — Input audio/text data.</li>
+<li><code>output</code> — TODO.</li>
+<li><code>field</code> — TODO.</li>
+<li><code>delimiter</code> — TODO.</li>
+<li><code>token_type</code> — TODO.</li>
+<li><code>space_symbol</code> — TODO.</li>
+<li><code>non_linguistic_symbols</code> — TODO.</li>
+<li><code>bpemodel</code> — TODO.</li>
+<li><code>log_level</code> — TODO.</li>
+<li><code>write_vocabulary</code> — TODO.</li>
+<li><code>vocabulary_size</code> — Size/dimension parameter.</li>
+<li><code>remove_non_linguistic_symbols</code> — TODO.</li>
+<li><code>cutoff</code> — TODO.</li>
+<li><code>add_symbol</code> — TODO.</li>
+<li><code>cleaner</code> — TODO.</li>
+<li><code>g2p</code> — TODO.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code>def tokenize(
+ input: str,
+ output: str,
+ field: Optional[str],
+ delimiter: Optional[str],
+ token_type: str,
+ space_symbol: str,
+ non_linguistic_symbols: Optional[str],
+ bpemodel: Optional[str],
+ log_level: str,
+ write_vocabulary: bool,
+ vocabulary_size: int,
+ remove_non_linguistic_symbols: bool,
+ cutoff: int,
+ add_symbol: List[str],
+ cleaner: Optional[str],
+ g2p: Optional[str],
+):
+
+ &quot;&quot;&quot;Tokenize.
+
+ Args:
+ input: Input audio/text data.
+ output: TODO.
+ field: TODO.
+ delimiter: TODO.
+ token_type: TODO.
+ space_symbol: TODO.
+ non_linguistic_symbols: TODO.
+ bpemodel: TODO.
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/bin/tokenize_text.py#L65" target="_blank" class="src-more">View full source on GitHub →</a></details></div>
+<div class="api-detail" id="e441"><span class="dtag badge-f">function</span><h2><code>get_parser()</code></h2><div class="dmod">funasr.bin.tokenize_text · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/bin/tokenize_text.py#L207" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Get parser.</p>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code>def get_parser() -&gt; argparse.ArgumentParser:
+ &quot;&quot;&quot;Get parser.&quot;&quot;&quot;
+ parser = argparse.ArgumentParser(
+ description=&quot;Tokenize texts&quot;,
+ formatter_class=argparse.ArgumentDefaultsHelpFormatter,
+ )
+ parser.add_argument(
+ &quot;--log_level&quot;,
+ type=lambda x: x.upper(),
+ default=&quot;INFO&quot;,
+ choices=(&quot;CRITICAL&quot;, &quot;ERROR&quot;, &quot;WARNING&quot;, &quot;INFO&quot;, &quot;DEBUG&quot;, &quot;NOTSET&quot;),
+ help=&quot;The verbose level of logging&quot;,
+ )
+
+ parser.add_argument(&quot;--input&quot;, &quot;-i&quot;, required=True, help=&quot;Input text. - indicates sys.stdin&quot;)
+ parser.add_argument(&quot;--output&quot;, &quot;-o&quot;, required=True, help=&quot;Output text. - indicates sys.stdout&quot;)
+ parser.add_argument(
+ &quot;--field&quot;,
+ &quot;-f&quot;,
+ help=&quot;The target columns of the input text as 1-based integer. e.g 2-&quot;,
+ )
+ parser.add_argument(
+ &quot;--token_type&quot;,
+ &quot;-t&quot;,
+ default=&quot;char&quot;,
+ choices=[&quot;char&quot;, &quot;bpe&quot;, &quot;word&quot;, &quot;phn&quot;],
+ help=&quot;Token type&quot;,
+ )
+ parser.add_argument(&quot;--delimiter&quot;, &quot;-d&quot;, default=None, help=&quot;The delimiter&quot;)
+ parser.add_argument(&quot;--space_symbol&quot;, default=&quot;&lt;space&gt;&quot;, help=&quot;The space symbol&quot;)
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/bin/tokenize_text.py#L207" target="_blank" class="src-more">View full source on GitHub →</a></details></div>
+<div class="api-detail" id="e442"><span class="dtag badge-f">function</span><h2><code>main(cmd)</code></h2><div class="dmod">funasr.bin.tokenize_text · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/bin/tokenize_text.py#L289" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Main.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>cmd</code> — TODO.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code>def main(cmd=None):
+ &quot;&quot;&quot;Main.
+
+ Args:
+ cmd: TODO.
+ &quot;&quot;&quot;
+ print(get_commandline_args(), file=sys.stderr)
+ parser = get_parser()
+ args = parser.parse_args(cmd)
+ kwargs = vars(args)
+ tokenize(**kwargs)
+</code></pre></details></div>
+<div class="api-detail" id="e443"><span class="dtag badge-f">function</span><h2><code>main_hydra(kwargs)</code></h2><div class="dmod">funasr.bin.train · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/bin/train.py#L42" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Main hydra.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>kwargs</code> — Additional keyword arguments.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code>def main_hydra(kwargs: DictConfig):
+ &quot;&quot;&quot;Main hydra.
+
+ Args:
+ kwargs: Additional keyword arguments.
+ &quot;&quot;&quot;
+ if kwargs.get(&quot;debug&quot;, False):
+ import pdb
+
+ pdb.set_trace()
+
+ assert &quot;model&quot; in kwargs
+ if &quot;model_conf&quot; not in kwargs:
+ logging.info(&quot;download models from model hub: {}&quot;.format(kwargs.get(&quot;hub&quot;, &quot;ms&quot;)))
+ kwargs = download_model(is_training=kwargs.get(&quot;is_training&quot;, True), **kwargs)
+
+ main(**kwargs)
+</code></pre></details></div>
+<div class="api-detail" id="e444"><span class="dtag badge-f">function</span><h2><code>main(**kwargs)</code></h2><div class="dmod">funasr.bin.train · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/bin/train.py#L61" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Main.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>**kwargs</code> — Additional keyword arguments.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code>def main(**kwargs):
+
+ # set random seed
+ &quot;&quot;&quot;Main.
+
+ Args:
+ **kwargs: Additional keyword arguments.
+ &quot;&quot;&quot;
+ set_all_random_seed(kwargs.get(&quot;seed&quot;, 0))
+ torch.backends.cudnn.enabled = kwargs.get(&quot;cudnn_enabled&quot;, torch.backends.cudnn.enabled)
+ torch.backends.cudnn.benchmark = kwargs.get(&quot;cudnn_benchmark&quot;, torch.backends.cudnn.benchmark)
+ torch.backends.cudnn.deterministic = kwargs.get(&quot;cudnn_deterministic&quot;, True)
+ # open tf32
+ torch.backends.cuda.matmul.allow_tf32 = kwargs.get(&quot;enable_tf32&quot;, True)
+
+ local_rank = int(os.environ.get(&quot;LOCAL_RANK&quot;, 0))
+ if local_rank == 0:
+ tables.print()
+ # Check if we are using DDP or FSDP
+ use_ddp = &quot;WORLD_SIZE&quot; in os.environ and int(os.environ[&quot;WORLD_SIZE&quot;]) &gt; 1
+ use_fsdp = kwargs.get(&quot;use_fsdp&quot;, False)
+ # use_ddp = False if use_fsdp else use_fsdp
+ if use_ddp or use_fsdp:
+ dist.init_process_group(backend=kwargs.get(&quot;backend&quot;, &quot;nccl&quot;), init_method=&quot;env://&quot;)
+ torch.cuda.set_device(local_rank)
+
+ logging.info(&quot;Build model, frontend, tokenizer&quot;)
+ device = kwargs.get(&quot;device&quot;, &quot;cuda&quot;)
+ kwargs[&quot;device&quot;] = &quot;cpu&quot;
+ model = AutoModel(**kwargs)
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/bin/train.py#L61" target="_blank" class="src-more">View full source on GitHub →</a></details></div>
+<div class="api-detail" id="e445"><span class="dtag badge-f">function</span><h2><code>main_hydra(kwargs)</code></h2><div class="dmod">funasr.bin.train_ds · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/bin/train_ds.py#L45" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Main hydra.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>kwargs</code> — Additional keyword arguments.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code>def main_hydra(kwargs: DictConfig):
+ &quot;&quot;&quot;Main hydra.
+
+ Args:
+ kwargs: Additional keyword arguments.
+ &quot;&quot;&quot;
+ if kwargs.get(&quot;debug&quot;, False):
+ import pdb
+
+ pdb.set_trace()
+
+ assert &quot;model&quot; in kwargs
+ if &quot;model_conf&quot; not in kwargs:
+ logging.info(&quot;download models from model hub: {}&quot;.format(kwargs.get(&quot;hub&quot;, &quot;ms&quot;)))
+ kwargs = download_model(is_training=kwargs.get(&quot;is_training&quot;, True), **kwargs)
+
+ main(**kwargs)
+</code></pre></details></div>
+<div class="api-detail" id="e446"><span class="dtag badge-f">function</span><h2><code>main(**kwargs)</code></h2><div class="dmod">funasr.bin.train_ds · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/bin/train_ds.py#L64" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Main.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>**kwargs</code> — Additional keyword arguments.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code>def main(**kwargs):
+
+ # set random seed
+ &quot;&quot;&quot;Main.
+
+ Args:
+ **kwargs: Additional keyword arguments.
+ &quot;&quot;&quot;
+ set_all_random_seed(kwargs.get(&quot;seed&quot;, 0))
+ torch.backends.cudnn.enabled = kwargs.get(&quot;cudnn_enabled&quot;, torch.backends.cudnn.enabled)
+ torch.backends.cudnn.benchmark = kwargs.get(&quot;cudnn_benchmark&quot;, torch.backends.cudnn.benchmark)
+ torch.backends.cudnn.deterministic = kwargs.get(&quot;cudnn_deterministic&quot;, True)
+ # open tf32
+ torch.backends.cuda.matmul.allow_tf32 = kwargs.get(&quot;enable_tf32&quot;, True)
+
+ rank = int(os.environ.get(&quot;RANK&quot;, 0))
+ local_rank = int(os.environ.get(&quot;LOCAL_RANK&quot;, 0))
+ world_size = int(os.environ.get(&quot;WORLD_SIZE&quot;, 1))
+
+ if local_rank == 0:
+ tables.print()
+
+ use_ddp = world_size &gt; 1
+ use_fsdp = kwargs.get(&quot;use_fsdp&quot;, False)
+ use_deepspeed = kwargs.get(&quot;use_deepspeed&quot;, False)
+ if use_deepspeed:
+ logging.info(f&quot;use_deepspeed: {use_deepspeed}&quot;)
+ deepspeed.init_distributed(dist_backend=kwargs.get(&quot;backend&quot;, &quot;nccl&quot;))
+ elif use_ddp or use_fsdp:
+ logging.info(f&quot;use_ddp: {use_ddp}, use_fsdp: {use_fsdp}&quot;)
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/bin/train_ds.py#L64" target="_blank" class="src-more">View full source on GitHub →</a></details></div>
+<div class="api-detail" id="e447"><span class="dtag badge-f">function</span><h2><code>download_dataset()</code></h2><div class="dmod">funasr.download.download_dataset_from_hub · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/download/download_dataset_from_hub.py#L1" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Download dataset.</p>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code>def download_dataset():
+ &quot;&quot;&quot;Download dataset.&quot;&quot;&quot;
+ pass
+</code></pre></details></div>
+<div class="api-detail" id="e448"><span class="dtag badge-f">function</span><h2><code>download_dataset_from_ms(**kwargs)</code></h2><div class="dmod">funasr.download.download_dataset_from_hub · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/download/download_dataset_from_hub.py#L6" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Download dataset from ms.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>**kwargs</code> — Additional keyword arguments.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code>def download_dataset_from_ms(**kwargs):
+ &quot;&quot;&quot;Download dataset from ms.
+
+ Args:
+ **kwargs: Additional keyword arguments.
+ &quot;&quot;&quot;
+ from modelscope.msdatasets import MsDataset
+
+ dataset_name = kwargs.get(&quot;dataset_name&quot;, &quot;speech_asr/speech_asr_aishell1_trainsets&quot;)
+ subset_name = kwargs.get(&quot;subset_name&quot;, &quot;default&quot;)
+ split = kwargs.get(&quot;split&quot;, &quot;train&quot;)
+ data_dump_dir = kwargs.get(&quot;data_dump_dir&quot;, None)
+ ds = MsDataset.load(
+ dataset_name=dataset_name, subset_name=subset_name, split=split, cache_dir=data_dump_dir
+ )
+</code></pre></details></div>
+<div class="api-detail" id="e449"><span class="dtag badge-f">function</span><h2><code>download_model(**kwargs)</code></h2><div class="dmod">funasr.download.download_model_from_hub · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/download/download_model_from_hub.py#L9" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Download model from hub and parse its configuration.</p>
+<p>Resolves model name aliases, downloads from ModelScope or HuggingFace,</p>
+<p>reads config.yaml and configuration.json, and returns complete kwargs</p>
+<p>for model instantiation.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>**kwargs</code> — Must include &#x27;model&#x27; (str). Optional: &#x27;hub&#x27;, &#x27;model_revision&#x27;,</li>
+<li class="sub">&#x27;is_training&#x27;, etc.</li>
+</ul>
+<h4>Returns:</h4>
+<ul>
+<li><code>dict</code> — Complete kwargs with resolved paths, model class name, and config.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code>def download_model(**kwargs):
+
+ &quot;&quot;&quot;Download model from hub and parse its configuration.
+
+ Resolves model name aliases, downloads from ModelScope or HuggingFace,
+ reads config.yaml and configuration.json, and returns complete kwargs
+ for model instantiation.
+
+ Args:
+ **kwargs: Must include &#x27;model&#x27; (str). Optional: &#x27;hub&#x27;, &#x27;model_revision&#x27;,
+ &#x27;is_training&#x27;, etc.
+
+ Returns:
+ dict: Complete kwargs with resolved paths, model class name, and config.
+ &quot;&quot;&quot;
+ hub = kwargs.get(&quot;hub&quot;, &quot;ms&quot;)
+ if hub == &quot;ms&quot; or hub == &quot;modelscope&quot;:
+ kwargs = download_from_ms(**kwargs)
+ elif hub == &quot;hf&quot; or hub == &quot;huggingface&quot;:
+ kwargs = download_from_hf(**kwargs)
+ elif hub == &quot;openai&quot;:
+ model_or_path = kwargs.get(&quot;model&quot;)
+ if os.path.exists(model_or_path):
+ # local path
+ kwargs[&quot;model_path&quot;] = model_or_path
+ kwargs[&quot;model&quot;] = &quot;WhisperWarp&quot;
+ else:
+ # model name
+ if model_or_path in name_maps_openai:
+ model_or_path = name_maps_openai[model_or_path]
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/download/download_model_from_hub.py#L9" target="_blank" class="src-more">View full source on GitHub →</a></details></div>
+<div class="api-detail" id="e450"><span class="dtag badge-f">function</span><h2><code>download_from_ms(**kwargs)</code></h2><div class="dmod">funasr.download.download_model_from_hub · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/download/download_model_from_hub.py#L44" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Download from ms.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>**kwargs</code> — Additional keyword arguments.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code>def download_from_ms(**kwargs):
+ &quot;&quot;&quot;Download from ms.
+
+ Args:
+ **kwargs: Additional keyword arguments.
+ &quot;&quot;&quot;
+ model_or_path = kwargs.get(&quot;model&quot;)
+ if model_or_path in name_maps_ms:
+ model_or_path = name_maps_ms[model_or_path]
+ model_revision = kwargs.get(&quot;model_revision&quot;, &quot;master&quot;)
+ if not os.path.exists(model_or_path) and &quot;model_path&quot; not in kwargs:
+ try:
+ model_or_path = get_or_download_model_dir(
+ model_or_path,
+ model_revision,
+ is_training=kwargs.get(&quot;is_training&quot;),
+ check_latest=kwargs.get(&quot;check_latest&quot;, True),
+ )
+ except Exception as e:
+ print(f&quot;Download: {model_or_path} failed!: {e}&quot;)
+
+ kwargs[&quot;model_path&quot;] = model_or_path if &quot;model_path&quot; not in kwargs else kwargs[&quot;model_path&quot;]
+ model_or_path = kwargs[&quot;model_path&quot;]
+ if os.path.exists(os.path.join(model_or_path, &quot;configuration.json&quot;)):
+ with open(os.path.join(model_or_path, &quot;configuration.json&quot;), &quot;r&quot;, encoding=&quot;utf-8&quot;) as f:
+ conf_json = json.load(f)
+
+ cfg = {}
+ if &quot;file_path_metas&quot; in conf_json:
+ add_file_root_path(model_or_path, conf_json[&quot;file_path_metas&quot;], cfg)
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/download/download_model_from_hub.py#L44" target="_blank" class="src-more">View full source on GitHub →</a></details></div>
+<div class="api-detail" id="e451"><span class="dtag badge-f">function</span><h2><code>download_from_hf(**kwargs)</code></h2><div class="dmod">funasr.download.download_model_from_hub · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/download/download_model_from_hub.py#L122" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Download from hf.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>**kwargs</code> — Additional keyword arguments.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code>def download_from_hf(**kwargs):
+ &quot;&quot;&quot;Download from hf.
+
+ Args:
+ **kwargs: Additional keyword arguments.
+ &quot;&quot;&quot;
+ model_or_path = kwargs.get(&quot;model&quot;)
+ if model_or_path in name_maps_hf:
+ model_or_path = name_maps_hf[model_or_path]
+ model_revision = kwargs.get(&quot;model_revision&quot;, &quot;master&quot;)
+ if not os.path.exists(model_or_path) and &quot;model_path&quot; not in kwargs:
+ try:
+ model_or_path = get_or_download_model_dir_hf(
+ model_or_path,
+ model_revision,
+ is_training=kwargs.get(&quot;is_training&quot;),
+ check_latest=kwargs.get(&quot;check_latest&quot;, True),
+ )
+ except Exception as e:
+ print(f&quot;Download: {model_or_path} failed!: {e}&quot;)
+
+ kwargs[&quot;model_path&quot;] = model_or_path if &quot;model_path&quot; not in kwargs else kwargs[&quot;model_path&quot;]
+
+ if os.path.exists(os.path.join(model_or_path, &quot;configuration.json&quot;)):
+ with open(os.path.join(model_or_path, &quot;configuration.json&quot;), &quot;r&quot;, encoding=&quot;utf-8&quot;) as f:
+ conf_json = json.load(f)
+
+ cfg = {}
+ if &quot;file_path_metas&quot; in conf_json:
+ add_file_root_path(model_or_path, conf_json[&quot;file_path_metas&quot;], cfg)
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/download/download_model_from_hub.py#L122" target="_blank" class="src-more">View full source on GitHub →</a></details></div>
+<div class="api-detail" id="e452"><span class="dtag badge-f">function</span><h2><code>add_file_root_path(model_or_path, file_path_metas, cfg)</code></h2><div class="dmod">funasr.download.download_model_from_hub · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/download/download_model_from_hub.py#L191" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Add file root path.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>model_or_path</code> — TODO.</li>
+<li><code>file_path_metas</code> — TODO.</li>
+<li><code>cfg</code> — Configuration overrides.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code>def add_file_root_path(model_or_path: str, file_path_metas: dict, cfg={}):
+
+ &quot;&quot;&quot;Add file root path.
+
+ Args:
+ model_or_path: TODO.
+ file_path_metas: TODO.
+ cfg: Configuration overrides.
+ &quot;&quot;&quot;
+ if isinstance(file_path_metas, dict):
+ if isinstance(cfg, list):
+ cfg.append({})
+
+ for k, v in file_path_metas.items():
+ if isinstance(v, str):
+ p = os.path.join(model_or_path, v)
+ if os.path.exists(p):
+ if isinstance(cfg, dict):
+ cfg[k] = p
+ elif isinstance(cfg, list):
+ # if len(cfg) == 0:
+ # cfg.append({})
+ cfg[-1][k] = p
+
+ elif isinstance(v, dict):
+ if isinstance(cfg, dict):
+ if k not in cfg:
+ cfg[k] = {}
+ add_file_root_path(model_or_path, v, cfg[k])
+ # elif isinstance(cfg, list):
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/download/download_model_from_hub.py#L191" target="_blank" class="src-more">View full source on GitHub →</a></details></div>
+<div class="api-detail" id="e453"><span class="dtag badge-f">function</span><h2><code>get_or_download_model_dir(model, model_revision, is_training, check_latest)</code></h2><div class="dmod">funasr.download.download_model_from_hub · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/download/download_model_from_hub.py#L241" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Get local model directory or download model if necessary.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>model (str)</code> — model id or path to local model directory.</li>
+<li><code>model_revision (str, optional)</code> — model version number.</li>
+</ul>
+<p>:param is_training:</p>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code>def get_or_download_model_dir(
+ model,
+ model_revision=None,
+ is_training=False,
+ check_latest=True,
+):
+ &quot;&quot;&quot;Get local model directory or download model if necessary.
+
+ Args:
+ model (str): model id or path to local model directory.
+ model_revision (str, optional): model version number.
+ :param is_training:
+ &quot;&quot;&quot;
+ from modelscope.hub.check_model import check_local_model_is_latest
+ from modelscope.hub.snapshot_download import snapshot_download
+
+ from modelscope.utils.constant import Invoke, ThirdParty
+
+ key = Invoke.LOCAL_TRAINER if is_training else Invoke.PIPELINE
+
+ if os.path.exists(model) and check_latest:
+ model_cache_dir = model if os.path.isdir(model) else os.path.dirname(model)
+ try:
+ check_local_model_is_latest(
+ model_cache_dir, user_agent={Invoke.KEY: key, ThirdParty.KEY: &quot;funasr&quot;}
+ )
+ except:
+ print(&quot;could not check the latest version&quot;)
+ else:
+ model_cache_dir = snapshot_download(
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/download/download_model_from_hub.py#L241" target="_blank" class="src-more">View full source on GitHub →</a></details></div>
+<div class="api-detail" id="e454"><span class="dtag badge-f">function</span><h2><code>get_or_download_model_dir_hf(model, model_revision, is_training, check_latest)</code></h2><div class="dmod">funasr.download.download_model_from_hub · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/download/download_model_from_hub.py#L276" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Get local model directory or download model if necessary.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>model (str)</code> — model id or path to local model directory.</li>
+<li><code>model_revision (str, optional)</code> — model version number.</li>
+</ul>
+<p>:param is_training:</p>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code>def get_or_download_model_dir_hf(
+ model,
+ model_revision=None,
+ is_training=False,
+ check_latest=True,
+):
+ &quot;&quot;&quot;Get local model directory or download model if necessary.
+
+ Args:
+ model (str): model id or path to local model directory.
+ model_revision (str, optional): model version number.
+ :param is_training:
+ &quot;&quot;&quot;
+ from huggingface_hub import snapshot_download
+
+ model_cache_dir = snapshot_download(model)
+ return model_cache_dir
+</code></pre></details></div>
+<div class="api-detail" id="e455"><span class="dtag badge-f">function</span><h2><code>download_from_url(url)</code></h2><div class="dmod">funasr.download.file · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/download/file.py#L14" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Download from url.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>url</code> — TODO.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code>def download_from_url(url):
+ &quot;&quot;&quot;Download from url.
+
+ Args:
+ url: TODO.
+ &quot;&quot;&quot;
+ result = urlparse(url)
+ file_path = None
+ if result.scheme is not None and len(result.scheme) &gt; 0:
+ storage = HTTPStorage()
+ # bytes
+ data = storage.read(url)
+ work_dir = tempfile.TemporaryDirectory().name
+ if not os.path.exists(work_dir):
+ os.makedirs(work_dir)
+ file_path = os.path.join(work_dir, os.path.basename(url))
+ with open(file_path, &quot;wb&quot;) as fb:
+ fb.write(data)
+ assert file_path is not None, f&quot;failed to download: {url}&quot;
+ return file_path
+</code></pre></details></div>
+<div class="api-detail" id="e456"><span class="dtag badge-c">class</span><h2>Storage</h2><div class="dmod">funasr.download.file · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/download/file.py#L36" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Abstract class of storage.</p>
+<p>All backends need to implement two apis: ``read()`` and ``read_text()``.</p>
+<p>``read()`` reads the file as a byte stream and ``read_text()`` reads</p>
+<p>the file as texts.</p>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code>class Storage(metaclass=ABCMeta):
+ &quot;&quot;&quot;Abstract class of storage.
+
+ All backends need to implement two apis: ``read()`` and ``read_text()``.
+ ``read()`` reads the file as a byte stream and ``read_text()`` reads
+ the file as texts.
+ &quot;&quot;&quot;
+
+ @abstractmethod
+ def read(self, filepath: str):
+ &quot;&quot;&quot;Read.
+
+ Args:
+ filepath: TODO.
+ &quot;&quot;&quot;
+ pass
+
+ @abstractmethod
+ def read_text(self, filepath: str):
+ &quot;&quot;&quot;Read text.
+
+ Args:
+ filepath: TODO.
+ &quot;&quot;&quot;
+ pass
+
+ @abstractmethod
+ def write(self, obj: bytes, filepath: Union[str, Path]) -&gt; None:
+ &quot;&quot;&quot;Write.
+
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/download/file.py#L36" target="_blank" class="src-more">View full source on GitHub →</a></details><h3 class="mt">Methods</h3><div class="mblk"><code>.read(filepath)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/download/file.py#L45" target="_blank" class="gh-link">L45</a><div class="mdoc"><p>Read.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>filepath</code> — TODO.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def read(self, filepath: str):
+ &quot;&quot;&quot;Read.
+
+ Args:
+ filepath: TODO.
+ &quot;&quot;&quot;
+ pass
+</code></pre></details></div><div class="mblk"><code>.read_text(filepath)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/download/file.py#L54" target="_blank" class="gh-link">L54</a><div class="mdoc"><p>Read text.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>filepath</code> — TODO.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def read_text(self, filepath: str):
+ &quot;&quot;&quot;Read text.
+
+ Args:
+ filepath: TODO.
+ &quot;&quot;&quot;
+ pass
+</code></pre></details></div><div class="mblk"><code>.write(obj, filepath)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/download/file.py#L63" target="_blank" class="gh-link">L63</a><div class="mdoc"><p>Write.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>obj</code> — TODO.</li>
+<li><code>filepath</code> — TODO.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def write(self, obj: bytes, filepath: Union[str, Path]) -&gt; None:
+ &quot;&quot;&quot;Write.
+
+ Args:
+ obj: TODO.
+ filepath: TODO.
+ &quot;&quot;&quot;
+ pass
+</code></pre></details></div><div class="mblk"><code>.write_text(obj, filepath, encoding)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/download/file.py#L73" target="_blank" class="gh-link">L73</a><div class="mdoc"><p>Write text.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>obj</code> — TODO.</li>
+<li><code>filepath</code> — TODO.</li>
+<li><code>encoding</code> — TODO.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def write_text(self, obj: str, filepath: Union[str, Path], encoding: str = &quot;utf-8&quot;) -&gt; None:
+ &quot;&quot;&quot;Write text.
+
+ Args:
+ obj: TODO.
+ filepath: TODO.
+ encoding: TODO.
+ &quot;&quot;&quot;
+ pass
+</code></pre></details></div></div>
+<div class="api-detail" id="e457"><span class="dtag badge-m">method</span><h2><code>Storage.read(filepath)</code></h2><div class="dmod">funasr.download.file.Storage · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/download/file.py#L45" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Read.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>filepath</code> — TODO.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def read(self, filepath: str):
+ &quot;&quot;&quot;Read.
+
+ Args:
+ filepath: TODO.
+ &quot;&quot;&quot;
+ pass
+</code></pre></details></div>
+<div class="api-detail" id="e458"><span class="dtag badge-m">method</span><h2><code>Storage.read_text(filepath)</code></h2><div class="dmod">funasr.download.file.Storage · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/download/file.py#L54" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Read text.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>filepath</code> — TODO.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def read_text(self, filepath: str):
+ &quot;&quot;&quot;Read text.
+
+ Args:
+ filepath: TODO.
+ &quot;&quot;&quot;
+ pass
+</code></pre></details></div>
+<div class="api-detail" id="e459"><span class="dtag badge-m">method</span><h2><code>Storage.write(obj, filepath)</code></h2><div class="dmod">funasr.download.file.Storage · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/download/file.py#L63" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Write.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>obj</code> — TODO.</li>
+<li><code>filepath</code> — TODO.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def write(self, obj: bytes, filepath: Union[str, Path]) -&gt; None:
+ &quot;&quot;&quot;Write.
+
+ Args:
+ obj: TODO.
+ filepath: TODO.
+ &quot;&quot;&quot;
+ pass
+</code></pre></details></div>
+<div class="api-detail" id="e460"><span class="dtag badge-m">method</span><h2><code>Storage.write_text(obj, filepath, encoding)</code></h2><div class="dmod">funasr.download.file.Storage · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/download/file.py#L73" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Write text.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>obj</code> — TODO.</li>
+<li><code>filepath</code> — TODO.</li>
+<li><code>encoding</code> — TODO.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def write_text(self, obj: str, filepath: Union[str, Path], encoding: str = &quot;utf-8&quot;) -&gt; None:
+ &quot;&quot;&quot;Write text.
+
+ Args:
+ obj: TODO.
+ filepath: TODO.
+ encoding: TODO.
+ &quot;&quot;&quot;
+ pass
+</code></pre></details></div>
+<div class="api-detail" id="e461"><span class="dtag badge-c">class</span><h2>LocalStorage</h2><div class="dmod">funasr.download.file · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/download/file.py#L84" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Local hard disk storage</p>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code>class LocalStorage(Storage):
+ &quot;&quot;&quot;Local hard disk storage&quot;&quot;&quot;
+
+ def read(self, filepath: Union[str, Path]) -&gt; bytes:
+ &quot;&quot;&quot;Read data from a given ``filepath`` with &#x27;rb&#x27; mode.
+
+ Args:
+ filepath (str or Path): Path to read data.
+
+ Returns:
+ bytes: Expected bytes object.
+ &quot;&quot;&quot;
+ with open(filepath, &quot;rb&quot;) as f:
+ content = f.read()
+ return content
+
+ def read_text(self, filepath: Union[str, Path], encoding: str = &quot;utf-8&quot;) -&gt; str:
+ &quot;&quot;&quot;Read data from a given ``filepath`` with &#x27;r&#x27; mode.
+
+ Args:
+ filepath (str or Path): Path to read data.
+ encoding (str): The encoding format used to open the ``filepath``.
+ Default: &#x27;utf-8&#x27;.
+
+ Returns:
+ str: Expected text reading from ``filepath``.
+ &quot;&quot;&quot;
+ with open(filepath, &quot;r&quot;, encoding=encoding) as f:
+ value_buf = f.read()
+ return value_buf
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/download/file.py#L84" target="_blank" class="src-more">View full source on GitHub →</a></details><h3 class="mt">Methods</h3><div class="mblk"><code>.read(filepath)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/download/file.py#L87" target="_blank" class="gh-link">L87</a><div class="mdoc"><p>Read data from a given ``filepath`` with &#x27;rb&#x27; mode.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>filepath (str or Path)</code> — Path to read data.</li>
+</ul>
+<h4>Returns:</h4>
+<ul>
+<li><code>bytes</code> — Expected bytes object.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def read(self, filepath: Union[str, Path]) -&gt; bytes:
+ &quot;&quot;&quot;Read data from a given ``filepath`` with &#x27;rb&#x27; mode.
+
+ Args:
+ filepath (str or Path): Path to read data.
+
+ Returns:
+ bytes: Expected bytes object.
+ &quot;&quot;&quot;
+ with open(filepath, &quot;rb&quot;) as f:
+ content = f.read()
+ return content
+</code></pre></details></div><div class="mblk"><code>.read_text(filepath, encoding)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/download/file.py#L100" target="_blank" class="gh-link">L100</a><div class="mdoc"><p>Read data from a given ``filepath`` with &#x27;r&#x27; mode.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>filepath (str or Path)</code> — Path to read data.</li>
+<li><code>encoding (str)</code> — The encoding format used to open the ``filepath``.</li>
+<li><code>Default</code> — &#x27;utf-8&#x27;.</li>
+</ul>
+<h4>Returns:</h4>
+<ul>
+<li><code>str</code> — Expected text reading from ``filepath``.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def read_text(self, filepath: Union[str, Path], encoding: str = &quot;utf-8&quot;) -&gt; str:
+ &quot;&quot;&quot;Read data from a given ``filepath`` with &#x27;r&#x27; mode.
+
+ Args:
+ filepath (str or Path): Path to read data.
+ encoding (str): The encoding format used to open the ``filepath``.
+ Default: &#x27;utf-8&#x27;.
+
+ Returns:
+ str: Expected text reading from ``filepath``.
+ &quot;&quot;&quot;
+ with open(filepath, &quot;r&quot;, encoding=encoding) as f:
+ value_buf = f.read()
+ return value_buf
+</code></pre></details></div><div class="mblk"><code>.write(obj, filepath)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/download/file.py#L115" target="_blank" class="gh-link">L115</a><div class="mdoc"><p>Write data to a given ``filepath`` with &#x27;wb&#x27; mode.</p>
+<h4>Note:</h4>
+<p>``write`` will create a directory if the directory of ``filepath``</p>
+<p>does not exist.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>obj (bytes)</code> — Data to be written.</li>
+<li><code>filepath (str or Path)</code> — Path to write data.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def write(self, obj: bytes, filepath: Union[str, Path]) -&gt; None:
+ &quot;&quot;&quot;Write data to a given ``filepath`` with &#x27;wb&#x27; mode.
+
+ Note:
+ ``write`` will create a directory if the directory of ``filepath``
+ does not exist.
+
+ Args:
+ obj (bytes): Data to be written.
+ filepath (str or Path): Path to write data.
+ &quot;&quot;&quot;
+ dirname = os.path.dirname(filepath)
+ if dirname and not os.path.exists(dirname):
+ os.makedirs(dirname, exist_ok=True)
+
+ with open(filepath, &quot;wb&quot;) as f:
+ f.write(obj)
+</code></pre></details></div><div class="mblk"><code>.write_text(obj, filepath, encoding)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/download/file.py#L133" target="_blank" class="gh-link">L133</a><div class="mdoc"><p>Write data to a given ``filepath`` with &#x27;w&#x27; mode.</p>
+<h4>Note:</h4>
+<p>``write_text`` will create a directory if the directory of</p>
+<p>``filepath`` does not exist.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>obj (str)</code> — Data to be written.</li>
+<li><code>filepath (str or Path)</code> — Path to write data.</li>
+<li><code>encoding (str)</code> — The encoding format used to open the ``filepath``.</li>
+<li><code>Default</code> — &#x27;utf-8&#x27;.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def write_text(self, obj: str, filepath: Union[str, Path], encoding: str = &quot;utf-8&quot;) -&gt; None:
+ &quot;&quot;&quot;Write data to a given ``filepath`` with &#x27;w&#x27; mode.
+
+ Note:
+ ``write_text`` will create a directory if the directory of
+ ``filepath`` does not exist.
+
+ Args:
+ obj (str): Data to be written.
+ filepath (str or Path): Path to write data.
+ encoding (str): The encoding format used to open the ``filepath``.
+ Default: &#x27;utf-8&#x27;.
+ &quot;&quot;&quot;
+ dirname = os.path.dirname(filepath)
+ if dirname and not os.path.exists(dirname):
+ os.makedirs(dirname, exist_ok=True)
+
+ with open(filepath, &quot;w&quot;, encoding=encoding) as f:
+ f.write(obj)
+</code></pre></details></div><div class="mblk"><code>.as_local_path(filepath)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/download/file.py#L154" target="_blank" class="gh-link">L154</a><div class="mdoc"><p>Only for unified API and do nothing.</p>
+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def as_local_path(self, filepath: Union[str, Path]) -&gt; Generator[Union[str, Path], None, None]:
+ &quot;&quot;&quot;Only for unified API and do nothing.&quot;&quot;&quot;
+ yield filepath
+</code></pre></details></div></div>
+<div class="api-detail" id="e462"><span class="dtag badge-m">method</span><h2><code>LocalStorage.read(filepath)</code></h2><div class="dmod">funasr.download.file.LocalStorage · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/download/file.py#L87" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Read data from a given ``filepath`` with &#x27;rb&#x27; mode.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>filepath (str or Path)</code> — Path to read data.</li>
+</ul>
+<h4>Returns:</h4>
+<ul>
+<li><code>bytes</code> — Expected bytes object.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def read(self, filepath: Union[str, Path]) -&gt; bytes:
+ &quot;&quot;&quot;Read data from a given ``filepath`` with &#x27;rb&#x27; mode.
+
+ Args:
+ filepath (str or Path): Path to read data.
+
+ Returns:
+ bytes: Expected bytes object.
+ &quot;&quot;&quot;
+ with open(filepath, &quot;rb&quot;) as f:
+ content = f.read()
+ return content
+</code></pre></details></div>
+<div class="api-detail" id="e463"><span class="dtag badge-m">method</span><h2><code>LocalStorage.read_text(filepath, encoding)</code></h2><div class="dmod">funasr.download.file.LocalStorage · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/download/file.py#L100" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Read data from a given ``filepath`` with &#x27;r&#x27; mode.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>filepath (str or Path)</code> — Path to read data.</li>
+<li><code>encoding (str)</code> — The encoding format used to open the ``filepath``.</li>
+<li><code>Default</code> — &#x27;utf-8&#x27;.</li>
+</ul>
+<h4>Returns:</h4>
+<ul>
+<li><code>str</code> — Expected text reading from ``filepath``.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def read_text(self, filepath: Union[str, Path], encoding: str = &quot;utf-8&quot;) -&gt; str:
+ &quot;&quot;&quot;Read data from a given ``filepath`` with &#x27;r&#x27; mode.
+
+ Args:
+ filepath (str or Path): Path to read data.
+ encoding (str): The encoding format used to open the ``filepath``.
+ Default: &#x27;utf-8&#x27;.
+
+ Returns:
+ str: Expected text reading from ``filepath``.
+ &quot;&quot;&quot;
+ with open(filepath, &quot;r&quot;, encoding=encoding) as f:
+ value_buf = f.read()
+ return value_buf
+</code></pre></details></div>
+<div class="api-detail" id="e464"><span class="dtag badge-m">method</span><h2><code>LocalStorage.write(obj, filepath)</code></h2><div class="dmod">funasr.download.file.LocalStorage · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/download/file.py#L115" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Write data to a given ``filepath`` with &#x27;wb&#x27; mode.</p>
+<h4>Note:</h4>
+<p>``write`` will create a directory if the directory of ``filepath``</p>
+<p>does not exist.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>obj (bytes)</code> — Data to be written.</li>
+<li><code>filepath (str or Path)</code> — Path to write data.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def write(self, obj: bytes, filepath: Union[str, Path]) -&gt; None:
+ &quot;&quot;&quot;Write data to a given ``filepath`` with &#x27;wb&#x27; mode.
+
+ Note:
+ ``write`` will create a directory if the directory of ``filepath``
+ does not exist.
+
+ Args:
+ obj (bytes): Data to be written.
+ filepath (str or Path): Path to write data.
+ &quot;&quot;&quot;
+ dirname = os.path.dirname(filepath)
+ if dirname and not os.path.exists(dirname):
+ os.makedirs(dirname, exist_ok=True)
+
+ with open(filepath, &quot;wb&quot;) as f:
+ f.write(obj)
+</code></pre></details></div>
+<div class="api-detail" id="e465"><span class="dtag badge-m">method</span><h2><code>LocalStorage.write_text(obj, filepath, encoding)</code></h2><div class="dmod">funasr.download.file.LocalStorage · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/download/file.py#L133" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Write data to a given ``filepath`` with &#x27;w&#x27; mode.</p>
+<h4>Note:</h4>
+<p>``write_text`` will create a directory if the directory of</p>
+<p>``filepath`` does not exist.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>obj (str)</code> — Data to be written.</li>
+<li><code>filepath (str or Path)</code> — Path to write data.</li>
+<li><code>encoding (str)</code> — The encoding format used to open the ``filepath``.</li>
+<li><code>Default</code> — &#x27;utf-8&#x27;.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def write_text(self, obj: str, filepath: Union[str, Path], encoding: str = &quot;utf-8&quot;) -&gt; None:
+ &quot;&quot;&quot;Write data to a given ``filepath`` with &#x27;w&#x27; mode.
+
+ Note:
+ ``write_text`` will create a directory if the directory of
+ ``filepath`` does not exist.
+
+ Args:
+ obj (str): Data to be written.
+ filepath (str or Path): Path to write data.
+ encoding (str): The encoding format used to open the ``filepath``.
+ Default: &#x27;utf-8&#x27;.
+ &quot;&quot;&quot;
+ dirname = os.path.dirname(filepath)
+ if dirname and not os.path.exists(dirname):
+ os.makedirs(dirname, exist_ok=True)
+
+ with open(filepath, &quot;w&quot;, encoding=encoding) as f:
+ f.write(obj)
+</code></pre></details></div>
+<div class="api-detail" id="e466"><span class="dtag badge-m">method</span><h2><code>LocalStorage.as_local_path(filepath)</code></h2><div class="dmod">funasr.download.file.LocalStorage · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/download/file.py#L154" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Only for unified API and do nothing.</p>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def as_local_path(self, filepath: Union[str, Path]) -&gt; Generator[Union[str, Path], None, None]:
+ &quot;&quot;&quot;Only for unified API and do nothing.&quot;&quot;&quot;
+ yield filepath
+</code></pre></details></div>
+<div class="api-detail" id="e467"><span class="dtag badge-c">class</span><h2>HTTPStorage</h2><div class="dmod">funasr.download.file · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/download/file.py#L159" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>HTTP and HTTPS storage.</p>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code>class HTTPStorage(Storage):
+ &quot;&quot;&quot;HTTP and HTTPS storage.&quot;&quot;&quot;
+
+ def read(self, url):
+ # TODO @wenmeng.zwm add progress bar if file is too large
+ &quot;&quot;&quot;Read.
+
+ Args:
+ url: TODO.
+ &quot;&quot;&quot;
+ r = requests.get(url)
+ r.raise_for_status()
+ return r.content
+
+ def read_text(self, url):
+ &quot;&quot;&quot;Read text.
+
+ Args:
+ url: TODO.
+ &quot;&quot;&quot;
+ r = requests.get(url)
+ r.raise_for_status()
+ return r.text
+
+ @contextlib.contextmanager
+ def as_local_path(self, filepath: str) -&gt; Generator[Union[str, Path], None, None]:
+ &quot;&quot;&quot;Download a file from ``filepath``.
+
+ ``as_local_path`` is decorated by :meth:`contextlib.contextmanager`. It
+ can be called with ``with`` statement, and when exists from the
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/download/file.py#L159" target="_blank" class="src-more">View full source on GitHub →</a></details><h3 class="mt">Methods</h3><div class="mblk"><code>.read(url)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/download/file.py#L162" target="_blank" class="gh-link">L162</a><div class="mdoc"><p>Read.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>url</code> — TODO.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def read(self, url):
+ # TODO @wenmeng.zwm add progress bar if file is too large
+ &quot;&quot;&quot;Read.
+
+ Args:
+ url: TODO.
+ &quot;&quot;&quot;
+ r = requests.get(url)
+ r.raise_for_status()
+ return r.content
+</code></pre></details></div><div class="mblk"><code>.read_text(url)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/download/file.py#L173" target="_blank" class="gh-link">L173</a><div class="mdoc"><p>Read text.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>url</code> — TODO.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def read_text(self, url):
+ &quot;&quot;&quot;Read text.
+
+ Args:
+ url: TODO.
+ &quot;&quot;&quot;
+ r = requests.get(url)
+ r.raise_for_status()
+ return r.text
+</code></pre></details></div><div class="mblk"><code>.as_local_path(filepath)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/download/file.py#L184" target="_blank" class="gh-link">L184</a><div class="mdoc"><p>Download a file from ``filepath``.</p>
+<p>``as_local_path`` is decorated by :meth:`contextlib.contextmanager`. It</p>
+<p>can be called with ``with`` statement, and when exists from the</p>
+<p>``with`` statement, the temporary path will be released.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>filepath (str)</code> — Download a file from ``filepath``.</li>
+</ul>
+<h4>Examples:</h4>
+<p>&gt;&gt;&gt; storage = HTTPStorage()</p>
+<p>&gt;&gt;&gt; # After existing from the ``with`` clause,</p>
+<p>&gt;&gt;&gt; # the path will be removed</p>
+<p>&gt;&gt;&gt; with storage.get_local_path(&#x27;http://path/to/file&#x27;) as path:</p>
+<p>... # do something here</p>
+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def as_local_path(self, filepath: str) -&gt; Generator[Union[str, Path], None, None]:
+ &quot;&quot;&quot;Download a file from ``filepath``.
+
+ ``as_local_path`` is decorated by :meth:`contextlib.contextmanager`. It
+ can be called with ``with`` statement, and when exists from the
+ ``with`` statement, the temporary path will be released.
+
+ Args:
+ filepath (str): Download a file from ``filepath``.
+
+ Examples:
+ &gt;&gt;&gt; storage = HTTPStorage()
+ &gt;&gt;&gt; # After existing from the ``with`` clause,
+ &gt;&gt;&gt; # the path will be removed
+ &gt;&gt;&gt; with storage.get_local_path(&#x27;http://path/to/file&#x27;) as path:
+ ... # do something here
+ &quot;&quot;&quot;
+ try:
+ f = tempfile.NamedTemporaryFile(delete=False)
+ f.write(self.read(filepath))
+ f.close()
+ yield f.name
+ finally:
+ os.remove(f.name)
+</code></pre></details></div><div class="mblk"><code>.write(obj, url)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/download/file.py#L209" target="_blank" class="gh-link">L209</a><div class="mdoc"><p>Write.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>obj</code> — TODO.</li>
+<li><code>url</code> — TODO.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def write(self, obj: bytes, url: Union[str, Path]) -&gt; None:
+ &quot;&quot;&quot;Write.
+
+ Args:
+ obj: TODO.
+ url: TODO.
+ &quot;&quot;&quot;
+ raise NotImplementedError(&quot;write is not supported by HTTP Storage&quot;)
+</code></pre></details></div><div class="mblk"><code>.write_text(obj, url, encoding)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/download/file.py#L218" target="_blank" class="gh-link">L218</a><div class="mdoc"><p>Write text.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>obj</code> — TODO.</li>
+<li><code>url</code> — TODO.</li>
+<li><code>encoding</code> — TODO.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def write_text(self, obj: str, url: Union[str, Path], encoding: str = &quot;utf-8&quot;) -&gt; None:
+ &quot;&quot;&quot;Write text.
+
+ Args:
+ obj: TODO.
+ url: TODO.
+ encoding: TODO.
+ &quot;&quot;&quot;
+ raise NotImplementedError(&quot;write_text is not supported by HTTP Storage&quot;)
+</code></pre></details></div></div>
+<div class="api-detail" id="e468"><span class="dtag badge-m">method</span><h2><code>HTTPStorage.read(url)</code></h2><div class="dmod">funasr.download.file.HTTPStorage · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/download/file.py#L162" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Read.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>url</code> — TODO.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def read(self, url):
+ # TODO @wenmeng.zwm add progress bar if file is too large
+ &quot;&quot;&quot;Read.
+
+ Args:
+ url: TODO.
+ &quot;&quot;&quot;
+ r = requests.get(url)
+ r.raise_for_status()
+ return r.content
+</code></pre></details></div>
+<div class="api-detail" id="e469"><span class="dtag badge-m">method</span><h2><code>HTTPStorage.read_text(url)</code></h2><div class="dmod">funasr.download.file.HTTPStorage · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/download/file.py#L173" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Read text.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>url</code> — TODO.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def read_text(self, url):
+ &quot;&quot;&quot;Read text.
+
+ Args:
+ url: TODO.
+ &quot;&quot;&quot;
+ r = requests.get(url)
+ r.raise_for_status()
+ return r.text
+</code></pre></details></div>
+<div class="api-detail" id="e470"><span class="dtag badge-m">method</span><h2><code>HTTPStorage.as_local_path(filepath)</code></h2><div class="dmod">funasr.download.file.HTTPStorage · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/download/file.py#L184" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Download a file from ``filepath``.</p>
+<p>``as_local_path`` is decorated by :meth:`contextlib.contextmanager`. It</p>
+<p>can be called with ``with`` statement, and when exists from the</p>
+<p>``with`` statement, the temporary path will be released.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>filepath (str)</code> — Download a file from ``filepath``.</li>
+</ul>
+<h4>Examples:</h4>
+<p>&gt;&gt;&gt; storage = HTTPStorage()</p>
+<p>&gt;&gt;&gt; # After existing from the ``with`` clause,</p>
+<p>&gt;&gt;&gt; # the path will be removed</p>
+<p>&gt;&gt;&gt; with storage.get_local_path(&#x27;http://path/to/file&#x27;) as path:</p>
+<p>... # do something here</p>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def as_local_path(self, filepath: str) -&gt; Generator[Union[str, Path], None, None]:
+ &quot;&quot;&quot;Download a file from ``filepath``.
+
+ ``as_local_path`` is decorated by :meth:`contextlib.contextmanager`. It
+ can be called with ``with`` statement, and when exists from the
+ ``with`` statement, the temporary path will be released.
+
+ Args:
+ filepath (str): Download a file from ``filepath``.
+
+ Examples:
+ &gt;&gt;&gt; storage = HTTPStorage()
+ &gt;&gt;&gt; # After existing from the ``with`` clause,
+ &gt;&gt;&gt; # the path will be removed
+ &gt;&gt;&gt; with storage.get_local_path(&#x27;http://path/to/file&#x27;) as path:
+ ... # do something here
+ &quot;&quot;&quot;
+ try:
+ f = tempfile.NamedTemporaryFile(delete=False)
+ f.write(self.read(filepath))
+ f.close()
+ yield f.name
+ finally:
+ os.remove(f.name)
+</code></pre></details></div>
+<div class="api-detail" id="e471"><span class="dtag badge-m">method</span><h2><code>HTTPStorage.write(obj, url)</code></h2><div class="dmod">funasr.download.file.HTTPStorage · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/download/file.py#L209" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Write.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>obj</code> — TODO.</li>
+<li><code>url</code> — TODO.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def write(self, obj: bytes, url: Union[str, Path]) -&gt; None:
+ &quot;&quot;&quot;Write.
+
+ Args:
+ obj: TODO.
+ url: TODO.
+ &quot;&quot;&quot;
+ raise NotImplementedError(&quot;write is not supported by HTTP Storage&quot;)
+</code></pre></details></div>
+<div class="api-detail" id="e472"><span class="dtag badge-m">method</span><h2><code>HTTPStorage.write_text(obj, url, encoding)</code></h2><div class="dmod">funasr.download.file.HTTPStorage · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/download/file.py#L218" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Write text.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>obj</code> — TODO.</li>
+<li><code>url</code> — TODO.</li>
+<li><code>encoding</code> — TODO.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def write_text(self, obj: str, url: Union[str, Path], encoding: str = &quot;utf-8&quot;) -&gt; None:
+ &quot;&quot;&quot;Write text.
+
+ Args:
+ obj: TODO.
+ url: TODO.
+ encoding: TODO.
+ &quot;&quot;&quot;
+ raise NotImplementedError(&quot;write_text is not supported by HTTP Storage&quot;)
+</code></pre></details></div>
+<div class="api-detail" id="e473"><span class="dtag badge-c">class</span><h2>OSSStorage</h2><div class="dmod">funasr.download.file · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/download/file.py#L229" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>OSS storage.</p>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code>class OSSStorage(Storage):
+ &quot;&quot;&quot;OSS storage.&quot;&quot;&quot;
+
+ def __init__(self, oss_config_file=None):
+ # read from config file or env var
+ &quot;&quot;&quot;Initialize OSSStorage.
+
+ Args:
+ oss_config_file: TODO.
+ &quot;&quot;&quot;
+ raise NotImplementedError(&quot;OSSStorage.__init__ to be implemented in the future&quot;)
+
+ def read(self, filepath):
+ &quot;&quot;&quot;Read.
+
+ Args:
+ filepath: TODO.
+ &quot;&quot;&quot;
+ raise NotImplementedError(&quot;OSSStorage.read to be implemented in the future&quot;)
+
+ def read_text(self, filepath, encoding=&quot;utf-8&quot;):
+ &quot;&quot;&quot;Read text.
+
+ Args:
+ filepath: TODO.
+ encoding: TODO.
+ &quot;&quot;&quot;
+ raise NotImplementedError(&quot;OSSStorage.read_text to be implemented in the future&quot;)
+
+ @contextlib.contextmanager
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/download/file.py#L229" target="_blank" class="src-more">View full source on GitHub →</a></details><h3 class="mt">Methods</h3><div class="mblk"><code>.read(filepath)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/download/file.py#L241" target="_blank" class="gh-link">L241</a><div class="mdoc"><p>Read.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>filepath</code> — TODO.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def read(self, filepath):
+ &quot;&quot;&quot;Read.
+
+ Args:
+ filepath: TODO.
+ &quot;&quot;&quot;
+ raise NotImplementedError(&quot;OSSStorage.read to be implemented in the future&quot;)
+</code></pre></details></div><div class="mblk"><code>.read_text(filepath, encoding)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/download/file.py#L249" target="_blank" class="gh-link">L249</a><div class="mdoc"><p>Read text.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>filepath</code> — TODO.</li>
+<li><code>encoding</code> — TODO.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def read_text(self, filepath, encoding=&quot;utf-8&quot;):
+ &quot;&quot;&quot;Read text.
+
+ Args:
+ filepath: TODO.
+ encoding: TODO.
+ &quot;&quot;&quot;
+ raise NotImplementedError(&quot;OSSStorage.read_text to be implemented in the future&quot;)
+</code></pre></details></div><div class="mblk"><code>.as_local_path(filepath)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/download/file.py#L259" target="_blank" class="gh-link">L259</a><div class="mdoc"><p>Download a file from ``filepath``.</p>
+<p>``as_local_path`` is decorated by :meth:`contextlib.contextmanager`. It</p>
+<p>can be called with ``with`` statement, and when exists from the</p>
+<p>``with`` statement, the temporary path will be released.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>filepath (str)</code> — Download a file from ``filepath``.</li>
+</ul>
+<h4>Examples:</h4>
+<p>&gt;&gt;&gt; storage = OSSStorage()</p>
+<p>&gt;&gt;&gt; # After existing from the ``with`` clause,</p>
+<p>&gt;&gt;&gt; # the path will be removed</p>
+<p>&gt;&gt;&gt; with storage.get_local_path(&#x27;http://path/to/file&#x27;) as path:</p>
+<p>... # do something here</p>
+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def as_local_path(self, filepath: str) -&gt; Generator[Union[str, Path], None, None]:
+ &quot;&quot;&quot;Download a file from ``filepath``.
+
+ ``as_local_path`` is decorated by :meth:`contextlib.contextmanager`. It
+ can be called with ``with`` statement, and when exists from the
+ ``with`` statement, the temporary path will be released.
+
+ Args:
+ filepath (str): Download a file from ``filepath``.
+
+ Examples:
+ &gt;&gt;&gt; storage = OSSStorage()
+ &gt;&gt;&gt; # After existing from the ``with`` clause,
+ &gt;&gt;&gt; # the path will be removed
+ &gt;&gt;&gt; with storage.get_local_path(&#x27;http://path/to/file&#x27;) as path:
+ ... # do something here
+ &quot;&quot;&quot;
+ try:
+ f = tempfile.NamedTemporaryFile(delete=False)
+ f.write(self.read(filepath))
+ f.close()
+ yield f.name
+ finally:
+ os.remove(f.name)
+</code></pre></details></div><div class="mblk"><code>.write(obj, filepath)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/download/file.py#L284" target="_blank" class="gh-link">L284</a><div class="mdoc"><p>Write.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>obj</code> — TODO.</li>
+<li><code>filepath</code> — TODO.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def write(self, obj: bytes, filepath: Union[str, Path]) -&gt; None:
+ &quot;&quot;&quot;Write.
+
+ Args:
+ obj: TODO.
+ filepath: TODO.
+ &quot;&quot;&quot;
+ raise NotImplementedError(&quot;OSSStorage.write to be implemented in the future&quot;)
+</code></pre></details></div><div class="mblk"><code>.write_text(obj, filepath, encoding)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/download/file.py#L293" target="_blank" class="gh-link">L293</a><div class="mdoc"><p>Write text.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>obj</code> — TODO.</li>
+<li><code>filepath</code> — TODO.</li>
+<li><code>encoding</code> — TODO.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def write_text(self, obj: str, filepath: Union[str, Path], encoding: str = &quot;utf-8&quot;) -&gt; None:
+ &quot;&quot;&quot;Write text.
+
+ Args:
+ obj: TODO.
+ filepath: TODO.
+ encoding: TODO.
+ &quot;&quot;&quot;
+ raise NotImplementedError(&quot;OSSStorage.write_text to be implemented in the future&quot;)
+</code></pre></details></div></div>
+<div class="api-detail" id="e474"><span class="dtag badge-m">method</span><h2><code>OSSStorage.read(filepath)</code></h2><div class="dmod">funasr.download.file.OSSStorage · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/download/file.py#L241" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Read.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>filepath</code> — TODO.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def read(self, filepath):
+ &quot;&quot;&quot;Read.
+
+ Args:
+ filepath: TODO.
+ &quot;&quot;&quot;
+ raise NotImplementedError(&quot;OSSStorage.read to be implemented in the future&quot;)
+</code></pre></details></div>
+<div class="api-detail" id="e475"><span class="dtag badge-m">method</span><h2><code>OSSStorage.read_text(filepath, encoding)</code></h2><div class="dmod">funasr.download.file.OSSStorage · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/download/file.py#L249" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Read text.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>filepath</code> — TODO.</li>
+<li><code>encoding</code> — TODO.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def read_text(self, filepath, encoding=&quot;utf-8&quot;):
+ &quot;&quot;&quot;Read text.
+
+ Args:
+ filepath: TODO.
+ encoding: TODO.
+ &quot;&quot;&quot;
+ raise NotImplementedError(&quot;OSSStorage.read_text to be implemented in the future&quot;)
+</code></pre></details></div>
+<div class="api-detail" id="e476"><span class="dtag badge-m">method</span><h2><code>OSSStorage.as_local_path(filepath)</code></h2><div class="dmod">funasr.download.file.OSSStorage · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/download/file.py#L259" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Download a file from ``filepath``.</p>
+<p>``as_local_path`` is decorated by :meth:`contextlib.contextmanager`. It</p>
+<p>can be called with ``with`` statement, and when exists from the</p>
+<p>``with`` statement, the temporary path will be released.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>filepath (str)</code> — Download a file from ``filepath``.</li>
+</ul>
+<h4>Examples:</h4>
+<p>&gt;&gt;&gt; storage = OSSStorage()</p>
+<p>&gt;&gt;&gt; # After existing from the ``with`` clause,</p>
+<p>&gt;&gt;&gt; # the path will be removed</p>
+<p>&gt;&gt;&gt; with storage.get_local_path(&#x27;http://path/to/file&#x27;) as path:</p>
+<p>... # do something here</p>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def as_local_path(self, filepath: str) -&gt; Generator[Union[str, Path], None, None]:
+ &quot;&quot;&quot;Download a file from ``filepath``.
+
+ ``as_local_path`` is decorated by :meth:`contextlib.contextmanager`. It
+ can be called with ``with`` statement, and when exists from the
+ ``with`` statement, the temporary path will be released.
+
+ Args:
+ filepath (str): Download a file from ``filepath``.
+
+ Examples:
+ &gt;&gt;&gt; storage = OSSStorage()
+ &gt;&gt;&gt; # After existing from the ``with`` clause,
+ &gt;&gt;&gt; # the path will be removed
+ &gt;&gt;&gt; with storage.get_local_path(&#x27;http://path/to/file&#x27;) as path:
+ ... # do something here
+ &quot;&quot;&quot;
+ try:
+ f = tempfile.NamedTemporaryFile(delete=False)
+ f.write(self.read(filepath))
+ f.close()
+ yield f.name
+ finally:
+ os.remove(f.name)
+</code></pre></details></div>
+<div class="api-detail" id="e477"><span class="dtag badge-m">method</span><h2><code>OSSStorage.write(obj, filepath)</code></h2><div class="dmod">funasr.download.file.OSSStorage · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/download/file.py#L284" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Write.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>obj</code> — TODO.</li>
+<li><code>filepath</code> — TODO.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def write(self, obj: bytes, filepath: Union[str, Path]) -&gt; None:
+ &quot;&quot;&quot;Write.
+
+ Args:
+ obj: TODO.
+ filepath: TODO.
+ &quot;&quot;&quot;
+ raise NotImplementedError(&quot;OSSStorage.write to be implemented in the future&quot;)
+</code></pre></details></div>
+<div class="api-detail" id="e478"><span class="dtag badge-m">method</span><h2><code>OSSStorage.write_text(obj, filepath, encoding)</code></h2><div class="dmod">funasr.download.file.OSSStorage · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/download/file.py#L293" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Write text.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>obj</code> — TODO.</li>
+<li><code>filepath</code> — TODO.</li>
+<li><code>encoding</code> — TODO.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def write_text(self, obj: str, filepath: Union[str, Path], encoding: str = &quot;utf-8&quot;) -&gt; None:
+ &quot;&quot;&quot;Write text.
+
+ Args:
+ obj: TODO.
+ filepath: TODO.
+ encoding: TODO.
+ &quot;&quot;&quot;
+ raise NotImplementedError(&quot;OSSStorage.write_text to be implemented in the future&quot;)
+</code></pre></details></div>
+<div class="api-detail" id="e479"><span class="dtag badge-c">class</span><h2>File</h2><div class="dmod">funasr.download.file · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/download/file.py#L307" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p class="muted">No documentation yet.</p></div><details class="src-block"><summary>📄 Source code</summary><pre><code>class File(object):
+ _prefix_to_storage: dict = {
+ &quot;oss&quot;: OSSStorage,
+ &quot;http&quot;: HTTPStorage,
+ &quot;https&quot;: HTTPStorage,
+ &quot;local&quot;: LocalStorage,
+ }
+
+ @staticmethod
+ def _get_storage(uri):
+ &quot;&quot;&quot;Internal: get storage.
+
+ Args:
+ uri: TODO.
+ &quot;&quot;&quot;
+ assert isinstance(uri, str), f&quot;uri should be str type, but got {type(uri)}&quot;
+
+ if &quot;://&quot; not in uri:
+ # local path
+ storage_type = &quot;local&quot;
+ else:
+ prefix, _ = uri.split(&quot;://&quot;)
+ storage_type = prefix
+
+ assert storage_type in File._prefix_to_storage, (
+ f&quot;Unsupported uri {uri}, valid prefixs: &quot; f&quot;{list(File._prefix_to_storage.keys())}&quot;
+ )
+
+ if storage_type not in G_STORAGES:
+ G_STORAGES[storage_type] = File._prefix_to_storage[storage_type]()
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/download/file.py#L307" target="_blank" class="src-more">View full source on GitHub →</a></details><h3 class="mt">Methods</h3><div class="mblk"><code>.read(uri)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/download/file.py#L341" target="_blank" class="gh-link">L341</a><div class="mdoc"><p>Read data from a given ``filepath`` with &#x27;rb&#x27; mode.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>filepath (str or Path)</code> — Path to read data.</li>
+</ul>
+<h4>Returns:</h4>
+<ul>
+<li><code>bytes</code> — Expected bytes object.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def read(uri: str) -&gt; bytes:
+ &quot;&quot;&quot;Read data from a given ``filepath`` with &#x27;rb&#x27; mode.
+
+ Args:
+ filepath (str or Path): Path to read data.
+
+ Returns:
+ bytes: Expected bytes object.
+ &quot;&quot;&quot;
+ storage = File._get_storage(uri)
+ return storage.read(uri)
+</code></pre></details></div><div class="mblk"><code>.read_text(uri, encoding)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/download/file.py#L354" target="_blank" class="gh-link">L354</a><div class="mdoc"><p>Read data from a given ``filepath`` with &#x27;r&#x27; mode.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>filepath (str or Path)</code> — Path to read data.</li>
+<li><code>encoding (str)</code> — The encoding format used to open the ``filepath``.</li>
+<li><code>Default</code> — &#x27;utf-8&#x27;.</li>
+</ul>
+<h4>Returns:</h4>
+<ul>
+<li><code>str</code> — Expected text reading from ``filepath``.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def read_text(uri: Union[str, Path], encoding: str = &quot;utf-8&quot;) -&gt; str:
+ &quot;&quot;&quot;Read data from a given ``filepath`` with &#x27;r&#x27; mode.
+
+ Args:
+ filepath (str or Path): Path to read data.
+ encoding (str): The encoding format used to open the ``filepath``.
+ Default: &#x27;utf-8&#x27;.
+
+ Returns:
+ str: Expected text reading from ``filepath``.
+ &quot;&quot;&quot;
+ storage = File._get_storage(uri)
+ return storage.read_text(uri)
+</code></pre></details></div><div class="mblk"><code>.write(obj, uri)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/download/file.py#L369" target="_blank" class="gh-link">L369</a><div class="mdoc"><p>Write data to a given ``filepath`` with &#x27;wb&#x27; mode.</p>
+<h4>Note:</h4>
+<p>``write`` will create a directory if the directory of ``filepath``</p>
+<p>does not exist.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>obj (bytes)</code> — Data to be written.</li>
+<li><code>filepath (str or Path)</code> — Path to write data.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def write(obj: bytes, uri: Union[str, Path]) -&gt; None:
+ &quot;&quot;&quot;Write data to a given ``filepath`` with &#x27;wb&#x27; mode.
+
+ Note:
+ ``write`` will create a directory if the directory of ``filepath``
+ does not exist.
+
+ Args:
+ obj (bytes): Data to be written.
+ filepath (str or Path): Path to write data.
+ &quot;&quot;&quot;
+ storage = File._get_storage(uri)
+ return storage.write(obj, uri)
+</code></pre></details></div><div class="mblk"><code>.write_text(obj, uri, encoding)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/download/file.py#L384" target="_blank" class="gh-link">L384</a><div class="mdoc"><p>Write data to a given ``filepath`` with &#x27;w&#x27; mode.</p>
+<h4>Note:</h4>
+<p>``write_text`` will create a directory if the directory of</p>
+<p>``filepath`` does not exist.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>obj (str)</code> — Data to be written.</li>
+<li><code>filepath (str or Path)</code> — Path to write data.</li>
+<li><code>encoding (str)</code> — The encoding format used to open the ``filepath``.</li>
+<li><code>Default</code> — &#x27;utf-8&#x27;.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def write_text(obj: str, uri: str, encoding: str = &quot;utf-8&quot;) -&gt; None:
+ &quot;&quot;&quot;Write data to a given ``filepath`` with &#x27;w&#x27; mode.
+
+ Note:
+ ``write_text`` will create a directory if the directory of
+ ``filepath`` does not exist.
+
+ Args:
+ obj (str): Data to be written.
+ filepath (str or Path): Path to write data.
+ encoding (str): The encoding format used to open the ``filepath``.
+ Default: &#x27;utf-8&#x27;.
+ &quot;&quot;&quot;
+ storage = File._get_storage(uri)
+ return storage.write_text(obj, uri)
+</code></pre></details></div><div class="mblk"><code>.as_local_path(uri)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/download/file.py#L401" target="_blank" class="gh-link">L401</a><div class="mdoc"><p>Only for unified API and do nothing.</p>
+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def as_local_path(uri: str) -&gt; Generator[Union[str, Path], None, None]:
+ &quot;&quot;&quot;Only for unified API and do nothing.&quot;&quot;&quot;
+ storage = File._get_storage(uri)
+ with storage.as_local_path(uri) as local_path:
+ yield local_path
+</code></pre></details></div></div>
+<div class="api-detail" id="e480"><span class="dtag badge-m">method</span><h2><code>File.read(uri)</code></h2><div class="dmod">funasr.download.file.File · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/download/file.py#L341" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Read data from a given ``filepath`` with &#x27;rb&#x27; mode.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>filepath (str or Path)</code> — Path to read data.</li>
+</ul>
+<h4>Returns:</h4>
+<ul>
+<li><code>bytes</code> — Expected bytes object.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def read(uri: str) -&gt; bytes:
+ &quot;&quot;&quot;Read data from a given ``filepath`` with &#x27;rb&#x27; mode.
+
+ Args:
+ filepath (str or Path): Path to read data.
+
+ Returns:
+ bytes: Expected bytes object.
+ &quot;&quot;&quot;
+ storage = File._get_storage(uri)
+ return storage.read(uri)
+</code></pre></details></div>
+<div class="api-detail" id="e481"><span class="dtag badge-m">method</span><h2><code>File.read_text(uri, encoding)</code></h2><div class="dmod">funasr.download.file.File · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/download/file.py#L354" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Read data from a given ``filepath`` with &#x27;r&#x27; mode.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>filepath (str or Path)</code> — Path to read data.</li>
+<li><code>encoding (str)</code> — The encoding format used to open the ``filepath``.</li>
+<li><code>Default</code> — &#x27;utf-8&#x27;.</li>
+</ul>
+<h4>Returns:</h4>
+<ul>
+<li><code>str</code> — Expected text reading from ``filepath``.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def read_text(uri: Union[str, Path], encoding: str = &quot;utf-8&quot;) -&gt; str:
+ &quot;&quot;&quot;Read data from a given ``filepath`` with &#x27;r&#x27; mode.
+
+ Args:
+ filepath (str or Path): Path to read data.
+ encoding (str): The encoding format used to open the ``filepath``.
+ Default: &#x27;utf-8&#x27;.
+
+ Returns:
+ str: Expected text reading from ``filepath``.
+ &quot;&quot;&quot;
+ storage = File._get_storage(uri)
+ return storage.read_text(uri)
+</code></pre></details></div>
+<div class="api-detail" id="e482"><span class="dtag badge-m">method</span><h2><code>File.write(obj, uri)</code></h2><div class="dmod">funasr.download.file.File · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/download/file.py#L369" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Write data to a given ``filepath`` with &#x27;wb&#x27; mode.</p>
+<h4>Note:</h4>
+<p>``write`` will create a directory if the directory of ``filepath``</p>
+<p>does not exist.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>obj (bytes)</code> — Data to be written.</li>
+<li><code>filepath (str or Path)</code> — Path to write data.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def write(obj: bytes, uri: Union[str, Path]) -&gt; None:
+ &quot;&quot;&quot;Write data to a given ``filepath`` with &#x27;wb&#x27; mode.
+
+ Note:
+ ``write`` will create a directory if the directory of ``filepath``
+ does not exist.
+
+ Args:
+ obj (bytes): Data to be written.
+ filepath (str or Path): Path to write data.
+ &quot;&quot;&quot;
+ storage = File._get_storage(uri)
+ return storage.write(obj, uri)
+</code></pre></details></div>
+<div class="api-detail" id="e483"><span class="dtag badge-m">method</span><h2><code>File.write_text(obj, uri, encoding)</code></h2><div class="dmod">funasr.download.file.File · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/download/file.py#L384" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Write data to a given ``filepath`` with &#x27;w&#x27; mode.</p>
+<h4>Note:</h4>
+<p>``write_text`` will create a directory if the directory of</p>
+<p>``filepath`` does not exist.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>obj (str)</code> — Data to be written.</li>
+<li><code>filepath (str or Path)</code> — Path to write data.</li>
+<li><code>encoding (str)</code> — The encoding format used to open the ``filepath``.</li>
+<li><code>Default</code> — &#x27;utf-8&#x27;.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def write_text(obj: str, uri: str, encoding: str = &quot;utf-8&quot;) -&gt; None:
+ &quot;&quot;&quot;Write data to a given ``filepath`` with &#x27;w&#x27; mode.
+
+ Note:
+ ``write_text`` will create a directory if the directory of
+ ``filepath`` does not exist.
+
+ Args:
+ obj (str): Data to be written.
+ filepath (str or Path): Path to write data.
+ encoding (str): The encoding format used to open the ``filepath``.
+ Default: &#x27;utf-8&#x27;.
+ &quot;&quot;&quot;
+ storage = File._get_storage(uri)
+ return storage.write_text(obj, uri)
+</code></pre></details></div>
+<div class="api-detail" id="e484"><span class="dtag badge-m">method</span><h2><code>File.as_local_path(uri)</code></h2><div class="dmod">funasr.download.file.File · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/download/file.py#L401" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Only for unified API and do nothing.</p>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def as_local_path(uri: str) -&gt; Generator[Union[str, Path], None, None]:
+ &quot;&quot;&quot;Only for unified API and do nothing.&quot;&quot;&quot;
+ storage = File._get_storage(uri)
+ with storage.as_local_path(uri) as local_path:
+ yield local_path
+</code></pre></details></div>
+<div class="api-detail" id="e485"><span class="dtag badge-f">function</span><h2><code>main()</code></h2><div class="dmod">funasr.download.runtime_sdk_download_tool · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/download/runtime_sdk_download_tool.py#L8" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Main.</p>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code>def main():
+ &quot;&quot;&quot;Main.&quot;&quot;&quot;
+ parser = argparse.ArgumentParser()
+ parser.add_argument(&quot;--model-name&quot;, type=str, required=True)
+ parser.add_argument(&quot;--export-dir&quot;, type=str, required=True)
+ parser.add_argument(&quot;--export&quot;, type=str2bool, default=True, help=&quot;whether to export model&quot;)
+ parser.add_argument(&quot;--type&quot;, type=str, default=&quot;onnx&quot;, help=&#x27;[&quot;onnx&quot;, &quot;torchscript&quot;, &quot;bladedisc&quot;]&#x27;)
+ parser.add_argument(&quot;--device&quot;, type=str, default=&quot;cpu&quot;, help=&#x27;[&quot;cpu&quot;, &quot;cuda&quot;]&#x27;)
+ parser.add_argument(&quot;--quantize&quot;, type=str2bool, default=False, help=&quot;export quantized model&quot;)
+ parser.add_argument(&quot;--fallback-num&quot;, type=int, default=0, help=&quot;amp fallback number&quot;)
+ parser.add_argument(&quot;--audio_in&quot;, type=str, default=None, help=&#x27;[&quot;wav&quot;, &quot;wav.scp&quot;]&#x27;)
+ parser.add_argument(&quot;--model_revision&quot;, type=str, default=None, help=&quot;model_revision&quot;)
+ parser.add_argument(&quot;--calib_num&quot;, type=int, default=200, help=&quot;calib max num&quot;)
+ args = parser.parse_args()
+
+ model_dir = args.model_name
+ output_dir = args.model_name
+ if not Path(args.model_name).exists():
+ from modelscope.hub.snapshot_download import snapshot_download
+
+ try:
+ model_dir = snapshot_download(
+ args.model_name, cache_dir=args.export_dir, revision=args.model_revision
+ )
+ output_dir = os.path.join(args.export_dir, args.model_name)
+ except:
+ raise &quot;model_dir must be model_name in modelscope or local path downloaded from modelscope, but is {}&quot;.format(
+ model_dir
+ )
+ if args.export:
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/download/runtime_sdk_download_tool.py#L8" target="_blank" class="src-more">View full source on GitHub →</a></details></div>
+<div class="api-detail" id="e486"><span class="dtag badge-f">function</span><h2><code>add_gradient_noise(model, iteration, duration, eta, scale_factor)</code></h2><div class="dmod">funasr.train_utils.add_gradient_noise · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/train_utils/add_gradient_noise.py#L4" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Adds noise from a standard normal distribution to the gradients.</p>
+<p>The standard deviation (`sigma`) is controlled</p>
+<p>by the three hyper-parameters below.</p>
+<p>`sigma` goes to zero (no noise) with more iterations.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>model</code> — Model.</li>
+<li><code>iteration</code> — Number of iterations.</li>
+<li><code>duration</code> — {100, 1000}: Number of durations to control</li>
+<li class="sub">the interval of the `sigma` change.</li>
+<li><code>eta</code> — {0.01, 0.3, 1.0}: The magnitude of `sigma`.</li>
+<li><code>scale_factor</code> — {0.55}: The scale of `sigma`.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code>def add_gradient_noise(
+ model: torch.nn.Module,
+ iteration: int,
+ duration: float = 100,
+ eta: float = 1.0,
+ scale_factor: float = 0.55,
+):
+ &quot;&quot;&quot;Adds noise from a standard normal distribution to the gradients.
+
+ The standard deviation (`sigma`) is controlled
+ by the three hyper-parameters below.
+ `sigma` goes to zero (no noise) with more iterations.
+
+ Args:
+ model: Model.
+ iteration: Number of iterations.
+ duration: {100, 1000}: Number of durations to control
+ the interval of the `sigma` change.
+ eta: {0.01, 0.3, 1.0}: The magnitude of `sigma`.
+ scale_factor: {0.55}: The scale of `sigma`.
+ &quot;&quot;&quot;
+ interval = (iteration // duration) + 1
+ sigma = eta / interval**scale_factor
+ for param in model.parameters():
+ if param.grad is not None:
+ _shape = param.grad.size()
+ noise = sigma * torch.randn(_shape).to(param.device)
+ param.grad += noise
+</code></pre></details></div>
+<div class="api-detail" id="e487"><span class="dtag badge-f">function</span><h2><code>average_checkpoints(output_dir, last_n, **kwargs)</code></h2><div class="dmod">funasr.train_utils.average_nbest_models · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/train_utils/average_nbest_models.py#L61" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Average the last &#x27;last_n&#x27; checkpoints&#x27; model state_dicts.</p>
+<p>If a tensor is of type torch.int, perform sum instead of average.</p>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code>def average_checkpoints(output_dir: str, last_n: int = 5, **kwargs):
+ &quot;&quot;&quot;
+ Average the last &#x27;last_n&#x27; checkpoints&#x27; model state_dicts.
+ If a tensor is of type torch.int, perform sum instead of average.
+ &quot;&quot;&quot;
+ checkpoint_paths = _get_checkpoint_paths(output_dir, last_n, **kwargs)
+ print(f&quot;average_checkpoints: {checkpoint_paths}&quot;)
+ state_dicts = []
+
+ # Load state_dicts from checkpoints
+ for path in checkpoint_paths:
+ if os.path.isfile(path):
+ state_dicts.append(torch.load(path, map_location=&quot;cpu&quot;)[&quot;state_dict&quot;])
+ else:
+ print(f&quot;Checkpoint file {path} not found.&quot;)
+
+ # Check if we have any state_dicts to average
+ if len(state_dicts) &lt; 1:
+ print(&quot;No checkpoints found for averaging.&quot;)
+ return
+
+ # Average or sum weights
+ avg_state_dict = OrderedDict()
+ for key in state_dicts[0].keys():
+ tensors = [state_dict[key].cpu() for state_dict in state_dicts]
+ # Check the type of the tensor
+ if str(tensors[0].dtype).startswith(&quot;torch.int&quot;):
+ # Perform sum for integer tensors
+ summed_tensor = sum(tensors)
+ avg_state_dict[key] = summed_tensor
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/train_utils/average_nbest_models.py#L61" target="_blank" class="src-more">View full source on GitHub →</a></details></div>
+<div class="api-detail" id="e488"><span class="dtag badge-f">function</span><h2><code>to_device(data, device, dtype, non_blocking, copy)</code></h2><div class="dmod">funasr.train_utils.device_funcs · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/train_utils/device_funcs.py#L8" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Change the device of object recursively</p>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code>def to_device(data, device=None, dtype=None, non_blocking=False, copy=False):
+ &quot;&quot;&quot;Change the device of object recursively&quot;&quot;&quot;
+ if isinstance(data, dict):
+ return {k: to_device(v, device, dtype, non_blocking, copy) for k, v in data.items()}
+ elif dataclasses.is_dataclass(data) and not isinstance(data, type):
+ return type(data)(
+ *[to_device(v, device, dtype, non_blocking, copy) for v in dataclasses.astuple(data)]
+ )
+ # maybe namedtuple. I don&#x27;t know the correct way to judge namedtuple.
+ elif isinstance(data, tuple) and type(data) is not tuple:
+ return type(data)(*[to_device(o, device, dtype, non_blocking, copy) for o in data])
+ elif isinstance(data, (list, tuple)):
+ return type(data)(to_device(v, device, dtype, non_blocking, copy) for v in data)
+ elif isinstance(data, np.ndarray):
+ return to_device(torch.from_numpy(data), device, dtype, non_blocking, copy)
+ elif isinstance(data, torch.Tensor):
+ return data.to(device, dtype, non_blocking, copy)
+ else:
+ return data
+</code></pre></details></div>
+<div class="api-detail" id="e489"><span class="dtag badge-f">function</span><h2><code>force_gatherable(data, device)</code></h2><div class="dmod">funasr.train_utils.device_funcs · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/train_utils/device_funcs.py#L29" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Change object to gatherable in torch.nn.DataParallel recursively</p>
+<p>The difference from to_device() is changing to torch.Tensor if float or int</p>
+<p>value is found.</p>
+<p>The restriction to the returned value in DataParallel:</p>
+<p>The object must be</p>
+<ul>
+<li>torch.cuda.Tensor</li>
+<li>1 or more dimension. 0-dimension-tensor sends warning.</li>
+</ul>
+<p>or a list, tuple, dict.</p>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code>def force_gatherable(data, device):
+ &quot;&quot;&quot;Change object to gatherable in torch.nn.DataParallel recursively
+
+ The difference from to_device() is changing to torch.Tensor if float or int
+ value is found.
+
+ The restriction to the returned value in DataParallel:
+ The object must be
+ - torch.cuda.Tensor
+ - 1 or more dimension. 0-dimension-tensor sends warning.
+ or a list, tuple, dict.
+
+ &quot;&quot;&quot;
+ if isinstance(data, dict):
+ return {k: force_gatherable(v, device) for k, v in data.items()}
+ # DataParallel can&#x27;t handle NamedTuple well
+ elif isinstance(data, tuple) and type(data) is not tuple:
+ return type(data)(*[force_gatherable(o, device) for o in data])
+ elif isinstance(data, (list, tuple, set)):
+ return type(data)(force_gatherable(v, device) for v in data)
+ elif isinstance(data, np.ndarray):
+ return force_gatherable(torch.from_numpy(data), device)
+ elif isinstance(data, torch.Tensor):
+ if data.dim() == 0:
+ # To 1-dim array
+ data = data[None]
+ return data.to(device)
+ elif isinstance(data, float):
+ return torch.tensor([data], dtype=torch.float, device=device)
+ elif isinstance(data, int):
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/train_utils/device_funcs.py#L29" target="_blank" class="src-more">View full source on GitHub →</a></details></div>
+<div class="api-detail" id="e490"><span class="dtag badge-c">class</span><h2>ForwardAdaptor</h2><div class="dmod">funasr.train_utils.forward_adaptor · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/train_utils/forward_adaptor.py#L4" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Wrapped module to parallelize specified method</p>
+<p>torch.nn.DataParallel parallelizes only &quot;forward()&quot;</p>
+<p>and, maybe, the method having the other name can&#x27;t be applied</p>
+<p>except for wrapping the module just like this class.</p>
+<h4>Examples:</h4>
+<p>&gt;&gt;&gt; class A(torch.nn.Module):</p>
+<p>... def foo(self, x):</p>
+<p>... ...</p>
+<p>&gt;&gt;&gt; model = A()</p>
+<p>&gt;&gt;&gt; model = ForwardAdaptor(model, &quot;foo&quot;)</p>
+<p>&gt;&gt;&gt; model = torch.nn.DataParallel(model, device_ids=[0, 1])</p>
+<p>&gt;&gt;&gt; x = torch.randn(2, 10)</p>
+<p>&gt;&gt;&gt; model(x)</p>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code>class ForwardAdaptor(torch.nn.Module):
+ &quot;&quot;&quot;Wrapped module to parallelize specified method
+
+ torch.nn.DataParallel parallelizes only &quot;forward()&quot;
+ and, maybe, the method having the other name can&#x27;t be applied
+ except for wrapping the module just like this class.
+
+ Examples:
+ &gt;&gt;&gt; class A(torch.nn.Module):
+ ... def foo(self, x):
+ ... ...
+ &gt;&gt;&gt; model = A()
+ &gt;&gt;&gt; model = ForwardAdaptor(model, &quot;foo&quot;)
+ &gt;&gt;&gt; model = torch.nn.DataParallel(model, device_ids=[0, 1])
+ &gt;&gt;&gt; x = torch.randn(2, 10)
+ &gt;&gt;&gt; model(x)
+ &quot;&quot;&quot;
+
+ def __init__(self, module: torch.nn.Module, name: str):
+ &quot;&quot;&quot;Initialize ForwardAdaptor.
+
+ Args:
+ module: TODO.
+ name: TODO.
+ &quot;&quot;&quot;
+ super().__init__()
+ self.module = module
+ self.name = name
+ if not hasattr(module, name):
+ raise ValueError(f&quot;{module} doesn&#x27;t have {name}&quot;)
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/train_utils/forward_adaptor.py#L4" target="_blank" class="src-more">View full source on GitHub →</a></details><h3 class="mt">Methods</h3><div class="mblk"><code>.forward(*args, **kwargs)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/train_utils/forward_adaptor.py#L35" target="_blank" class="gh-link">L35</a><div class="mdoc"><p>Forward pass for training.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>*args</code> — Variable positional arguments.</li>
+<li><code>**kwargs</code> — Additional keyword arguments.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def forward(self, *args, **kwargs):
+ &quot;&quot;&quot;Forward pass for training.
+
+ Args:
+ *args: Variable positional arguments.
+ **kwargs: Additional keyword arguments.
+ &quot;&quot;&quot;
+ func = getattr(self.module, self.name)
+ return func(*args, **kwargs)
+</code></pre></details></div></div>
+<div class="api-detail" id="e491"><span class="dtag badge-m">method</span><h2><code>ForwardAdaptor.forward(*args, **kwargs)</code></h2><div class="dmod">funasr.train_utils.forward_adaptor.ForwardAdaptor · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/train_utils/forward_adaptor.py#L35" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Forward pass for training.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>*args</code> — Variable positional arguments.</li>
+<li><code>**kwargs</code> — Additional keyword arguments.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def forward(self, *args, **kwargs):
+ &quot;&quot;&quot;Forward pass for training.
+
+ Args:
+ *args: Variable positional arguments.
+ **kwargs: Additional keyword arguments.
+ &quot;&quot;&quot;
+ func = getattr(self.module, self.name)
+ return func(*args, **kwargs)
+</code></pre></details></div>
+<div class="api-detail" id="e492"><span class="dtag badge-f">function</span><h2><code>initialize(model, init)</code></h2><div class="dmod">funasr.train_utils.initialize · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/train_utils/initialize.py#L9" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Initialize weights of a neural network module.</p>
+<p>Parameters are initialized using the given method or distribution.</p>
+<p>Custom initialization routines can be implemented into submodules</p>
+<p>as function `espnet_initialization_fn` within the custom module.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>model</code> — Target.</li>
+<li><code>init</code> — Method of initialization.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code>def initialize(model: torch.nn.Module, init: str):
+ &quot;&quot;&quot;Initialize weights of a neural network module.
+
+ Parameters are initialized using the given method or distribution.
+
+ Custom initialization routines can be implemented into submodules
+ as function `espnet_initialization_fn` within the custom module.
+
+ Args:
+ model: Target.
+ init: Method of initialization.
+ &quot;&quot;&quot;
+
+ # weight init
+ for p in model.parameters():
+ if p.dim() &gt; 1:
+ if init == &quot;xavier_uniform&quot;:
+ torch.nn.init.xavier_uniform_(p.data)
+ elif init == &quot;xavier_normal&quot;:
+ torch.nn.init.xavier_normal_(p.data)
+ elif init == &quot;kaiming_uniform&quot;:
+ torch.nn.init.kaiming_uniform_(p.data, nonlinearity=&quot;relu&quot;)
+ elif init == &quot;kaiming_normal&quot;:
+ torch.nn.init.kaiming_normal_(p.data, nonlinearity=&quot;relu&quot;)
+ else:
+ raise ValueError(&quot;Unknown initialization: &quot; + init)
+ # bias init
+ for p in model.parameters():
+ if p.dim() == 1:
+ p.data.zero_()
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/train_utils/initialize.py#L9" target="_blank" class="src-more">View full source on GitHub →</a></details></div>
+<div class="api-detail" id="e493"><span class="dtag badge-f">function</span><h2><code>load_pretrained_model(path, model, ignore_init_mismatch, map_location, oss_bucket, scope_map, excludes, **kwargs)</code></h2><div class="dmod">funasr.train_utils.load_pretrained_model · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/train_utils/load_pretrained_model.py#L14" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Load a model state and set it to the model.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>init_param</code> — &lt;file_path&gt;:&lt;src_key&gt;:&lt;dst_key&gt;:&lt;exclude_Keys&gt;</li>
+</ul>
+<h4>Examples:</h4>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code>def load_pretrained_model(
+ path: str,
+ model: torch.nn.Module,
+ ignore_init_mismatch: bool = True,
+ map_location: str = &quot;cpu&quot;,
+ oss_bucket=None,
+ scope_map=[],
+ excludes=None,
+ **kwargs,
+):
+ &quot;&quot;&quot;Load a model state and set it to the model.
+
+ Args:
+ init_param: &lt;file_path&gt;:&lt;src_key&gt;:&lt;dst_key&gt;:&lt;exclude_Keys&gt;
+
+ Examples:
+
+ &quot;&quot;&quot;
+
+ obj = model
+ dst_state = obj.state_dict()
+
+ logging.info(f&quot;ckpt: {path}&quot;)
+
+ if oss_bucket is None:
+ ori_state = torch.load(path, map_location=map_location)
+ else:
+ buffer = BytesIO(oss_bucket.get_object(path).read())
+ ori_state = torch.load(buffer, map_location=map_location)
+
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/train_utils/load_pretrained_model.py#L14" target="_blank" class="src-more">View full source on GitHub →</a></details></div>
+<div class="api-detail" id="e494"><span class="dtag badge-f">function</span><h2><code>get_human_readable_count(number)</code></h2><div class="dmod">funasr.train_utils.model_summary · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/train_utils/model_summary.py#L5" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Return human_readable_count</p>
+<p>Originated from:</p>
+<p>https://github.com/PyTorchLightning/pytorch-lightning/blob/master/pytorch_lightning/core/memory.py</p>
+<p>Abbreviates an integer number with K, M, B, T for thousands, millions,</p>
+<p>billions and trillions, respectively.</p>
+<h4>Examples:</h4>
+<p>&gt;&gt;&gt; get_human_readable_count(123)</p>
+<p>&#x27;123 &#x27;</p>
+<p>&gt;&gt;&gt; get_human_readable_count(1234) # (one thousand)</p>
+<p>&#x27;1 K&#x27;</p>
+<p>&gt;&gt;&gt; get_human_readable_count(2e6) # (two million)</p>
+<p>&#x27;2 M&#x27;</p>
+<p>&gt;&gt;&gt; get_human_readable_count(3e9) # (three billion)</p>
+<p>&#x27;3 B&#x27;</p>
+<p>&gt;&gt;&gt; get_human_readable_count(4e12) # (four trillion)</p>
+<p>&#x27;4 T&#x27;</p>
+<p>&gt;&gt;&gt; get_human_readable_count(5e15) # (more than trillion)</p>
+<p>&#x27;5,000 T&#x27;</p>
+<h4>Args:</h4>
+<ul>
+<li><code>number</code> — a positive integer number</li>
+</ul>
+<p>Return:</p>
+<p>A string formatted according to the pattern described above.</p>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code>def get_human_readable_count(number: int) -&gt; str:
+ &quot;&quot;&quot;Return human_readable_count
+
+ Originated from:
+ https://github.com/PyTorchLightning/pytorch-lightning/blob/master/pytorch_lightning/core/memory.py
+
+ Abbreviates an integer number with K, M, B, T for thousands, millions,
+ billions and trillions, respectively.
+ Examples:
+ &gt;&gt;&gt; get_human_readable_count(123)
+ &#x27;123 &#x27;
+ &gt;&gt;&gt; get_human_readable_count(1234) # (one thousand)
+ &#x27;1 K&#x27;
+ &gt;&gt;&gt; get_human_readable_count(2e6) # (two million)
+ &#x27;2 M&#x27;
+ &gt;&gt;&gt; get_human_readable_count(3e9) # (three billion)
+ &#x27;3 B&#x27;
+ &gt;&gt;&gt; get_human_readable_count(4e12) # (four trillion)
+ &#x27;4 T&#x27;
+ &gt;&gt;&gt; get_human_readable_count(5e15) # (more than trillion)
+ &#x27;5,000 T&#x27;
+ Args:
+ number: a positive integer number
+ Return:
+ A string formatted according to the pattern described above.
+ &quot;&quot;&quot;
+ assert number &gt;= 0
+ labels = [&quot; &quot;, &quot;K&quot;, &quot;M&quot;, &quot;B&quot;, &quot;T&quot;]
+ num_digits = int(np.floor(np.log10(number)) + 1 if number &gt; 0 else 1)
+ num_groups = int(np.ceil(num_digits / 3))
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/train_utils/model_summary.py#L5" target="_blank" class="src-more">View full source on GitHub →</a></details></div>
+<div class="api-detail" id="e495"><span class="dtag badge-f">function</span><h2><code>to_bytes(dtype)</code></h2><div class="dmod">funasr.train_utils.model_summary · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/train_utils/model_summary.py#L42" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>To bytes.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>dtype</code> — TODO.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code>def to_bytes(dtype) -&gt; int:
+ # torch.float16 -&gt; 16
+ &quot;&quot;&quot;To bytes.
+
+ Args:
+ dtype: TODO.
+ &quot;&quot;&quot;
+ return int(str(dtype)[-2:]) // 8
+</code></pre></details></div>
+<div class="api-detail" id="e496"><span class="dtag badge-f">function</span><h2><code>model_summary(model)</code></h2><div class="dmod">funasr.train_utils.model_summary · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/train_utils/model_summary.py#L52" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Model summary.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>model</code> — Model instance or model name.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code>def model_summary(model: torch.nn.Module) -&gt; str:
+ &quot;&quot;&quot;Model summary.
+
+ Args:
+ model: Model instance or model name.
+ &quot;&quot;&quot;
+ message = &quot;Model structure:\n&quot;
+ message += str(model)
+
+ tot_params, num_params = 0, 0
+ for name, param in model.named_parameters():
+ print(
+ &quot;name: {}, dtype: {}, device: {}, trainable: {}, shape: {}, numel: {}&quot;.format(
+ name, param.dtype, param.device, param.requires_grad, param.shape, param.numel()
+ )
+ )
+ tot_params += param.numel()
+ if param.requires_grad:
+ num_params += param.numel()
+
+ percent_trainable = &quot;{:.1f}&quot;.format(num_params * 100.0 / tot_params)
+ tot_params = get_human_readable_count(tot_params)
+ num_params = get_human_readable_count(num_params)
+ message += &quot;\n\nModel summary:\n&quot;
+ message += f&quot; Class Name: {model.__class__.__name__}\n&quot;
+ message += f&quot; Total Number of model parameters: {tot_params}\n&quot;
+ message += f&quot; Number of trainable parameters: {num_params} ({percent_trainable}%)\n&quot;
+
+ dtype = next(iter(model.parameters())).dtype
+ message += f&quot; Type: {dtype}&quot;
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/train_utils/model_summary.py#L52" target="_blank" class="src-more">View full source on GitHub →</a></details></div>
+<div class="api-detail" id="e497"><span class="dtag badge-f">function</span><h2><code>recursive_sum(obj, weight, distributed)</code></h2><div class="dmod">funasr.train_utils.recursive_op · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/train_utils/recursive_op.py#L9" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Recursive sum.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>obj</code> — TODO.</li>
+<li><code>weight</code> — TODO.</li>
+<li><code>distributed</code> — TODO.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code>def recursive_sum(obj, weight: torch.Tensor, distributed: bool = False):
+ &quot;&quot;&quot;Recursive sum.
+
+ Args:
+ obj: TODO.
+ weight: TODO.
+ distributed: TODO.
+ &quot;&quot;&quot;
+ assert weight.dim() == 1, weight.size()
+ if isinstance(obj, (tuple, list)):
+ return type(obj)(recursive_sum(v, weight, distributed) for v in obj)
+ elif isinstance(obj, dict):
+ return {k: recursive_sum(v, weight, distributed) for k, v in obj.items()}
+ elif isinstance(obj, torch.Tensor):
+ assert obj.size() == weight.size(), (obj.size(), weight.size())
+ obj = (obj * weight.type(obj.dtype)).sum()
+ if distributed:
+ torch.distributed.all_reduce(obj, op=ReduceOp.SUM)
+ return obj
+ elif obj is None:
+ return None
+ else:
+ raise ValueError(type(obj))
+</code></pre></details></div>
+<div class="api-detail" id="e498"><span class="dtag badge-f">function</span><h2><code>recursive_divide(a, b)</code></h2><div class="dmod">funasr.train_utils.recursive_op · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/train_utils/recursive_op.py#L34" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Recursive divide.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>a</code> — TODO.</li>
+<li><code>b</code> — TODO.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code>def recursive_divide(a, b: torch.Tensor):
+ &quot;&quot;&quot;Recursive divide.
+
+ Args:
+ a: TODO.
+ b: TODO.
+ &quot;&quot;&quot;
+ if isinstance(a, (tuple, list)):
+ return type(a)(recursive_divide(v, b) for v in a)
+ elif isinstance(a, dict):
+ return {k: recursive_divide(v, b) for k, v in a.items()}
+ elif isinstance(a, torch.Tensor):
+ assert a.size() == b.size(), (a.size(), b.size())
+ return a / b.type(a.dtype)
+ elif a is None:
+ return None
+ else:
+ raise ValueError(type(a))
+</code></pre></details></div>
+<div class="api-detail" id="e499"><span class="dtag badge-f">function</span><h2><code>recursive_average(obj, weight, distributed)</code></h2><div class="dmod">funasr.train_utils.recursive_op · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/train_utils/recursive_op.py#L54" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Recursive average.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>obj</code> — TODO.</li>
+<li><code>weight</code> — TODO.</li>
+<li><code>distributed</code> — TODO.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code>def recursive_average(obj, weight: torch.Tensor, distributed: bool = False):
+ &quot;&quot;&quot;Recursive average.
+
+ Args:
+ obj: TODO.
+ weight: TODO.
+ distributed: TODO.
+ &quot;&quot;&quot;
+ obj = recursive_sum(obj, weight, distributed)
+ weight = weight.sum()
+ if distributed:
+ torch.distributed.all_reduce(weight, op=ReduceOp.SUM)
+ # Normalize weight to be sum-to-1
+ obj = recursive_divide(obj, weight)
+ return obj, weight
+</code></pre></details></div>
+<div class="api-detail" id="e500"><span class="dtag badge-f">function</span><h2><code>set_all_random_seed(seed)</code></h2><div class="dmod">funasr.train_utils.set_all_random_seed · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/train_utils/set_all_random_seed.py#L7" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Set all random seed.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>seed</code> — TODO.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code>def set_all_random_seed(seed: int):
+ &quot;&quot;&quot;Set all random seed.
+
+ Args:
+ seed: TODO.
+ &quot;&quot;&quot;
+ random.seed(seed)
+ np.random.seed(seed)
+ torch.random.manual_seed(seed)
+</code></pre></details></div>
+<div class="api-detail" id="e501"><span class="dtag badge-f">function</span><h2><code>maybe_autocast(enabled)</code></h2><div class="dmod">funasr.train_utils.trainer · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/train_utils/trainer.py#L25" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Maybe autocast.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>enabled</code> — TODO.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code>def maybe_autocast(enabled):
+ &quot;&quot;&quot;Maybe autocast.
+
+ Args:
+ enabled: TODO.
+ &quot;&quot;&quot;
+ if enabled:
+ with autocast():
+ yield
+ else:
+ yield
+</code></pre></details></div>
+<div class="api-detail" id="e502"><span class="dtag badge-c">class</span><h2>Trainer</h2><div class="dmod">funasr.train_utils.trainer · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/train_utils/trainer.py#L38" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>A simple trainer class for training a PyTorch model, saving checkpoints at the end of each epoch,</p>
+<p>and optionally resuming from a saved checkpoint.</p>
+<p>Attributes:</p>
+<ul>
+<li><code>max_epoch (int)</code> — Maximum number of epochs for training.</li>
+<li><code>model (torch.nn.Module)</code> — The model to be trained.</li>
+<li><code>optim (torch.optim.Optimizer)</code> — The optimizer to use for training.</li>
+<li><code>scheduler (torch.optim.lr_scheduler._LRScheduler)</code> — The learning rate scheduler.</li>
+<li><code>dataloader_train (torch.utils.data.DataLoader)</code> — DataLoader for the training dataset.</li>
+<li><code>dataloader_val (torch.utils.data.DataLoader)</code> — DataLoader for the validation dataset.</li>
+<li><code>output_dir (str)</code> — Directory where model checkpoints will be saved.</li>
+<li><code>resume (str, optional)</code> — Path to a checkpoint to resume training from.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code>class Trainer:
+ &quot;&quot;&quot;
+ A simple trainer class for training a PyTorch model, saving checkpoints at the end of each epoch,
+ and optionally resuming from a saved checkpoint.
+
+ Attributes:
+ max_epoch (int): Maximum number of epochs for training.
+ model (torch.nn.Module): The model to be trained.
+ optim (torch.optim.Optimizer): The optimizer to use for training.
+ scheduler (torch.optim.lr_scheduler._LRScheduler): The learning rate scheduler.
+ dataloader_train (torch.utils.data.DataLoader): DataLoader for the training dataset.
+ dataloader_val (torch.utils.data.DataLoader): DataLoader for the validation dataset.
+ output_dir (str): Directory where model checkpoints will be saved.
+ resume (str, optional): Path to a checkpoint to resume training from.
+ &quot;&quot;&quot;
+
+ def __init__(
+ self,
+ local_rank,
+ use_ddp: bool = False,
+ use_fsdp: bool = False,
+ use_fp16: bool = False,
+ output_dir: str = &quot;./&quot;,
+ **kwargs,
+ ):
+ &quot;&quot;&quot;
+ Initializes the Trainer class with the model, optimizer, scheduler, dataloaders, and other settings.
+
+ Args:
+ model (torch.nn.Module): The model to be trained.
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/train_utils/trainer.py#L38" target="_blank" class="src-more">View full source on GitHub →</a></details><h3 class="mt">Methods</h3><div class="mblk"><code>.save_checkpoint(epoch, step, model, optim, scheduler, scaler, step_in_epoch, **kwargs)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/train_utils/trainer.py#L143" target="_blank" class="gh-link">L143</a><div class="mdoc"><p>Saves a checkpoint containing the model&#x27;s state, the optimizer&#x27;s state,</p>
+<p>and the scheduler&#x27;s state at the end of the given epoch. This method is</p>
+<p>intended to be called at the end of each epoch to save the training progress.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>epoch (int)</code> — The epoch number at which the checkpoint is being saved.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def save_checkpoint(
+ self,
+ epoch,
+ step=None,
+ model=None,
+ optim=None,
+ scheduler=None,
+ scaler=None,
+ step_in_epoch=None,
+ **kwargs,
+ ):
+ &quot;&quot;&quot;
+ Saves a checkpoint containing the model&#x27;s state, the optimizer&#x27;s state,
+ and the scheduler&#x27;s state at the end of the given epoch. This method is
+ intended to be called at the end of each epoch to save the training progress.
+
+ Args:
+ epoch (int): The epoch number at which the checkpoint is being saved.
+ &quot;&quot;&quot;
+
+ step_in_epoch = None if step is None else step_in_epoch
+ if self.rank == 0:
+ logging.info(f&quot;Save checkpoint: {epoch}, rank: {self.local_rank}\n&quot;)
+ # self.step_or_epoch += 1
+ state = {
+ &quot;epoch&quot;: epoch,
+ &quot;step&quot;: step,
+ &quot;total_step&quot;: self.batch_total,
+ &quot;state_dict&quot;: model.state_dict(),
+ &quot;optimizer&quot;: optim.state_dict(),
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/train_utils/trainer.py#L143" target="_blank" class="src-more">View full source →</a></details></div><div class="mblk"><code>.resume_checkpoint(model, optim, scheduler, scaler)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/train_utils/trainer.py#L260" target="_blank" class="gh-link">L260</a><div class="mdoc"><p>Resumes training from a checkpoint at the given file path.</p>
+<p>Loads the model&#x27;s state, the optimizer&#x27;s state, and the scheduler&#x27;s state.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>resume_path (str)</code> — The file path to the checkpoint to resume from.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def resume_checkpoint(
+ self,
+ model=None,
+ optim=None,
+ scheduler=None,
+ scaler=None,
+ ):
+ &quot;&quot;&quot;
+ Resumes training from a checkpoint at the given file path.
+ Loads the model&#x27;s state, the optimizer&#x27;s state, and the scheduler&#x27;s state.
+
+ Args:
+ resume_path (str): The file path to the checkpoint to resume from.
+ &quot;&quot;&quot;
+ if self.resume:
+ ckpt = os.path.join(self.output_dir, &quot;model.pt&quot;)
+ if os.path.isfile(ckpt):
+ checkpoint = torch.load(ckpt, map_location=&quot;cpu&quot;)
+ self.start_epoch = checkpoint[&quot;epoch&quot;]
+ # self.model.load_state_dict(checkpoint[&#x27;state_dict&#x27;])
+ src_state = checkpoint[&quot;state_dict&quot;]
+ dst_state = model.state_dict()
+ for k in dst_state.keys():
+ if not k.startswith(&quot;module.&quot;) and &quot;module.&quot; + k in src_state.keys():
+ k_ddp = &quot;module.&quot; + k
+ elif k.startswith(&quot;module.&quot;) and &quot;module.&quot; + k not in src_state.keys():
+ k_ddp = k.replace(&quot;module.&quot;, &quot;&quot;, 1)
+ else:
+ k_ddp = k
+
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/train_utils/trainer.py#L260" target="_blank" class="src-more">View full source →</a></details></div><div class="mblk"><code>.train_epoch(model, optim, scheduler, scaler, dataloader_train, dataloader_val, epoch, writer, **kwargs)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/train_utils/trainer.py#L340" target="_blank" class="gh-link">L340</a><div class="mdoc"><p>Defines the training process for a single epoch with gradient accumulation.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>epoch (int)</code> — The current epoch number.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def train_epoch(
+ self,
+ model=None,
+ optim=None,
+ scheduler=None,
+ scaler=None,
+ dataloader_train=None,
+ dataloader_val=None,
+ epoch=None,
+ writer=None,
+ **kwargs,
+ ):
+ &quot;&quot;&quot;
+ Defines the training process for a single epoch with gradient accumulation.
+ Args:
+ epoch (int): The current epoch number.
+ &quot;&quot;&quot;
+ if self.use_ddp or self.use_fsdp:
+ dist.barrier()
+ logging.info(f&quot;Train epoch: {epoch}, rank: {self.rank}\n&quot;)
+ model.train()
+
+ # Set the number of steps for gradient accumulation
+ accum_grad = self.accum_grad
+ # Initialize the gradient accumulation
+ optim.zero_grad()
+ speed_stats = {}
+
+ iterator_stop = torch.tensor(0).to(self.device)
+
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/train_utils/trainer.py#L340" target="_blank" class="src-more">View full source →</a></details></div><div class="mblk"><code>.validate_epoch(model, dataloader_val, epoch, writer, **kwargs)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/train_utils/trainer.py#L537" target="_blank" class="gh-link">L537</a><div class="mdoc"><p>Defines the validation process for a single epoch.</p>
+<p>Should be implemented with the actual model validation steps.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>epoch (int)</code> — The current epoch number.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def validate_epoch(
+ self,
+ model=None,
+ dataloader_val=None,
+ epoch=None,
+ writer=None,
+ **kwargs,
+ ):
+ &quot;&quot;&quot;
+ Defines the validation process for a single epoch.
+ Should be implemented with the actual model validation steps.
+
+ Args:
+ epoch (int): The current epoch number.
+ &quot;&quot;&quot;
+ if self.use_ddp or self.use_fsdp:
+ dist.barrier()
+ logging.info(f&quot;Validate epoch: {epoch}, rank: {self.rank}\n&quot;)
+ model.eval()
+
+ with torch.no_grad():
+
+ speed_stats = {}
+ time5 = time.perf_counter()
+ iterator_stop = torch.tensor(0).to(self.device)
+ dataloader_val.batch_sampler.set_epoch(epoch)
+ for batch_idx, batch in enumerate(dataloader_val):
+ if self.use_ddp or self.use_fsdp:
+ dist.all_reduce(iterator_stop, dist.ReduceOp.SUM)
+ if iterator_stop &gt; 0:
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/train_utils/trainer.py#L537" target="_blank" class="src-more">View full source →</a></details></div><div class="mblk"><code>.log(epoch, batch_idx, step_in_epoch, batch_num_epoch, lr, loss, speed_stats, stats, writer, tag, data_split_i, data_split_num, log_step, **kwargs)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/train_utils/trainer.py#L651" target="_blank" class="gh-link">L651</a><div class="mdoc"><p>Log.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>epoch</code> — TODO.</li>
+<li><code>batch_idx</code> — TODO.</li>
+<li><code>step_in_epoch</code> — TODO.</li>
+<li><code>batch_num_epoch</code> — TODO.</li>
+<li><code>lr</code> — TODO.</li>
+<li><code>loss</code> — TODO.</li>
+<li><code>speed_stats</code> — TODO.</li>
+<li><code>stats</code> — TODO.</li>
+<li><code>writer</code> — TODO.</li>
+<li><code>tag</code> — TODO.</li>
+<li><code>data_split_i</code> — TODO.</li>
+<li><code>data_split_num</code> — TODO.</li>
+<li><code>log_step</code> — TODO.</li>
+<li><code>**kwargs</code> — Additional keyword arguments.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def log(
+ self,
+ epoch=0,
+ batch_idx=0,
+ step_in_epoch=0,
+ batch_num_epoch=-1,
+ lr=0.0,
+ loss=0.0,
+ speed_stats=None,
+ stats=None,
+ writer=None,
+ tag=&quot;train&quot;,
+ data_split_i=0,
+ data_split_num=1,
+ log_step=None,
+ **kwargs,
+ ):
+
+ &quot;&quot;&quot;Log.
+
+ Args:
+ epoch: TODO.
+ batch_idx: TODO.
+ step_in_epoch: TODO.
+ batch_num_epoch: TODO.
+ lr: TODO.
+ loss: TODO.
+ speed_stats: TODO.
+ stats: TODO.
+ writer: TODO.
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/train_utils/trainer.py#L651" target="_blank" class="src-more">View full source →</a></details></div><div class="mblk"><code>.close(writer)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/train_utils/trainer.py#L744" target="_blank" class="gh-link">L744</a><div class="mdoc"><p>Close.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>writer</code> — TODO.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def close(self, writer=None):
+
+ &quot;&quot;&quot;Close.
+
+ Args:
+ writer: TODO.
+ &quot;&quot;&quot;
+ if self.use_ddp or self.use_fsdp:
+ dist.barrier()
+
+ if writer is not None:
+ writer.close()
+
+ if self.use_ddp or self.use_fsdp:
+ torch.distributed.destroy_process_group()
+</code></pre></details></div></div>
+<div class="api-detail" id="e503"><span class="dtag badge-m">method</span><h2><code>Trainer.save_checkpoint(epoch, step, model, optim, scheduler, scaler, step_in_epoch, **kwargs)</code></h2><div class="dmod">funasr.train_utils.trainer.Trainer · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/train_utils/trainer.py#L143" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Saves a checkpoint containing the model&#x27;s state, the optimizer&#x27;s state,</p>
+<p>and the scheduler&#x27;s state at the end of the given epoch. This method is</p>
+<p>intended to be called at the end of each epoch to save the training progress.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>epoch (int)</code> — The epoch number at which the checkpoint is being saved.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def save_checkpoint(
+ self,
+ epoch,
+ step=None,
+ model=None,
+ optim=None,
+ scheduler=None,
+ scaler=None,
+ step_in_epoch=None,
+ **kwargs,
+ ):
+ &quot;&quot;&quot;
+ Saves a checkpoint containing the model&#x27;s state, the optimizer&#x27;s state,
+ and the scheduler&#x27;s state at the end of the given epoch. This method is
+ intended to be called at the end of each epoch to save the training progress.
+
+ Args:
+ epoch (int): The epoch number at which the checkpoint is being saved.
+ &quot;&quot;&quot;
+
+ step_in_epoch = None if step is None else step_in_epoch
+ if self.rank == 0:
+ logging.info(f&quot;Save checkpoint: {epoch}, rank: {self.local_rank}\n&quot;)
+ # self.step_or_epoch += 1
+ state = {
+ &quot;epoch&quot;: epoch,
+ &quot;step&quot;: step,
+ &quot;total_step&quot;: self.batch_total,
+ &quot;state_dict&quot;: model.state_dict(),
+ &quot;optimizer&quot;: optim.state_dict(),
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/train_utils/trainer.py#L143" target="_blank" class="src-more">View full source on GitHub →</a></details></div>
+<div class="api-detail" id="e504"><span class="dtag badge-m">method</span><h2><code>Trainer.resume_checkpoint(model, optim, scheduler, scaler)</code></h2><div class="dmod">funasr.train_utils.trainer.Trainer · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/train_utils/trainer.py#L260" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Resumes training from a checkpoint at the given file path.</p>
+<p>Loads the model&#x27;s state, the optimizer&#x27;s state, and the scheduler&#x27;s state.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>resume_path (str)</code> — The file path to the checkpoint to resume from.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def resume_checkpoint(
+ self,
+ model=None,
+ optim=None,
+ scheduler=None,
+ scaler=None,
+ ):
+ &quot;&quot;&quot;
+ Resumes training from a checkpoint at the given file path.
+ Loads the model&#x27;s state, the optimizer&#x27;s state, and the scheduler&#x27;s state.
+
+ Args:
+ resume_path (str): The file path to the checkpoint to resume from.
+ &quot;&quot;&quot;
+ if self.resume:
+ ckpt = os.path.join(self.output_dir, &quot;model.pt&quot;)
+ if os.path.isfile(ckpt):
+ checkpoint = torch.load(ckpt, map_location=&quot;cpu&quot;)
+ self.start_epoch = checkpoint[&quot;epoch&quot;]
+ # self.model.load_state_dict(checkpoint[&#x27;state_dict&#x27;])
+ src_state = checkpoint[&quot;state_dict&quot;]
+ dst_state = model.state_dict()
+ for k in dst_state.keys():
+ if not k.startswith(&quot;module.&quot;) and &quot;module.&quot; + k in src_state.keys():
+ k_ddp = &quot;module.&quot; + k
+ elif k.startswith(&quot;module.&quot;) and &quot;module.&quot; + k not in src_state.keys():
+ k_ddp = k.replace(&quot;module.&quot;, &quot;&quot;, 1)
+ else:
+ k_ddp = k
+
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/train_utils/trainer.py#L260" target="_blank" class="src-more">View full source on GitHub →</a></details></div>
+<div class="api-detail" id="e505"><span class="dtag badge-m">method</span><h2><code>Trainer.train_epoch(model, optim, scheduler, scaler, dataloader_train, dataloader_val, epoch, writer, **kwargs)</code></h2><div class="dmod">funasr.train_utils.trainer.Trainer · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/train_utils/trainer.py#L340" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Defines the training process for a single epoch with gradient accumulation.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>epoch (int)</code> — The current epoch number.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def train_epoch(
+ self,
+ model=None,
+ optim=None,
+ scheduler=None,
+ scaler=None,
+ dataloader_train=None,
+ dataloader_val=None,
+ epoch=None,
+ writer=None,
+ **kwargs,
+ ):
+ &quot;&quot;&quot;
+ Defines the training process for a single epoch with gradient accumulation.
+ Args:
+ epoch (int): The current epoch number.
+ &quot;&quot;&quot;
+ if self.use_ddp or self.use_fsdp:
+ dist.barrier()
+ logging.info(f&quot;Train epoch: {epoch}, rank: {self.rank}\n&quot;)
+ model.train()
+
+ # Set the number of steps for gradient accumulation
+ accum_grad = self.accum_grad
+ # Initialize the gradient accumulation
+ optim.zero_grad()
+ speed_stats = {}
+
+ iterator_stop = torch.tensor(0).to(self.device)
+
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/train_utils/trainer.py#L340" target="_blank" class="src-more">View full source on GitHub →</a></details></div>
+<div class="api-detail" id="e506"><span class="dtag badge-m">method</span><h2><code>Trainer.validate_epoch(model, dataloader_val, epoch, writer, **kwargs)</code></h2><div class="dmod">funasr.train_utils.trainer.Trainer · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/train_utils/trainer.py#L537" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Defines the validation process for a single epoch.</p>
+<p>Should be implemented with the actual model validation steps.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>epoch (int)</code> — The current epoch number.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def validate_epoch(
+ self,
+ model=None,
+ dataloader_val=None,
+ epoch=None,
+ writer=None,
+ **kwargs,
+ ):
+ &quot;&quot;&quot;
+ Defines the validation process for a single epoch.
+ Should be implemented with the actual model validation steps.
+
+ Args:
+ epoch (int): The current epoch number.
+ &quot;&quot;&quot;
+ if self.use_ddp or self.use_fsdp:
+ dist.barrier()
+ logging.info(f&quot;Validate epoch: {epoch}, rank: {self.rank}\n&quot;)
+ model.eval()
+
+ with torch.no_grad():
+
+ speed_stats = {}
+ time5 = time.perf_counter()
+ iterator_stop = torch.tensor(0).to(self.device)
+ dataloader_val.batch_sampler.set_epoch(epoch)
+ for batch_idx, batch in enumerate(dataloader_val):
+ if self.use_ddp or self.use_fsdp:
+ dist.all_reduce(iterator_stop, dist.ReduceOp.SUM)
+ if iterator_stop &gt; 0:
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/train_utils/trainer.py#L537" target="_blank" class="src-more">View full source on GitHub →</a></details></div>
+<div class="api-detail" id="e507"><span class="dtag badge-m">method</span><h2><code>Trainer.log(epoch, batch_idx, step_in_epoch, batch_num_epoch, lr, loss, speed_stats, stats, writer, tag, data_split_i, data_split_num, log_step, **kwargs)</code></h2><div class="dmod">funasr.train_utils.trainer.Trainer · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/train_utils/trainer.py#L651" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Log.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>epoch</code> — TODO.</li>
+<li><code>batch_idx</code> — TODO.</li>
+<li><code>step_in_epoch</code> — TODO.</li>
+<li><code>batch_num_epoch</code> — TODO.</li>
+<li><code>lr</code> — TODO.</li>
+<li><code>loss</code> — TODO.</li>
+<li><code>speed_stats</code> — TODO.</li>
+<li><code>stats</code> — TODO.</li>
+<li><code>writer</code> — TODO.</li>
+<li><code>tag</code> — TODO.</li>
+<li><code>data_split_i</code> — TODO.</li>
+<li><code>data_split_num</code> — TODO.</li>
+<li><code>log_step</code> — TODO.</li>
+<li><code>**kwargs</code> — Additional keyword arguments.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def log(
+ self,
+ epoch=0,
+ batch_idx=0,
+ step_in_epoch=0,
+ batch_num_epoch=-1,
+ lr=0.0,
+ loss=0.0,
+ speed_stats=None,
+ stats=None,
+ writer=None,
+ tag=&quot;train&quot;,
+ data_split_i=0,
+ data_split_num=1,
+ log_step=None,
+ **kwargs,
+ ):
+
+ &quot;&quot;&quot;Log.
+
+ Args:
+ epoch: TODO.
+ batch_idx: TODO.
+ step_in_epoch: TODO.
+ batch_num_epoch: TODO.
+ lr: TODO.
+ loss: TODO.
+ speed_stats: TODO.
+ stats: TODO.
+ writer: TODO.
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/train_utils/trainer.py#L651" target="_blank" class="src-more">View full source on GitHub →</a></details></div>
+<div class="api-detail" id="e508"><span class="dtag badge-m">method</span><h2><code>Trainer.close(writer)</code></h2><div class="dmod">funasr.train_utils.trainer.Trainer · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/train_utils/trainer.py#L744" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Close.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>writer</code> — TODO.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def close(self, writer=None):
+
+ &quot;&quot;&quot;Close.
+
+ Args:
+ writer: TODO.
+ &quot;&quot;&quot;
+ if self.use_ddp or self.use_fsdp:
+ dist.barrier()
+
+ if writer is not None:
+ writer.close()
+
+ if self.use_ddp or self.use_fsdp:
+ torch.distributed.destroy_process_group()
+</code></pre></details></div>
+<div class="api-detail" id="e509"><span class="dtag badge-f">function</span><h2><code>maybe_autocast(dtype, use_deepspeed)</code></h2><div class="dmod">funasr.train_utils.trainer_ds · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/train_utils/trainer_ds.py#L27" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Maybe autocast.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>dtype</code> — TODO.</li>
+<li><code>use_deepspeed</code> — TODO.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code>def maybe_autocast(dtype=None, use_deepspeed=False):
+ &quot;&quot;&quot;Maybe autocast.
+
+ Args:
+ dtype: TODO.
+ use_deepspeed: TODO.
+ &quot;&quot;&quot;
+ if use_deepspeed:
+ with torch.cuda.amp.autocast(enabled=True, dtype=dtype, cache_enabled=False):
+ yield
+ else:
+ if dtype == torch.float16 or dtype == torch.bfloat16:
+ with autocast(enabled=True, dtype=dtype):
+ yield
+ else:
+ yield
+</code></pre></details></div>
+<div class="api-detail" id="e510"><span class="dtag badge-c">class</span><h2>Trainer</h2><div class="dmod">funasr.train_utils.trainer_ds · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/train_utils/trainer_ds.py#L45" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>A simple trainer class for training a PyTorch model, saving checkpoints at the end of each epoch,</p>
+<p>and optionally resuming from a saved checkpoint.</p>
+<p>Attributes:</p>
+<ul>
+<li><code>max_epoch (int)</code> — Maximum number of epochs for training.</li>
+<li><code>model (torch.nn.Module)</code> — The model to be trained.</li>
+<li><code>optim (torch.optim.Optimizer)</code> — The optimizer to use for training.</li>
+<li><code>scheduler (torch.optim.lr_scheduler._LRScheduler)</code> — The learning rate scheduler.</li>
+<li><code>dataloader_train (torch.utils.data.DataLoader)</code> — DataLoader for the training dataset.</li>
+<li><code>dataloader_val (torch.utils.data.DataLoader)</code> — DataLoader for the validation dataset.</li>
+<li><code>output_dir (str)</code> — Directory where model checkpoints will be saved.</li>
+<li><code>resume (str, optional)</code> — Path to a checkpoint to resume training from.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code>class Trainer:
+ &quot;&quot;&quot;
+ A simple trainer class for training a PyTorch model, saving checkpoints at the end of each epoch,
+ and optionally resuming from a saved checkpoint.
+
+ Attributes:
+ max_epoch (int): Maximum number of epochs for training.
+ model (torch.nn.Module): The model to be trained.
+ optim (torch.optim.Optimizer): The optimizer to use for training.
+ scheduler (torch.optim.lr_scheduler._LRScheduler): The learning rate scheduler.
+ dataloader_train (torch.utils.data.DataLoader): DataLoader for the training dataset.
+ dataloader_val (torch.utils.data.DataLoader): DataLoader for the validation dataset.
+ output_dir (str): Directory where model checkpoints will be saved.
+ resume (str, optional): Path to a checkpoint to resume training from.
+ &quot;&quot;&quot;
+
+ def __init__(
+ self,
+ rank=0,
+ local_rank=0,
+ world_size=1,
+ use_ddp: bool = False,
+ use_fsdp: bool = False,
+ use_fp16: bool = False,
+ use_bf16: bool = False,
+ use_deepspeed: bool = False,
+ output_dir: str = &quot;./&quot;,
+ **kwargs,
+ ):
+ &quot;&quot;&quot;
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/train_utils/trainer_ds.py#L45" target="_blank" class="src-more">View full source on GitHub →</a></details><h3 class="mt">Methods</h3><div class="mblk"><code>.save_checkpoint(epoch, step, model, optim, scheduler, scaler, step_in_epoch, **kwargs)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/train_utils/trainer_ds.py#L171" target="_blank" class="gh-link">L171</a><div class="mdoc"><p>Saves a checkpoint containing the model&#x27;s state, the optimizer&#x27;s state,</p>
+<p>and the scheduler&#x27;s state at the end of the given epoch. This method is</p>
+<p>intended to be called at the end of each epoch to save the training progress.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>epoch (int)</code> — The epoch number at which the checkpoint is being saved.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def save_checkpoint(
+ self,
+ epoch,
+ step=None,
+ model=None,
+ optim=None,
+ scheduler=None,
+ scaler=None,
+ step_in_epoch=None,
+ **kwargs,
+ ):
+ &quot;&quot;&quot;
+ Saves a checkpoint containing the model&#x27;s state, the optimizer&#x27;s state,
+ and the scheduler&#x27;s state at the end of the given epoch. This method is
+ intended to be called at the end of each epoch to save the training progress.
+
+ Args:
+ epoch (int): The epoch number at which the checkpoint is being saved.
+ &quot;&quot;&quot;
+ if self.use_ddp or self.use_fsdp:
+ dist.barrier()
+ step_in_epoch = None if step is None else step_in_epoch
+ if self.use_deepspeed:
+
+ logging.info(f&quot;Save checkpoint: {epoch}, rank: {self.local_rank}\n&quot;)
+ # self.step_or_epoch += 1
+ state = {
+ &quot;epoch&quot;: epoch,
+ # &quot;state_dict&quot;: model.state_dict(),
+ # &quot;optimizer&quot;: optim.state_dict(),
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/train_utils/trainer_ds.py#L171" target="_blank" class="src-more">View full source →</a></details></div><div class="mblk"><code>.resume_checkpoint(model, optim, scheduler, scaler)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/train_utils/trainer_ds.py#L414" target="_blank" class="gh-link">L414</a><div class="mdoc"><p>Resumes training from a checkpoint at the given file path.</p>
+<p>Loads the model&#x27;s state, the optimizer&#x27;s state, and the scheduler&#x27;s state.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>resume_path (str)</code> — The file path to the checkpoint to resume from.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def resume_checkpoint(
+ self,
+ model=None,
+ optim=None,
+ scheduler=None,
+ scaler=None,
+ ):
+ &quot;&quot;&quot;
+ Resumes training from a checkpoint at the given file path.
+ Loads the model&#x27;s state, the optimizer&#x27;s state, and the scheduler&#x27;s state.
+
+ Args:
+ resume_path (str): The file path to the checkpoint to resume from.
+ &quot;&quot;&quot;
+ if self.resume:
+
+ if self.use_deepspeed:
+ ckpt = os.path.join(self.output_dir, &quot;model.pt&quot;)
+ if os.path.exists(ckpt):
+ _, checkpoint = model.load_checkpoint(self.output_dir, &quot;model.pt&quot;)
+ self.start_epoch = checkpoint[&quot;epoch&quot;]
+ self.saved_ckpts = checkpoint[&quot;saved_ckpts&quot;]
+ self.val_acc_step_or_epoch = (
+ checkpoint[&quot;val_acc_step_or_epoch&quot;]
+ if &quot;val_acc_step_or_epoch&quot; in checkpoint
+ else {}
+ )
+ self.val_loss_step_or_epoch = (
+ checkpoint[&quot;val_loss_step_or_epoch&quot;]
+ if &quot;val_loss_step_or_epoch&quot; in checkpoint
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/train_utils/trainer_ds.py#L414" target="_blank" class="src-more">View full source →</a></details></div><div class="mblk"><code>.train_epoch(model, optim, scheduler, scaler, dataloader_train, dataloader_val, epoch, **kwargs)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/train_utils/trainer_ds.py#L554" target="_blank" class="gh-link">L554</a><div class="mdoc"><p>Defines the training process for a single epoch with gradient accumulation.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>epoch (int)</code> — The current epoch number.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def train_epoch(
+ self,
+ model=None,
+ optim=None,
+ scheduler=None,
+ scaler=None,
+ dataloader_train=None,
+ dataloader_val=None,
+ epoch=None,
+ **kwargs,
+ ):
+ &quot;&quot;&quot;
+ Defines the training process for a single epoch with gradient accumulation.
+ Args:
+ epoch (int): The current epoch number.
+ &quot;&quot;&quot;
+ if self.use_ddp or self.use_fsdp or self.use_deepspeed:
+ dist.barrier()
+ logging.info(f&quot;Train epoch: {epoch}, rank: {self.rank}\n&quot;)
+ model.train()
+
+ # Set the number of steps for gradient accumulation
+ accum_grad = self.accum_grad
+ # Initialize the gradient accumulation
+ optim.zero_grad()
+ speed_stats = {}
+
+ iterator_stop = torch.tensor(0).to(self.device)
+
+ dataloader_train.batch_sampler.set_epoch(epoch)
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/train_utils/trainer_ds.py#L554" target="_blank" class="src-more">View full source →</a></details></div><div class="mblk"><code>.forward_step(model, batch, loss_dict)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/train_utils/trainer_ds.py#L676" target="_blank" class="gh-link">L676</a><div class="mdoc"><p>Forward step.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>model</code> — Model instance or model name.</li>
+<li><code>batch</code> — TODO.</li>
+<li><code>loss_dict</code> — TODO.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def forward_step(self, model, batch, loss_dict={}):
+ &quot;&quot;&quot;Forward step.
+
+ Args:
+ model: Model instance or model name.
+ batch: TODO.
+ loss_dict: TODO.
+ &quot;&quot;&quot;
+ with maybe_autocast(dtype=self.dtype, use_deepspeed=self.use_deepspeed):
+ retval = model(**batch)
+
+ loss, stats, weight = retval
+ stats = {k: v for k, v in stats.items() if v is not None}
+
+ loss_dict[&quot;loss&quot;] = loss
+ loss_dict[&quot;stats&quot;] = stats
+ loss_dict[&quot;weight&quot;] = weight
+</code></pre></details></div><div class="mblk"><code>.backward_step(model, scaler, loss_dict)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/train_utils/trainer_ds.py#L694" target="_blank" class="gh-link">L694</a><div class="mdoc"><p>Backward step.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>model</code> — Model instance or model name.</li>
+<li><code>scaler</code> — TODO.</li>
+<li><code>loss_dict</code> — TODO.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def backward_step(self, model, scaler, loss_dict={}):
+ &quot;&quot;&quot;Backward step.
+
+ Args:
+ model: Model instance or model name.
+ scaler: TODO.
+ loss_dict: TODO.
+ &quot;&quot;&quot;
+ loss = loss_dict[&quot;loss&quot;]
+
+ if self.use_deepspeed:
+ scaled_loss = model.backward(loss)
+ else:
+ loss = loss / self.accum_grad
+ if scaler:
+ scaler.scale(loss).backward()
+ else:
+ loss.backward()
+</code></pre></details></div><div class="mblk"><code>.update_step(model, optim, scheduler, scaler, loss_dict)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/train_utils/trainer_ds.py#L713" target="_blank" class="gh-link">L713</a><div class="mdoc"><p>Update step.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>model</code> — Model instance or model name.</li>
+<li><code>optim</code> — TODO.</li>
+<li><code>scheduler</code> — TODO.</li>
+<li><code>scaler</code> — TODO.</li>
+<li><code>loss_dict</code> — TODO.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def update_step(self, model, optim, scheduler, scaler, loss_dict=None):
+ &quot;&quot;&quot;Update step.
+
+ Args:
+ model: Model instance or model name.
+ optim: TODO.
+ scheduler: TODO.
+ scaler: TODO.
+ loss_dict: TODO.
+ &quot;&quot;&quot;
+ batch_idx = loss_dict[&quot;batch_idx&quot;]
+ if self.use_deepspeed:
+ model.step()
+ else:
+ if (batch_idx + 1) % self.accum_grad == 0:
+ # Perform gradient clipping if it is set
+ if self.grad_clip &gt; 0:
+ grad_norm = torch.nn.utils.clip_grad_norm_(
+ model.parameters(),
+ max_norm=self.grad_clip,
+ norm_type=self.grad_clip_type,
+ )
+ if not torch.isfinite(grad_norm):
+ logging.warning(
+ f&quot;The grad norm is {grad_norm}. Skipping updating the model.&quot;
+ )
+ optim.zero_grad() # Reset gradients
+ return
+
+ # Execute an optimization step (update model parameters)
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/train_utils/trainer_ds.py#L713" target="_blank" class="src-more">View full source →</a></details></div><div class="mblk"><code>.validate_epoch(model, dataloader_val, epoch, writer, **kwargs)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/train_utils/trainer_ds.py#L754" target="_blank" class="gh-link">L754</a><div class="mdoc"><p>Defines the validation process for a single epoch.</p>
+<p>Should be implemented with the actual model validation steps.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>epoch (int)</code> — The current epoch number.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def validate_epoch(
+ self,
+ model=None,
+ dataloader_val=None,
+ epoch=None,
+ writer=None,
+ **kwargs,
+ ):
+ &quot;&quot;&quot;
+ Defines the validation process for a single epoch.
+ Should be implemented with the actual model validation steps.
+
+ Args:
+ epoch (int): The current epoch number.
+ &quot;&quot;&quot;
+ self.val_loss_avg = 0.0
+ self.val_acc_avg = 0.0
+
+ if self.use_ddp or self.use_fsdp or self.use_deepspeed:
+ dist.barrier()
+ logging.info(f&quot;Validate epoch: {epoch}, rank: {self.rank}\n&quot;)
+ model.eval()
+
+ with torch.no_grad():
+
+ speed_stats = {}
+ time_beg = time.perf_counter()
+ time5 = time_beg
+
+ dataloader_val.batch_sampler.set_epoch(epoch)
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/train_utils/trainer_ds.py#L754" target="_blank" class="src-more">View full source →</a></details></div><div class="mblk"><code>.log(loss_dict, tag, **kwargs)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/train_utils/trainer_ds.py#L848" target="_blank" class="gh-link">L848</a><div class="mdoc"><p>Log.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>loss_dict</code> — TODO.</li>
+<li><code>tag</code> — TODO.</li>
+<li><code>**kwargs</code> — Additional keyword arguments.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def log(
+ self,
+ loss_dict: dict = None,
+ tag=&quot;train&quot;,
+ **kwargs,
+ ):
+ &quot;&quot;&quot;Log.
+
+ Args:
+ loss_dict: TODO.
+ tag: TODO.
+ **kwargs: Additional keyword arguments.
+ &quot;&quot;&quot;
+ loss = loss_dict[&quot;loss&quot;].detach().cpu().item()
+ epoch = loss_dict[&quot;epoch&quot;]
+ batch_idx = loss_dict[&quot;batch_idx&quot;]
+ step_in_epoch = loss_dict[&quot;step_in_epoch&quot;]
+ batch_total = loss_dict[&quot;batch_total&quot;]
+ batch_num_epoch = loss_dict[&quot;batch_num_epoch&quot;]
+ lr = loss_dict[&quot;lr&quot;]
+
+ speed_stats = loss_dict[&quot;speed_stats&quot;]
+ stats = loss_dict[&quot;stats&quot;]
+ data_split_i = loss_dict[&quot;data_split_i&quot;]
+ data_split_num = loss_dict[&quot;data_split_num&quot;]
+ log_step = loss_dict.get(&quot;log_step&quot;, None)
+
+ if (batch_idx + 1) % self.log_interval == 0:
+ batch_idx = log_step if log_step is not None else batch_idx
+ gpu_info = (
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/train_utils/trainer_ds.py#L848" target="_blank" class="src-more">View full source →</a></details></div><div class="mblk"><code>.close(writer)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/train_utils/trainer_ds.py#L929" target="_blank" class="gh-link">L929</a><div class="mdoc"><p>Close.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>writer</code> — TODO.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def close(self, writer=None):
+
+ &quot;&quot;&quot;Close.
+
+ Args:
+ writer: TODO.
+ &quot;&quot;&quot;
+ if self.use_ddp or self.use_fsdp:
+ dist.barrier()
+
+ if writer is not None:
+ writer.close()
+
+ if self.use_ddp or self.use_fsdp:
+ torch.distributed.destroy_process_group()
+</code></pre></details></div><div class="mblk"><code>.warp_model(model, **kwargs)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/train_utils/trainer_ds.py#L945" target="_blank" class="gh-link">L945</a><div class="mdoc"><p>Warp model.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>model</code> — Model instance or model name.</li>
+<li><code>**kwargs</code> — Additional keyword arguments.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def warp_model(self, model, **kwargs):
+
+ &quot;&quot;&quot;Warp model.
+
+ Args:
+ model: Model instance or model name.
+ **kwargs: Additional keyword arguments.
+ &quot;&quot;&quot;
+ if self.use_deepspeed:
+ from deepspeed.runtime.zero.stage_1_and_2 import (
+ estimate_zero2_model_states_mem_needs_all_live,
+ )
+ from deepspeed.runtime.zero.stage3 import estimate_zero3_model_states_mem_needs_all_live
+ from deepspeed.utils.zero_to_fp32 import convert_zero_checkpoint_to_fp32_state_dict
+
+ local_world_size = int(os.environ.get(&quot;LOCAL_WORLD_SIZE&quot;, 1))
+ world_size = int(os.environ.get(&quot;WORLD_SIZE&quot;, 1))
+
+ # NOTE(xcsong): look in detail how the memory estimator API works:
+ # https://deepspeed.readthedocs.io/en/latest/memory.html#discussion
+ if int(os.environ.get(&quot;RANK&quot;, 0)) == 0:
+ logging.info(&quot;Estimating model states memory needs (zero2)...&quot;)
+ estimate_zero2_model_states_mem_needs_all_live(
+ model,
+ num_gpus_per_node=local_world_size,
+ num_nodes=world_size // local_world_size,
+ )
+ logging.info(&quot;Estimating model states memory needs (zero3)...&quot;)
+ estimate_zero3_model_states_mem_needs_all_live(
+ model,
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/train_utils/trainer_ds.py#L945" target="_blank" class="src-more">View full source →</a></details></div><div class="mblk"><code>.warp_optim_scheduler(model, **kwargs)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/train_utils/trainer_ds.py#L997" target="_blank" class="gh-link">L997</a><div class="mdoc"><p>Warp optim scheduler.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>model</code> — Model instance or model name.</li>
+<li><code>**kwargs</code> — Additional keyword arguments.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def warp_optim_scheduler(self, model, **kwargs):
+ &quot;&quot;&quot;Warp optim scheduler.
+
+ Args:
+ model: Model instance or model name.
+ **kwargs: Additional keyword arguments.
+ &quot;&quot;&quot;
+ from funasr.optimizers import optim_classes
+ from funasr.schedulers import scheduler_classes
+ from omegaconf import OmegaConf, DictConfig
+ import json
+
+ # optim
+ logging.info(&quot;Build optim&quot;)
+ optim = kwargs.get(&quot;optim&quot;, &quot;adam&quot;)
+ assert optim in optim_classes
+ optim_class = optim_classes.get(optim)
+ optim = optim_class(model.parameters(), **kwargs.get(&quot;optim_conf&quot;))
+
+ # scheduler
+ logging.info(&quot;Build scheduler&quot;)
+ scheduler = kwargs.get(&quot;scheduler&quot;, &quot;warmuplr&quot;)
+ assert scheduler in scheduler_classes
+ scheduler_class = scheduler_classes.get(scheduler)
+ scheduler = scheduler_class(optim, **kwargs.get(&quot;scheduler_conf&quot;))
+
+ if self.use_deepspeed:
+ import deepspeed
+
+ args = OmegaConf.create({&quot;deepspeed_config&quot;: self.deepspeed_config})
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/train_utils/trainer_ds.py#L997" target="_blank" class="src-more">View full source →</a></details></div></div>
+<div class="api-detail" id="e511"><span class="dtag badge-m">method</span><h2><code>Trainer.save_checkpoint(epoch, step, model, optim, scheduler, scaler, step_in_epoch, **kwargs)</code></h2><div class="dmod">funasr.train_utils.trainer_ds.Trainer · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/train_utils/trainer_ds.py#L171" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Saves a checkpoint containing the model&#x27;s state, the optimizer&#x27;s state,</p>
+<p>and the scheduler&#x27;s state at the end of the given epoch. This method is</p>
+<p>intended to be called at the end of each epoch to save the training progress.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>epoch (int)</code> — The epoch number at which the checkpoint is being saved.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def save_checkpoint(
+ self,
+ epoch,
+ step=None,
+ model=None,
+ optim=None,
+ scheduler=None,
+ scaler=None,
+ step_in_epoch=None,
+ **kwargs,
+ ):
+ &quot;&quot;&quot;
+ Saves a checkpoint containing the model&#x27;s state, the optimizer&#x27;s state,
+ and the scheduler&#x27;s state at the end of the given epoch. This method is
+ intended to be called at the end of each epoch to save the training progress.
+
+ Args:
+ epoch (int): The epoch number at which the checkpoint is being saved.
+ &quot;&quot;&quot;
+ if self.use_ddp or self.use_fsdp:
+ dist.barrier()
+ step_in_epoch = None if step is None else step_in_epoch
+ if self.use_deepspeed:
+
+ logging.info(f&quot;Save checkpoint: {epoch}, rank: {self.local_rank}\n&quot;)
+ # self.step_or_epoch += 1
+ state = {
+ &quot;epoch&quot;: epoch,
+ # &quot;state_dict&quot;: model.state_dict(),
+ # &quot;optimizer&quot;: optim.state_dict(),
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/train_utils/trainer_ds.py#L171" target="_blank" class="src-more">View full source on GitHub →</a></details></div>
+<div class="api-detail" id="e512"><span class="dtag badge-m">method</span><h2><code>Trainer.resume_checkpoint(model, optim, scheduler, scaler)</code></h2><div class="dmod">funasr.train_utils.trainer_ds.Trainer · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/train_utils/trainer_ds.py#L414" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Resumes training from a checkpoint at the given file path.</p>
+<p>Loads the model&#x27;s state, the optimizer&#x27;s state, and the scheduler&#x27;s state.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>resume_path (str)</code> — The file path to the checkpoint to resume from.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def resume_checkpoint(
+ self,
+ model=None,
+ optim=None,
+ scheduler=None,
+ scaler=None,
+ ):
+ &quot;&quot;&quot;
+ Resumes training from a checkpoint at the given file path.
+ Loads the model&#x27;s state, the optimizer&#x27;s state, and the scheduler&#x27;s state.
+
+ Args:
+ resume_path (str): The file path to the checkpoint to resume from.
+ &quot;&quot;&quot;
+ if self.resume:
+
+ if self.use_deepspeed:
+ ckpt = os.path.join(self.output_dir, &quot;model.pt&quot;)
+ if os.path.exists(ckpt):
+ _, checkpoint = model.load_checkpoint(self.output_dir, &quot;model.pt&quot;)
+ self.start_epoch = checkpoint[&quot;epoch&quot;]
+ self.saved_ckpts = checkpoint[&quot;saved_ckpts&quot;]
+ self.val_acc_step_or_epoch = (
+ checkpoint[&quot;val_acc_step_or_epoch&quot;]
+ if &quot;val_acc_step_or_epoch&quot; in checkpoint
+ else {}
+ )
+ self.val_loss_step_or_epoch = (
+ checkpoint[&quot;val_loss_step_or_epoch&quot;]
+ if &quot;val_loss_step_or_epoch&quot; in checkpoint
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/train_utils/trainer_ds.py#L414" target="_blank" class="src-more">View full source on GitHub →</a></details></div>
+<div class="api-detail" id="e513"><span class="dtag badge-m">method</span><h2><code>Trainer.train_epoch(model, optim, scheduler, scaler, dataloader_train, dataloader_val, epoch, **kwargs)</code></h2><div class="dmod">funasr.train_utils.trainer_ds.Trainer · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/train_utils/trainer_ds.py#L554" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Defines the training process for a single epoch with gradient accumulation.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>epoch (int)</code> — The current epoch number.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def train_epoch(
+ self,
+ model=None,
+ optim=None,
+ scheduler=None,
+ scaler=None,
+ dataloader_train=None,
+ dataloader_val=None,
+ epoch=None,
+ **kwargs,
+ ):
+ &quot;&quot;&quot;
+ Defines the training process for a single epoch with gradient accumulation.
+ Args:
+ epoch (int): The current epoch number.
+ &quot;&quot;&quot;
+ if self.use_ddp or self.use_fsdp or self.use_deepspeed:
+ dist.barrier()
+ logging.info(f&quot;Train epoch: {epoch}, rank: {self.rank}\n&quot;)
+ model.train()
+
+ # Set the number of steps for gradient accumulation
+ accum_grad = self.accum_grad
+ # Initialize the gradient accumulation
+ optim.zero_grad()
+ speed_stats = {}
+
+ iterator_stop = torch.tensor(0).to(self.device)
+
+ dataloader_train.batch_sampler.set_epoch(epoch)
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/train_utils/trainer_ds.py#L554" target="_blank" class="src-more">View full source on GitHub →</a></details></div>
+<div class="api-detail" id="e514"><span class="dtag badge-m">method</span><h2><code>Trainer.forward_step(model, batch, loss_dict)</code></h2><div class="dmod">funasr.train_utils.trainer_ds.Trainer · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/train_utils/trainer_ds.py#L676" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Forward step.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>model</code> — Model instance or model name.</li>
+<li><code>batch</code> — TODO.</li>
+<li><code>loss_dict</code> — TODO.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def forward_step(self, model, batch, loss_dict={}):
+ &quot;&quot;&quot;Forward step.
+
+ Args:
+ model: Model instance or model name.
+ batch: TODO.
+ loss_dict: TODO.
+ &quot;&quot;&quot;
+ with maybe_autocast(dtype=self.dtype, use_deepspeed=self.use_deepspeed):
+ retval = model(**batch)
+
+ loss, stats, weight = retval
+ stats = {k: v for k, v in stats.items() if v is not None}
+
+ loss_dict[&quot;loss&quot;] = loss
+ loss_dict[&quot;stats&quot;] = stats
+ loss_dict[&quot;weight&quot;] = weight
+</code></pre></details></div>
+<div class="api-detail" id="e515"><span class="dtag badge-m">method</span><h2><code>Trainer.backward_step(model, scaler, loss_dict)</code></h2><div class="dmod">funasr.train_utils.trainer_ds.Trainer · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/train_utils/trainer_ds.py#L694" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Backward step.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>model</code> — Model instance or model name.</li>
+<li><code>scaler</code> — TODO.</li>
+<li><code>loss_dict</code> — TODO.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def backward_step(self, model, scaler, loss_dict={}):
+ &quot;&quot;&quot;Backward step.
+
+ Args:
+ model: Model instance or model name.
+ scaler: TODO.
+ loss_dict: TODO.
+ &quot;&quot;&quot;
+ loss = loss_dict[&quot;loss&quot;]
+
+ if self.use_deepspeed:
+ scaled_loss = model.backward(loss)
+ else:
+ loss = loss / self.accum_grad
+ if scaler:
+ scaler.scale(loss).backward()
+ else:
+ loss.backward()
+</code></pre></details></div>
+<div class="api-detail" id="e516"><span class="dtag badge-m">method</span><h2><code>Trainer.update_step(model, optim, scheduler, scaler, loss_dict)</code></h2><div class="dmod">funasr.train_utils.trainer_ds.Trainer · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/train_utils/trainer_ds.py#L713" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Update step.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>model</code> — Model instance or model name.</li>
+<li><code>optim</code> — TODO.</li>
+<li><code>scheduler</code> — TODO.</li>
+<li><code>scaler</code> — TODO.</li>
+<li><code>loss_dict</code> — TODO.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def update_step(self, model, optim, scheduler, scaler, loss_dict=None):
+ &quot;&quot;&quot;Update step.
+
+ Args:
+ model: Model instance or model name.
+ optim: TODO.
+ scheduler: TODO.
+ scaler: TODO.
+ loss_dict: TODO.
+ &quot;&quot;&quot;
+ batch_idx = loss_dict[&quot;batch_idx&quot;]
+ if self.use_deepspeed:
+ model.step()
+ else:
+ if (batch_idx + 1) % self.accum_grad == 0:
+ # Perform gradient clipping if it is set
+ if self.grad_clip &gt; 0:
+ grad_norm = torch.nn.utils.clip_grad_norm_(
+ model.parameters(),
+ max_norm=self.grad_clip,
+ norm_type=self.grad_clip_type,
+ )
+ if not torch.isfinite(grad_norm):
+ logging.warning(
+ f&quot;The grad norm is {grad_norm}. Skipping updating the model.&quot;
+ )
+ optim.zero_grad() # Reset gradients
+ return
+
+ # Execute an optimization step (update model parameters)
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/train_utils/trainer_ds.py#L713" target="_blank" class="src-more">View full source on GitHub →</a></details></div>
+<div class="api-detail" id="e517"><span class="dtag badge-m">method</span><h2><code>Trainer.validate_epoch(model, dataloader_val, epoch, writer, **kwargs)</code></h2><div class="dmod">funasr.train_utils.trainer_ds.Trainer · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/train_utils/trainer_ds.py#L754" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Defines the validation process for a single epoch.</p>
+<p>Should be implemented with the actual model validation steps.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>epoch (int)</code> — The current epoch number.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def validate_epoch(
+ self,
+ model=None,
+ dataloader_val=None,
+ epoch=None,
+ writer=None,
+ **kwargs,
+ ):
+ &quot;&quot;&quot;
+ Defines the validation process for a single epoch.
+ Should be implemented with the actual model validation steps.
+
+ Args:
+ epoch (int): The current epoch number.
+ &quot;&quot;&quot;
+ self.val_loss_avg = 0.0
+ self.val_acc_avg = 0.0
+
+ if self.use_ddp or self.use_fsdp or self.use_deepspeed:
+ dist.barrier()
+ logging.info(f&quot;Validate epoch: {epoch}, rank: {self.rank}\n&quot;)
+ model.eval()
+
+ with torch.no_grad():
+
+ speed_stats = {}
+ time_beg = time.perf_counter()
+ time5 = time_beg
+
+ dataloader_val.batch_sampler.set_epoch(epoch)
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/train_utils/trainer_ds.py#L754" target="_blank" class="src-more">View full source on GitHub →</a></details></div>
+<div class="api-detail" id="e518"><span class="dtag badge-m">method</span><h2><code>Trainer.log(loss_dict, tag, **kwargs)</code></h2><div class="dmod">funasr.train_utils.trainer_ds.Trainer · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/train_utils/trainer_ds.py#L848" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Log.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>loss_dict</code> — TODO.</li>
+<li><code>tag</code> — TODO.</li>
+<li><code>**kwargs</code> — Additional keyword arguments.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def log(
+ self,
+ loss_dict: dict = None,
+ tag=&quot;train&quot;,
+ **kwargs,
+ ):
+ &quot;&quot;&quot;Log.
+
+ Args:
+ loss_dict: TODO.
+ tag: TODO.
+ **kwargs: Additional keyword arguments.
+ &quot;&quot;&quot;
+ loss = loss_dict[&quot;loss&quot;].detach().cpu().item()
+ epoch = loss_dict[&quot;epoch&quot;]
+ batch_idx = loss_dict[&quot;batch_idx&quot;]
+ step_in_epoch = loss_dict[&quot;step_in_epoch&quot;]
+ batch_total = loss_dict[&quot;batch_total&quot;]
+ batch_num_epoch = loss_dict[&quot;batch_num_epoch&quot;]
+ lr = loss_dict[&quot;lr&quot;]
+
+ speed_stats = loss_dict[&quot;speed_stats&quot;]
+ stats = loss_dict[&quot;stats&quot;]
+ data_split_i = loss_dict[&quot;data_split_i&quot;]
+ data_split_num = loss_dict[&quot;data_split_num&quot;]
+ log_step = loss_dict.get(&quot;log_step&quot;, None)
+
+ if (batch_idx + 1) % self.log_interval == 0:
+ batch_idx = log_step if log_step is not None else batch_idx
+ gpu_info = (
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/train_utils/trainer_ds.py#L848" target="_blank" class="src-more">View full source on GitHub →</a></details></div>
+<div class="api-detail" id="e519"><span class="dtag badge-m">method</span><h2><code>Trainer.close(writer)</code></h2><div class="dmod">funasr.train_utils.trainer_ds.Trainer · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/train_utils/trainer_ds.py#L929" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Close.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>writer</code> — TODO.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def close(self, writer=None):
+
+ &quot;&quot;&quot;Close.
+
+ Args:
+ writer: TODO.
+ &quot;&quot;&quot;
+ if self.use_ddp or self.use_fsdp:
+ dist.barrier()
+
+ if writer is not None:
+ writer.close()
+
+ if self.use_ddp or self.use_fsdp:
+ torch.distributed.destroy_process_group()
+</code></pre></details></div>
+<div class="api-detail" id="e520"><span class="dtag badge-m">method</span><h2><code>Trainer.warp_model(model, **kwargs)</code></h2><div class="dmod">funasr.train_utils.trainer_ds.Trainer · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/train_utils/trainer_ds.py#L945" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Warp model.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>model</code> — Model instance or model name.</li>
+<li><code>**kwargs</code> — Additional keyword arguments.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def warp_model(self, model, **kwargs):
+
+ &quot;&quot;&quot;Warp model.
+
+ Args:
+ model: Model instance or model name.
+ **kwargs: Additional keyword arguments.
+ &quot;&quot;&quot;
+ if self.use_deepspeed:
+ from deepspeed.runtime.zero.stage_1_and_2 import (
+ estimate_zero2_model_states_mem_needs_all_live,
+ )
+ from deepspeed.runtime.zero.stage3 import estimate_zero3_model_states_mem_needs_all_live
+ from deepspeed.utils.zero_to_fp32 import convert_zero_checkpoint_to_fp32_state_dict
+
+ local_world_size = int(os.environ.get(&quot;LOCAL_WORLD_SIZE&quot;, 1))
+ world_size = int(os.environ.get(&quot;WORLD_SIZE&quot;, 1))
+
+ # NOTE(xcsong): look in detail how the memory estimator API works:
+ # https://deepspeed.readthedocs.io/en/latest/memory.html#discussion
+ if int(os.environ.get(&quot;RANK&quot;, 0)) == 0:
+ logging.info(&quot;Estimating model states memory needs (zero2)...&quot;)
+ estimate_zero2_model_states_mem_needs_all_live(
+ model,
+ num_gpus_per_node=local_world_size,
+ num_nodes=world_size // local_world_size,
+ )
+ logging.info(&quot;Estimating model states memory needs (zero3)...&quot;)
+ estimate_zero3_model_states_mem_needs_all_live(
+ model,
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/train_utils/trainer_ds.py#L945" target="_blank" class="src-more">View full source on GitHub →</a></details></div>
+<div class="api-detail" id="e521"><span class="dtag badge-m">method</span><h2><code>Trainer.warp_optim_scheduler(model, **kwargs)</code></h2><div class="dmod">funasr.train_utils.trainer_ds.Trainer · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/train_utils/trainer_ds.py#L997" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Warp optim scheduler.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>model</code> — Model instance or model name.</li>
+<li><code>**kwargs</code> — Additional keyword arguments.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def warp_optim_scheduler(self, model, **kwargs):
+ &quot;&quot;&quot;Warp optim scheduler.
+
+ Args:
+ model: Model instance or model name.
+ **kwargs: Additional keyword arguments.
+ &quot;&quot;&quot;
+ from funasr.optimizers import optim_classes
+ from funasr.schedulers import scheduler_classes
+ from omegaconf import OmegaConf, DictConfig
+ import json
+
+ # optim
+ logging.info(&quot;Build optim&quot;)
+ optim = kwargs.get(&quot;optim&quot;, &quot;adam&quot;)
+ assert optim in optim_classes
+ optim_class = optim_classes.get(optim)
+ optim = optim_class(model.parameters(), **kwargs.get(&quot;optim_conf&quot;))
+
+ # scheduler
+ logging.info(&quot;Build scheduler&quot;)
+ scheduler = kwargs.get(&quot;scheduler&quot;, &quot;warmuplr&quot;)
+ assert scheduler in scheduler_classes
+ scheduler_class = scheduler_classes.get(scheduler)
+ scheduler = scheduler_class(optim, **kwargs.get(&quot;scheduler_conf&quot;))
+
+ if self.use_deepspeed:
+ import deepspeed
+
+ args = OmegaConf.create({&quot;deepspeed_config&quot;: self.deepspeed_config})
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/train_utils/trainer_ds.py#L997" target="_blank" class="src-more">View full source on GitHub →</a></details></div>
+<div class="api-detail" id="e522"><span class="dtag badge-c">class</span><h2>AudioDataset</h2><div class="dmod">funasr.datasets.datasets · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/datasets/audio_datasets/datasets.py#L10" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>AudioDataset</p>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code>class AudioDataset(torch.utils.data.Dataset):
+ &quot;&quot;&quot;
+ AudioDataset
+ &quot;&quot;&quot;
+
+ def __init__(
+ self,
+ path,
+ index_ds: str = None,
+ frontend=None,
+ tokenizer=None,
+ is_training: bool = True,
+ int_pad_value: int = -1,
+ float_pad_value: float = 0.0,
+ **kwargs,
+ ):
+ &quot;&quot;&quot;Initialize AudioDataset.
+
+ Args:
+ path: TODO.
+ index_ds: TODO.
+ frontend: Audio frontend for feature extraction.
+ tokenizer: Tokenizer instance for text encoding/decoding.
+ is_training: Boolean flag for training.
+ int_pad_value: TODO.
+ float_pad_value: TODO.
+ **kwargs: Additional keyword arguments.
+ &quot;&quot;&quot;
+ super().__init__()
+ index_ds_class = tables.index_ds_classes.get(index_ds)
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/datasets/audio_datasets/datasets.py#L10" target="_blank" class="src-more">View full source on GitHub →</a></details><h3 class="mt">Methods</h3><div class="mblk"><code>.get_source_len(index)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/datasets/audio_datasets/datasets.py#L67" target="_blank" class="gh-link">L67</a><div class="mdoc"><p>Get source len.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>index</code> — TODO.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def get_source_len(self, index):
+ &quot;&quot;&quot;Get source len.
+
+ Args:
+ index: TODO.
+ &quot;&quot;&quot;
+ item = self.index_ds[index]
+ return self.index_ds.get_source_len(item)
+</code></pre></details></div><div class="mblk"><code>.get_target_len(index)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/datasets/audio_datasets/datasets.py#L76" target="_blank" class="gh-link">L76</a><div class="mdoc"><p>Get target len.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>index</code> — TODO.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def get_target_len(self, index):
+ &quot;&quot;&quot;Get target len.
+
+ Args:
+ index: TODO.
+ &quot;&quot;&quot;
+ item = self.index_ds[index]
+ return self.index_ds.get_target_len(item)
+</code></pre></details></div><div class="mblk"><code>.collator(samples)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/datasets/audio_datasets/datasets.py#L127" target="_blank" class="gh-link">L127</a><div class="mdoc"><p>Collator.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>samples</code> — TODO.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def collator(self, samples: list = None):
+ &quot;&quot;&quot;Collator.
+
+ Args:
+ samples: TODO.
+ &quot;&quot;&quot;
+ outputs = {}
+ for sample in samples:
+ for key in sample.keys():
+ if key not in outputs:
+ outputs[key] = []
+ outputs[key].append(sample[key])
+
+ for key, data_list in outputs.items():
+ if isinstance(data_list[0], torch.Tensor):
+ if data_list[0].dtype == torch.int64 or data_list[0].dtype == torch.int32:
+
+ pad_value = self.int_pad_value
+ else:
+ pad_value = self.float_pad_value
+
+ outputs[key] = torch.nn.utils.rnn.pad_sequence(
+ data_list, batch_first=True, padding_value=pad_value
+ )
+ return outputs
+</code></pre></details></div></div>
+<div class="api-detail" id="e523"><span class="dtag badge-m">method</span><h2><code>AudioDataset.get_source_len(index)</code></h2><div class="dmod">funasr.datasets.datasets.AudioDataset · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/datasets/audio_datasets/datasets.py#L67" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Get source len.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>index</code> — TODO.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def get_source_len(self, index):
+ &quot;&quot;&quot;Get source len.
+
+ Args:
+ index: TODO.
+ &quot;&quot;&quot;
+ item = self.index_ds[index]
+ return self.index_ds.get_source_len(item)
+</code></pre></details></div>
+<div class="api-detail" id="e524"><span class="dtag badge-m">method</span><h2><code>AudioDataset.get_target_len(index)</code></h2><div class="dmod">funasr.datasets.datasets.AudioDataset · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/datasets/audio_datasets/datasets.py#L76" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Get target len.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>index</code> — TODO.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def get_target_len(self, index):
+ &quot;&quot;&quot;Get target len.
+
+ Args:
+ index: TODO.
+ &quot;&quot;&quot;
+ item = self.index_ds[index]
+ return self.index_ds.get_target_len(item)
+</code></pre></details></div>
+<div class="api-detail" id="e525"><span class="dtag badge-m">method</span><h2><code>AudioDataset.collator(samples)</code></h2><div class="dmod">funasr.datasets.datasets.AudioDataset · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/datasets/audio_datasets/datasets.py#L127" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Collator.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>samples</code> — TODO.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def collator(self, samples: list = None):
+ &quot;&quot;&quot;Collator.
+
+ Args:
+ samples: TODO.
+ &quot;&quot;&quot;
+ outputs = {}
+ for sample in samples:
+ for key in sample.keys():
+ if key not in outputs:
+ outputs[key] = []
+ outputs[key].append(sample[key])
+
+ for key, data_list in outputs.items():
+ if isinstance(data_list[0], torch.Tensor):
+ if data_list[0].dtype == torch.int64 or data_list[0].dtype == torch.int32:
+
+ pad_value = self.int_pad_value
+ else:
+ pad_value = self.float_pad_value
+
+ outputs[key] = torch.nn.utils.rnn.pad_sequence(
+ data_list, batch_first=True, padding_value=pad_value
+ )
+ return outputs
+</code></pre></details></div>
+<div class="api-detail" id="e526"><span class="dtag badge-c">class</span><h2>AudioDatasetHotword</h2><div class="dmod">funasr.datasets.datasets · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/datasets/audio_datasets/datasets.py#L155" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p class="muted">No documentation yet.</p></div><details class="src-block"><summary>📄 Source code</summary><pre><code>class AudioDatasetHotword(AudioDataset):
+ # for finetuning contextual_paraformer and seaco_paraformer
+ def __init__(
+ self,
+ *args,
+ seaco_id: bool = 0,
+ **kwargs,
+ ):
+ &quot;&quot;&quot;Initialize AudioDatasetHotword.
+
+ Args:
+ *args: Variable positional arguments.
+ **kwargs: Additional keyword arguments.
+ &quot;&quot;&quot;
+ super().__init__(*args, **kwargs)
+ self.seaco_id = seaco_id
+
+ def __getitem__(self, index):
+ &quot;&quot;&quot;Internal: getitem .
+
+ Args:
+ index: TODO.
+ &quot;&quot;&quot;
+ item = self.index_ds[index]
+ # import pdb;
+ # pdb.set_trace()
+ source = item[&quot;source&quot;]
+ data_src = load_audio_text_image_video(source, fs=self.fs)
+ if self.preprocessor_speech:
+ data_src = self.preprocessor_speech(data_src, fs=self.fs)
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/datasets/audio_datasets/datasets.py#L155" target="_blank" class="src-more">View full source on GitHub →</a></details><h3 class="mt">Methods</h3><div class="mblk"><code>.collator(samples)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/datasets/audio_datasets/datasets.py#L268" target="_blank" class="gh-link">L268</a><div class="mdoc"><p>Collator.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>samples</code> — TODO.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def collator(self, samples: list = None):
+ &quot;&quot;&quot;Collator.
+
+ Args:
+ samples: TODO.
+ &quot;&quot;&quot;
+ outputs = {}
+ hotword_indxs = []
+ seaco_id = samples[0][&quot;seaco_id&quot;]
+ for sample in samples:
+ for key in sample.keys():
+ if key == &quot;seaco_id&quot;:
+ continue
+ elif key == &quot;hotword_indx&quot;:
+ hotword_indxs.append(sample[key])
+ else:
+ if key not in outputs:
+ outputs[key] = []
+ outputs[key].append(sample[key])
+
+ for key, data_list in outputs.items():
+ if isinstance(data_list[0], torch.Tensor):
+ if data_list[0].dtype == torch.int64 or data_list[0].dtype == torch.int32:
+ pad_value = self.int_pad_value
+ else:
+ pad_value = self.float_pad_value
+ outputs[key] = torch.nn.utils.rnn.pad_sequence(
+ data_list, batch_first=True, padding_value=pad_value
+ )
+
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/datasets/audio_datasets/datasets.py#L268" target="_blank" class="src-more">View full source →</a></details></div></div>
+<div class="api-detail" id="e527"><span class="dtag badge-m">method</span><h2><code>AudioDatasetHotword.collator(samples)</code></h2><div class="dmod">funasr.datasets.datasets.AudioDatasetHotword · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/datasets/audio_datasets/datasets.py#L268" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Collator.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>samples</code> — TODO.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def collator(self, samples: list = None):
+ &quot;&quot;&quot;Collator.
+
+ Args:
+ samples: TODO.
+ &quot;&quot;&quot;
+ outputs = {}
+ hotword_indxs = []
+ seaco_id = samples[0][&quot;seaco_id&quot;]
+ for sample in samples:
+ for key in sample.keys():
+ if key == &quot;seaco_id&quot;:
+ continue
+ elif key == &quot;hotword_indx&quot;:
+ hotword_indxs.append(sample[key])
+ else:
+ if key not in outputs:
+ outputs[key] = []
+ outputs[key].append(sample[key])
+
+ for key, data_list in outputs.items():
+ if isinstance(data_list[0], torch.Tensor):
+ if data_list[0].dtype == torch.int64 or data_list[0].dtype == torch.int32:
+ pad_value = self.int_pad_value
+ else:
+ pad_value = self.float_pad_value
+ outputs[key] = torch.nn.utils.rnn.pad_sequence(
+ data_list, batch_first=True, padding_value=pad_value
+ )
+
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/datasets/audio_datasets/datasets.py#L268" target="_blank" class="src-more">View full source on GitHub →</a></details></div>
+<div class="api-detail" id="e528"><span class="dtag badge-f">function</span><h2><code>EspnetStyleBatchSampler_fn(dataset, **kwargs)</code></h2><div class="dmod">funasr.datasets.espnet_samplers · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/datasets/audio_datasets/espnet_samplers.py#L14" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Espnetstylebatchsampler fn.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>dataset</code> — TODO.</li>
+<li><code>**kwargs</code> — Additional keyword arguments.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code>def EspnetStyleBatchSampler_fn(dataset, **kwargs):
+ &quot;&quot;&quot;Espnetstylebatchsampler fn.
+
+ Args:
+ dataset: TODO.
+ **kwargs: Additional keyword arguments.
+ &quot;&quot;&quot;
+ dataloader_args = {}
+
+ batch_sampler = EspnetStyleBatchSampler(dataset, **kwargs)
+ dataloader_args[&quot;batch_sampler&quot;] = batch_sampler
+ dataloader_args[&quot;num_workers&quot;] = kwargs.get(&quot;num_workers&quot;, 4)
+ dataloader_args[&quot;pin_memory&quot;] = kwargs.get(&quot;pin_memory&quot;, True)
+
+ return dataloader_args
+</code></pre></details></div>
+<div class="api-detail" id="e529"><span class="dtag badge-c">class</span><h2>EspnetStyleBatchSampler</h2><div class="dmod">funasr.datasets.espnet_samplers · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/datasets/audio_datasets/espnet_samplers.py#L37" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p class="muted">No documentation yet.</p></div><details class="src-block"><summary>📄 Source code</summary><pre><code>class EspnetStyleBatchSampler(DistributedSampler):
+ def __init__(
+ self,
+ dataset,
+ batch_size,
+ batch_type=&quot;token&quot;,
+ rank=None,
+ num_replicas=None,
+ rank_split=False,
+ shuffle=True,
+ drop_last=False,
+ is_training: bool = True,
+ sort_size: int = 1024,
+ start_step: int = 0,
+ **kwargs,
+ ):
+
+ &quot;&quot;&quot;Initialize EspnetStyleBatchSampler.
+
+ Args:
+ dataset: TODO.
+ batch_size: Number of samples per batch.
+ batch_type: TODO.
+ rank: TODO.
+ num_replicas: TODO.
+ rank_split: TODO.
+ shuffle: TODO.
+ drop_last: TODO.
+ is_training: Boolean flag for training.
+ sort_size: Size/dimension parameter.
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/datasets/audio_datasets/espnet_samplers.py#L37" target="_blank" class="src-more">View full source on GitHub →</a></details><h3 class="mt">Methods</h3><div class="mblk"><code>.set_epoch(epoch)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/datasets/audio_datasets/espnet_samplers.py#L187" target="_blank" class="gh-link">L187</a><div class="mdoc"><p>Set epoch.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>epoch</code> — TODO.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def set_epoch(self, epoch):
+ # Set the epoch for shuffling
+ &quot;&quot;&quot;Set epoch.
+
+ Args:
+ epoch: TODO.
+ &quot;&quot;&quot;
+ self.epoch = epoch
+</code></pre></details></div></div>
+<div class="api-detail" id="e530"><span class="dtag badge-m">method</span><h2><code>EspnetStyleBatchSampler.set_epoch(epoch)</code></h2><div class="dmod">funasr.datasets.espnet_samplers.EspnetStyleBatchSampler · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/datasets/audio_datasets/espnet_samplers.py#L187" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Set epoch.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>epoch</code> — TODO.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def set_epoch(self, epoch):
+ # Set the epoch for shuffling
+ &quot;&quot;&quot;Set epoch.
+
+ Args:
+ epoch: TODO.
+ &quot;&quot;&quot;
+ self.epoch = epoch
+</code></pre></details></div>
+<div class="api-detail" id="e531"><span class="dtag badge-c">class</span><h2>IndexDSJsonlRankFull</h2><div class="dmod">funasr.datasets.index_ds · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/datasets/audio_datasets/index_ds.py#L16" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p class="muted">No documentation yet.</p></div><details class="src-block"><summary>📄 Source code</summary><pre><code>class IndexDSJsonlRankFull(torch.utils.data.Dataset):
+
+ def __init__(self, path: str, **kwargs):
+ &quot;&quot;&quot;Initialize IndexDSJsonlRankFull.
+
+ Args:
+ path: TODO.
+ **kwargs: Additional keyword arguments.
+ &quot;&quot;&quot;
+ super().__init__()
+ self.max_source_length = kwargs.get(&quot;max_source_length&quot;, 2048)
+ self.min_source_length = kwargs.get(&quot;min_source_length&quot;, 0)
+ self.max_target_length = kwargs.get(&quot;max_target_length&quot;, 2048)
+ self.min_target_length = kwargs.get(&quot;min_target_length&quot;, 0)
+ self.max_token_length = kwargs.get(&quot;max_token_length&quot;, 2200)
+
+ is_training = kwargs.get(&quot;is_training&quot;, True)
+ if not (path.endswith(&quot;.jsonl&quot;) or path.endswith(&quot;.json&quot;)):
+ # jsonl list file
+ data_split_num = kwargs.get(&quot;data_split_num&quot;, 1)
+ data_split_i = kwargs.get(&quot;data_split_i&quot;, 0)
+
+ if not is_training:
+ data_split_num = 1
+ data_split_i = 0
+ with open(path, encoding=&quot;utf-8&quot;) as fin:
+ file_list_all = fin.readlines()
+
+ num_per_slice = (len(file_list_all) - 1) // data_split_num + 1 # 16
+ file_list = file_list_all[
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/datasets/audio_datasets/index_ds.py#L16" target="_blank" class="src-more">View full source on GitHub →</a></details><h3 class="mt">Methods</h3><div class="mblk"><code>.get_source_len(data_dict)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/datasets/audio_datasets/index_ds.py#L158" target="_blank" class="gh-link">L158</a><div class="mdoc"><p>Get source len.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>data_dict</code> — TODO.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def get_source_len(self, data_dict):
+ &quot;&quot;&quot;Get source len.
+
+ Args:
+ data_dict: TODO.
+ &quot;&quot;&quot;
+ return data_dict.get(&quot;source_len&quot;, 1)
+</code></pre></details></div><div class="mblk"><code>.get_target_len(data_dict)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/datasets/audio_datasets/index_ds.py#L166" target="_blank" class="gh-link">L166</a><div class="mdoc"><p>Get target len.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>data_dict</code> — TODO.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def get_target_len(self, data_dict):
+
+ &quot;&quot;&quot;Get target len.
+
+ Args:
+ data_dict: TODO.
+ &quot;&quot;&quot;
+ return data_dict.get(&quot;target_len&quot;, 0)
+</code></pre></details></div></div>
+<div class="api-detail" id="e532"><span class="dtag badge-m">method</span><h2><code>IndexDSJsonlRankFull.get_source_len(data_dict)</code></h2><div class="dmod">funasr.datasets.index_ds.IndexDSJsonlRankFull · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/datasets/audio_datasets/index_ds.py#L158" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Get source len.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>data_dict</code> — TODO.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def get_source_len(self, data_dict):
+ &quot;&quot;&quot;Get source len.
+
+ Args:
+ data_dict: TODO.
+ &quot;&quot;&quot;
+ return data_dict.get(&quot;source_len&quot;, 1)
+</code></pre></details></div>
+<div class="api-detail" id="e533"><span class="dtag badge-m">method</span><h2><code>IndexDSJsonlRankFull.get_target_len(data_dict)</code></h2><div class="dmod">funasr.datasets.index_ds.IndexDSJsonlRankFull · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/datasets/audio_datasets/index_ds.py#L166" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Get target len.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>data_dict</code> — TODO.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def get_target_len(self, data_dict):
+
+ &quot;&quot;&quot;Get target len.
+
+ Args:
+ data_dict: TODO.
+ &quot;&quot;&quot;
+ return data_dict.get(&quot;target_len&quot;, 0)
+</code></pre></details></div>
+<div class="api-detail" id="e534"><span class="dtag badge-f">function</span><h2><code>gen_scp_from_jsonl(jsonl_file, data_type_list, wav_scp_file, text_file)</code></h2><div class="dmod">funasr.datasets.jsonl2scp · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/datasets/audio_datasets/jsonl2scp.py#L12" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Gen scp from jsonl.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>jsonl_file</code> — TODO.</li>
+<li><code>data_type_list</code> — TODO.</li>
+<li><code>wav_scp_file</code> — TODO.</li>
+<li><code>text_file</code> — TODO.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code>def gen_scp_from_jsonl(jsonl_file, data_type_list, wav_scp_file, text_file):
+
+ &quot;&quot;&quot;Gen scp from jsonl.
+
+ Args:
+ jsonl_file: TODO.
+ data_type_list: TODO.
+ wav_scp_file: TODO.
+ text_file: TODO.
+ &quot;&quot;&quot;
+ wav_f = open(wav_scp_file, &quot;w&quot;)
+ text_f = open(text_file, &quot;w&quot;)
+ with open(jsonl_file, encoding=&quot;utf-8&quot;) as fin:
+ for line in fin:
+ data = json.loads(line.strip())
+
+ prompt = data.get(&quot;prompt&quot;, &quot;&lt;ASR&gt;&quot;)
+ source = data[data_type_list[0]]
+ target = data[data_type_list[1]]
+ source_len = data.get(&quot;source_len&quot;, 1)
+ target_len = data.get(&quot;target_len&quot;, 0)
+ if &quot;aishell&quot; in source:
+ target = target.replace(&quot; &quot;, &quot;&quot;)
+ key = data[&quot;key&quot;]
+ wav_f.write(f&quot;{key}\t{source}\n&quot;)
+ wav_f.flush()
+ text_f.write(f&quot;{key}\t{target}\n&quot;)
+ text_f.flush()
+
+ wav_f.close()
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/datasets/audio_datasets/jsonl2scp.py#L12" target="_blank" class="src-more">View full source on GitHub →</a></details></div>
+<div class="api-detail" id="e535"><span class="dtag badge-f">function</span><h2><code>main_hydra(cfg)</code></h2><div class="dmod">funasr.datasets.jsonl2scp · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/datasets/audio_datasets/jsonl2scp.py#L46" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Main hydra.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>cfg</code> — Configuration overrides.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code>def main_hydra(cfg: DictConfig):
+
+ &quot;&quot;&quot;Main hydra.
+
+ Args:
+ cfg: Configuration overrides.
+ &quot;&quot;&quot;
+ kwargs = OmegaConf.to_container(cfg, resolve=True)
+ print(kwargs)
+
+ scp_file_list = kwargs.get(
+ &quot;scp_file_list&quot;,
+ (&quot;/Users/zhifu/funasr1.0/test_local/wav.scp&quot;, &quot;/Users/zhifu/funasr1.0/test_local/text.txt&quot;),
+ )
+ if isinstance(scp_file_list, str):
+ scp_file_list = eval(scp_file_list)
+ data_type_list = kwargs.get(&quot;data_type_list&quot;, (&quot;source&quot;, &quot;target&quot;))
+ jsonl_file = kwargs.get(
+ &quot;jsonl_file_in&quot;, &quot;/Users/zhifu/funasr1.0/test_local/audio_datasets.jsonl&quot;
+ )
+ gen_scp_from_jsonl(jsonl_file, data_type_list, *scp_file_list)
+</code></pre></details></div>
+<div class="api-detail" id="e536"><span class="dtag badge-c">class</span><h2>SpeechPreprocessSpeedPerturb</h2><div class="dmod">funasr.datasets.preprocessor · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/datasets/audio_datasets/preprocessor.py#L19" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p class="muted">No documentation yet.</p></div><details class="src-block"><summary>📄 Source code</summary><pre><code>class SpeechPreprocessSpeedPerturb(nn.Module):
+ def __init__(self, speed_perturb: list = None, **kwargs):
+ &quot;&quot;&quot;Initialize SpeechPreprocessSpeedPerturb.
+
+ Args:
+ speed_perturb: TODO.
+ **kwargs: Additional keyword arguments.
+ &quot;&quot;&quot;
+ super().__init__()
+ self.speed_perturb = speed_perturb
+
+ def forward(self, waveform, fs, **kwargs):
+ &quot;&quot;&quot;Forward pass for training.
+
+ Args:
+ waveform: TODO.
+ fs: TODO.
+ **kwargs: Additional keyword arguments.
+ &quot;&quot;&quot;
+ if self.speed_perturb is None:
+ return waveform
+ speed = random.choice(self.speed_perturb)
+ if speed != 1.0:
+ if not isinstance(waveform, torch.Tensor):
+ waveform = torch.tensor(waveform)
+ waveform, _ = torchaudio.sox_effects.apply_effects_tensor(
+ waveform.view(1, -1), fs, [[&quot;speed&quot;, str(speed)], [&quot;rate&quot;, str(fs)]]
+ )
+ waveform = waveform.view(-1)
+
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/datasets/audio_datasets/preprocessor.py#L19" target="_blank" class="src-more">View full source on GitHub →</a></details><h3 class="mt">Methods</h3><div class="mblk"><code>.forward(waveform, fs, **kwargs)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/datasets/audio_datasets/preprocessor.py#L30" target="_blank" class="gh-link">L30</a><div class="mdoc"><p>Forward pass for training.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>waveform</code> — TODO.</li>
+<li><code>fs</code> — TODO.</li>
+<li><code>**kwargs</code> — Additional keyword arguments.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def forward(self, waveform, fs, **kwargs):
+ &quot;&quot;&quot;Forward pass for training.
+
+ Args:
+ waveform: TODO.
+ fs: TODO.
+ **kwargs: Additional keyword arguments.
+ &quot;&quot;&quot;
+ if self.speed_perturb is None:
+ return waveform
+ speed = random.choice(self.speed_perturb)
+ if speed != 1.0:
+ if not isinstance(waveform, torch.Tensor):
+ waveform = torch.tensor(waveform)
+ waveform, _ = torchaudio.sox_effects.apply_effects_tensor(
+ waveform.view(1, -1), fs, [[&quot;speed&quot;, str(speed)], [&quot;rate&quot;, str(fs)]]
+ )
+ waveform = waveform.view(-1)
+
+ return waveform
+</code></pre></details></div></div>
+<div class="api-detail" id="e537"><span class="dtag badge-m">method</span><h2><code>SpeechPreprocessSpeedPerturb.forward(waveform, fs, **kwargs)</code></h2><div class="dmod">funasr.datasets.preprocessor.SpeechPreprocessSpeedPerturb · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/datasets/audio_datasets/preprocessor.py#L30" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Forward pass for training.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>waveform</code> — TODO.</li>
+<li><code>fs</code> — TODO.</li>
+<li><code>**kwargs</code> — Additional keyword arguments.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def forward(self, waveform, fs, **kwargs):
+ &quot;&quot;&quot;Forward pass for training.
+
+ Args:
+ waveform: TODO.
+ fs: TODO.
+ **kwargs: Additional keyword arguments.
+ &quot;&quot;&quot;
+ if self.speed_perturb is None:
+ return waveform
+ speed = random.choice(self.speed_perturb)
+ if speed != 1.0:
+ if not isinstance(waveform, torch.Tensor):
+ waveform = torch.tensor(waveform)
+ waveform, _ = torchaudio.sox_effects.apply_effects_tensor(
+ waveform.view(1, -1), fs, [[&quot;speed&quot;, str(speed)], [&quot;rate&quot;, str(fs)]]
+ )
+ waveform = waveform.view(-1)
+
+ return waveform
+</code></pre></details></div>
+<div class="api-detail" id="e538"><span class="dtag badge-c">class</span><h2>TextPreprocessSegDict</h2><div class="dmod">funasr.datasets.preprocessor · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/datasets/audio_datasets/preprocessor.py#L53" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p class="muted">No documentation yet.</p></div><details class="src-block"><summary>📄 Source code</summary><pre><code>class TextPreprocessSegDict(nn.Module):
+ def __init__(
+ self,
+ seg_dict: str = None,
+ text_cleaner: Collection[str] = None,
+ split_with_space: bool = False,
+ **kwargs
+ ):
+ &quot;&quot;&quot;Initialize TextPreprocessSegDict.
+
+ Args:
+ seg_dict: TODO.
+ text_cleaner: TODO.
+ split_with_space: TODO.
+ **kwargs: Additional keyword arguments.
+ &quot;&quot;&quot;
+ super().__init__()
+
+ self.text_cleaner = TextCleaner(text_cleaner)
+
+ def forward(self, text, **kwargs):
+ &quot;&quot;&quot;Forward pass for training.
+
+ Args:
+ text: Text tensor or string input.
+ **kwargs: Additional keyword arguments.
+ &quot;&quot;&quot;
+ text = self.text_cleaner(text)
+
+ return text
+</code></pre></details><h3 class="mt">Methods</h3><div class="mblk"><code>.forward(text, **kwargs)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/datasets/audio_datasets/preprocessor.py#L73" target="_blank" class="gh-link">L73</a><div class="mdoc"><p>Forward pass for training.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>text</code> — Text tensor or string input.</li>
+<li><code>**kwargs</code> — Additional keyword arguments.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def forward(self, text, **kwargs):
+ &quot;&quot;&quot;Forward pass for training.
+
+ Args:
+ text: Text tensor or string input.
+ **kwargs: Additional keyword arguments.
+ &quot;&quot;&quot;
+ text = self.text_cleaner(text)
+
+ return text
+</code></pre></details></div></div>
+<div class="api-detail" id="e539"><span class="dtag badge-m">method</span><h2><code>TextPreprocessSegDict.forward(text, **kwargs)</code></h2><div class="dmod">funasr.datasets.preprocessor.TextPreprocessSegDict · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/datasets/audio_datasets/preprocessor.py#L73" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Forward pass for training.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>text</code> — Text tensor or string input.</li>
+<li><code>**kwargs</code> — Additional keyword arguments.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def forward(self, text, **kwargs):
+ &quot;&quot;&quot;Forward pass for training.
+
+ Args:
+ text: Text tensor or string input.
+ **kwargs: Additional keyword arguments.
+ &quot;&quot;&quot;
+ text = self.text_cleaner(text)
+
+ return text
+</code></pre></details></div>
+<div class="api-detail" id="e540"><span class="dtag badge-f">function</span><h2><code>CustomDistributedBatchSampler_fn(dataset, **kwargs)</code></h2><div class="dmod">funasr.datasets.samplers · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/datasets/audio_datasets/samplers.py#L20" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Customdistributedbatchsampler fn.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>dataset</code> — TODO.</li>
+<li><code>**kwargs</code> — Additional keyword arguments.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code>def CustomDistributedBatchSampler_fn(dataset, **kwargs):
+ &quot;&quot;&quot;Customdistributedbatchsampler fn.
+
+ Args:
+ dataset: TODO.
+ **kwargs: Additional keyword arguments.
+ &quot;&quot;&quot;
+ dataloader_args = {}
+ batch_type = kwargs.get(&quot;batch_type&quot;, &quot;example&quot;)
+ if batch_type == &quot;example&quot;:
+ batch_sampler = CustomDistributedBatchSampler(dataset, **kwargs)
+
+ else:
+ if kwargs.get(&quot;sort_size&quot;, -1) &gt; 0:
+ batch_sampler = CustomDistributedBufferDynamicBatchSampler(dataset, **kwargs)
+ else:
+ batch_sampler = CustomDistributedDynamicBatchSampler(dataset, **kwargs)
+ # batch_sampler = CustomDistributedDynamicBatchSampler(dataset, **kwargs)
+
+ dataloader_args[&quot;batch_sampler&quot;] = batch_sampler
+ dataloader_args[&quot;num_workers&quot;] = kwargs.get(&quot;num_workers&quot;, 4)
+ dataloader_args[&quot;pin_memory&quot;] = kwargs.get(&quot;pin_memory&quot;, True)
+
+ return dataloader_args
+</code></pre></details></div>
+<div class="api-detail" id="e541"><span class="dtag badge-c">class</span><h2>CustomDistributedBatchSampler</h2><div class="dmod">funasr.datasets.samplers · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/datasets/audio_datasets/samplers.py#L46" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p class="muted">No documentation yet.</p></div><details class="src-block"><summary>📄 Source code</summary><pre><code>class CustomDistributedBatchSampler(Sampler):
+ def __init__(
+ self,
+ dataset,
+ batch_size,
+ num_replicas=None,
+ rank=None,
+ shuffle=True,
+ drop_last=False,
+ is_training: bool = True,
+ **kwargs,
+ ):
+
+ &quot;&quot;&quot;Initialize CustomDistributedBatchSampler.
+
+ Args:
+ dataset: TODO.
+ batch_size: Number of samples per batch.
+ num_replicas: TODO.
+ rank: TODO.
+ shuffle: TODO.
+ drop_last: TODO.
+ is_training: Boolean flag for training.
+ **kwargs: Additional keyword arguments.
+ &quot;&quot;&quot;
+ try:
+ rank = dist.get_rank()
+ num_replicas = dist.get_world_size()
+ except:
+ rank = 0
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/datasets/audio_datasets/samplers.py#L46" target="_blank" class="src-more">View full source on GitHub →</a></details><h3 class="mt">Methods</h3><div class="mblk"><code>.set_epoch(epoch)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/datasets/audio_datasets/samplers.py#L148" target="_blank" class="gh-link">L148</a><div class="mdoc"><p>Set epoch.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>epoch</code> — TODO.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def set_epoch(self, epoch):
+ &quot;&quot;&quot;Set epoch.
+
+ Args:
+ epoch: TODO.
+ &quot;&quot;&quot;
+ self.epoch = epoch
+</code></pre></details></div></div>
+<div class="api-detail" id="e542"><span class="dtag badge-m">method</span><h2><code>CustomDistributedBatchSampler.set_epoch(epoch)</code></h2><div class="dmod">funasr.datasets.samplers.CustomDistributedBatchSampler · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/datasets/audio_datasets/samplers.py#L148" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Set epoch.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>epoch</code> — TODO.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def set_epoch(self, epoch):
+ &quot;&quot;&quot;Set epoch.
+
+ Args:
+ epoch: TODO.
+ &quot;&quot;&quot;
+ self.epoch = epoch
+</code></pre></details></div>
+<div class="api-detail" id="e543"><span class="dtag badge-c">class</span><h2>CustomDistributedBufferBatchSampler</h2><div class="dmod">funasr.datasets.samplers · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/datasets/audio_datasets/samplers.py#L157" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p class="muted">No documentation yet.</p></div><details class="src-block"><summary>📄 Source code</summary><pre><code>class CustomDistributedBufferBatchSampler(Sampler):
+ def __init__(
+ self,
+ dataset,
+ batch_size,
+ num_replicas=None,
+ rank=None,
+ shuffle=True,
+ drop_last=False,
+ is_training: bool = True,
+ sort_size: int = 1024,
+ **kwargs,
+ ):
+
+ &quot;&quot;&quot;Initialize CustomDistributedBufferBatchSampler.
+
+ Args:
+ dataset: TODO.
+ batch_size: Number of samples per batch.
+ num_replicas: TODO.
+ rank: TODO.
+ shuffle: TODO.
+ drop_last: TODO.
+ is_training: Boolean flag for training.
+ sort_size: Size/dimension parameter.
+ **kwargs: Additional keyword arguments.
+ &quot;&quot;&quot;
+ try:
+ rank = dist.get_rank()
+ num_replicas = dist.get_world_size()
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/datasets/audio_datasets/samplers.py#L157" target="_blank" class="src-more">View full source on GitHub →</a></details><h3 class="mt">Methods</h3><div class="mblk"><code>.set_epoch(epoch)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/datasets/audio_datasets/samplers.py#L284" target="_blank" class="gh-link">L284</a><div class="mdoc"><p>Set epoch.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>epoch</code> — TODO.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def set_epoch(self, epoch):
+ &quot;&quot;&quot;Set epoch.
+
+ Args:
+ epoch: TODO.
+ &quot;&quot;&quot;
+ self.epoch = epoch
+</code></pre></details></div></div>
+<div class="api-detail" id="e544"><span class="dtag badge-m">method</span><h2><code>CustomDistributedBufferBatchSampler.set_epoch(epoch)</code></h2><div class="dmod">funasr.datasets.samplers.CustomDistributedBufferBatchSampler · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/datasets/audio_datasets/samplers.py#L284" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Set epoch.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>epoch</code> — TODO.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def set_epoch(self, epoch):
+ &quot;&quot;&quot;Set epoch.
+
+ Args:
+ epoch: TODO.
+ &quot;&quot;&quot;
+ self.epoch = epoch
+</code></pre></details></div>
+<div class="api-detail" id="e545"><span class="dtag badge-c">class</span><h2>CustomDistributedDynamicBatchSampler</h2><div class="dmod">funasr.datasets.samplers · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/datasets/audio_datasets/samplers.py#L293" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p class="muted">No documentation yet.</p></div><details class="src-block"><summary>📄 Source code</summary><pre><code>class CustomDistributedDynamicBatchSampler(DistributedSampler):
+ def __init__(
+ self,
+ dataset,
+ batch_size,
+ num_replicas=None,
+ rank=None,
+ shuffle=True,
+ drop_last=False,
+ is_training: bool = True,
+ **kwargs,
+ ):
+
+ &quot;&quot;&quot;Initialize CustomDistributedDynamicBatchSampler.
+
+ Args:
+ dataset: TODO.
+ batch_size: Number of samples per batch.
+ num_replicas: TODO.
+ rank: TODO.
+ shuffle: TODO.
+ drop_last: TODO.
+ is_training: Boolean flag for training.
+ **kwargs: Additional keyword arguments.
+ &quot;&quot;&quot;
+ try:
+ rank = dist.get_rank()
+ num_replicas = dist.get_world_size()
+ except:
+ rank = 0
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/datasets/audio_datasets/samplers.py#L293" target="_blank" class="src-more">View full source on GitHub →</a></details><h3 class="mt">Methods</h3><div class="mblk"><code>.set_epoch(epoch)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/datasets/audio_datasets/samplers.py#L384" target="_blank" class="gh-link">L384</a><div class="mdoc"><p>Set epoch.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>epoch</code> — TODO.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def set_epoch(self, epoch):
+ &quot;&quot;&quot;Set epoch.
+
+ Args:
+ epoch: TODO.
+ &quot;&quot;&quot;
+ self.epoch = epoch
+</code></pre></details></div></div>
+<div class="api-detail" id="e546"><span class="dtag badge-m">method</span><h2><code>CustomDistributedDynamicBatchSampler.set_epoch(epoch)</code></h2><div class="dmod">funasr.datasets.samplers.CustomDistributedDynamicBatchSampler · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/datasets/audio_datasets/samplers.py#L384" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Set epoch.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>epoch</code> — TODO.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def set_epoch(self, epoch):
+ &quot;&quot;&quot;Set epoch.
+
+ Args:
+ epoch: TODO.
+ &quot;&quot;&quot;
+ self.epoch = epoch
+</code></pre></details></div>
+<div class="api-detail" id="e547"><span class="dtag badge-c">class</span><h2>CustomDistributedBufferDynamicBatchSampler</h2><div class="dmod">funasr.datasets.samplers · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/datasets/audio_datasets/samplers.py#L393" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p class="muted">No documentation yet.</p></div><details class="src-block"><summary>📄 Source code</summary><pre><code>class CustomDistributedBufferDynamicBatchSampler(DistributedSampler):
+ def __init__(
+ self,
+ dataset,
+ batch_size,
+ batch_type=&quot;token&quot;,
+ num_replicas=None,
+ rank=None,
+ rank_split=False,
+ shuffle=True,
+ drop_last=False,
+ is_training: bool = True,
+ sort_size: int = 1024,
+ start_step: int = 0,
+ **kwargs,
+ ):
+
+ &quot;&quot;&quot;Initialize CustomDistributedBufferDynamicBatchSampler.
+
+ Args:
+ dataset: TODO.
+ batch_size: Number of samples per batch.
+ batch_type: TODO.
+ num_replicas: TODO.
+ rank: TODO.
+ rank_split: TODO.
+ shuffle: TODO.
+ drop_last: TODO.
+ is_training: Boolean flag for training.
+ sort_size: Size/dimension parameter.
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/datasets/audio_datasets/samplers.py#L393" target="_blank" class="src-more">View full source on GitHub →</a></details><h3 class="mt">Methods</h3><div class="mblk"><code>.set_epoch(epoch)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/datasets/audio_datasets/samplers.py#L526" target="_blank" class="gh-link">L526</a><div class="mdoc"><p>Set epoch.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>epoch</code> — TODO.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def set_epoch(self, epoch):
+ &quot;&quot;&quot;Set epoch.
+
+ Args:
+ epoch: TODO.
+ &quot;&quot;&quot;
+ self.epoch = epoch
+</code></pre></details></div></div>
+<div class="api-detail" id="e548"><span class="dtag badge-m">method</span><h2><code>CustomDistributedBufferDynamicBatchSampler.set_epoch(epoch)</code></h2><div class="dmod">funasr.datasets.samplers.CustomDistributedBufferDynamicBatchSampler · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/datasets/audio_datasets/samplers.py#L526" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Set epoch.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>epoch</code> — TODO.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def set_epoch(self, epoch):
+ &quot;&quot;&quot;Set epoch.
+
+ Args:
+ epoch: TODO.
+ &quot;&quot;&quot;
+ self.epoch = epoch
+</code></pre></details></div>
+<div class="api-detail" id="e549"><span class="dtag badge-c">class</span><h2>DistributedSamplerWarp</h2><div class="dmod">funasr.datasets.samplers · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/datasets/audio_datasets/samplers.py#L535" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p class="muted">No documentation yet.</p></div><details class="src-block"><summary>📄 Source code</summary><pre><code>class DistributedSamplerWarp(BatchSampler):
+ def __init__(
+ self, dataset, batch_size, num_replicas=None, rank=None, shuffle=True, drop_last=False
+ ):
+ &quot;&quot;&quot;Initialize DistributedSamplerWarp.
+
+ Args:
+ dataset: TODO.
+ batch_size: Number of samples per batch.
+ num_replicas: TODO.
+ rank: TODO.
+ shuffle: TODO.
+ drop_last: TODO.
+ &quot;&quot;&quot;
+ if num_replicas is None:
+ if not torch.distributed.is_available():
+ raise RuntimeError(&quot;Requires distributed package to be available&quot;)
+ num_replicas = torch.distributed.get_world_size()
+ if rank is None:
+ if not torch.distributed.is_available():
+ raise RuntimeError(&quot;Requires distributed package to be available&quot;)
+ rank = torch.distributed.get_rank()
+
+ self.dataset = dataset
+ self.batch_size = batch_size
+ self.num_replicas = num_replicas
+ self.rank = rank
+ self.shuffle = shuffle
+ self.drop_last = drop_last
+
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/datasets/audio_datasets/samplers.py#L535" target="_blank" class="src-more">View full source on GitHub →</a></details><h3 class="mt">Methods</h3><div class="mblk"><code>.set_epoch(epoch)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/datasets/audio_datasets/samplers.py#L582" target="_blank" class="gh-link">L582</a><div class="mdoc"><p>Set epoch.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>epoch</code> — TODO.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def set_epoch(self, epoch):
+ &quot;&quot;&quot;Set epoch.
+
+ Args:
+ epoch: TODO.
+ &quot;&quot;&quot;
+ self.epoch = epoch
+</code></pre></details></div></div>
+<div class="api-detail" id="e550"><span class="dtag badge-m">method</span><h2><code>DistributedSamplerWarp.set_epoch(epoch)</code></h2><div class="dmod">funasr.datasets.samplers.DistributedSamplerWarp · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/datasets/audio_datasets/samplers.py#L582" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Set epoch.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>epoch</code> — TODO.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def set_epoch(self, epoch):
+ &quot;&quot;&quot;Set epoch.
+
+ Args:
+ epoch: TODO.
+ &quot;&quot;&quot;
+ self.epoch = epoch
+</code></pre></details></div>
+<div class="api-detail" id="e551"><span class="dtag badge-f">function</span><h2><code>gen_jsonl_from_wav_text_list(path, data_type_list, jsonl_file_out, **kwargs)</code></h2><div class="dmod">funasr.datasets.scp2jsonl · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/datasets/audio_datasets/scp2jsonl.py#L13" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Gen jsonl from wav text list.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>path</code> — TODO.</li>
+<li><code>data_type_list</code> — TODO.</li>
+<li><code>jsonl_file_out</code> — TODO.</li>
+<li><code>**kwargs</code> — Additional keyword arguments.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code>def gen_jsonl_from_wav_text_list(
+ path, data_type_list=(&quot;source&quot;, &quot;target&quot;), jsonl_file_out: str = None, **kwargs
+):
+ &quot;&quot;&quot;Gen jsonl from wav text list.
+
+ Args:
+ path: TODO.
+ data_type_list: TODO.
+ jsonl_file_out: TODO.
+ **kwargs: Additional keyword arguments.
+ &quot;&quot;&quot;
+ try:
+ rank = dist.get_rank()
+ world_size = dist.get_world_size()
+ except:
+ rank = 0
+ world_size = 1
+
+ cpu_cores = os.cpu_count() or 1
+ print(f&quot;convert wav.scp text to jsonl, ncpu: {cpu_cores}&quot;)
+ if rank == 0:
+ json_dict = {}
+ for data_type, data_file in zip(data_type_list, path):
+ json_dict[data_type] = {}
+ with open(data_file, &quot;r&quot;) as f:
+
+ data_file_lists = f.readlines()
+ lines_for_each_th = (len(data_file_lists) - 1) // cpu_cores + 1
+ task_num = cpu_cores if len(data_file_lists) &gt; cpu_cores else 1
+ # import pdb;pdb.set_trace()
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/datasets/audio_datasets/scp2jsonl.py#L13" target="_blank" class="src-more">View full source on GitHub →</a></details></div>
+<div class="api-detail" id="e552"><span class="dtag badge-f">function</span><h2><code>parse_context_length(data_list, data_type, id)</code></h2><div class="dmod">funasr.datasets.scp2jsonl · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/datasets/audio_datasets/scp2jsonl.py#L83" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Parse context length.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>data_list</code> — TODO.</li>
+<li><code>data_type</code> — TODO.</li>
+<li><code>id</code> — TODO.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code>def parse_context_length(data_list: list, data_type: str, id=0):
+ &quot;&quot;&quot;Parse context length.
+
+ Args:
+ data_list: TODO.
+ data_type: TODO.
+ id: TODO.
+ &quot;&quot;&quot;
+ pbar = tqdm(total=len(data_list), dynamic_ncols=True)
+ res = {}
+ for i, line in enumerate(data_list):
+ pbar.update(1)
+ pbar.set_description(f&quot;cpu: {id}&quot;)
+ lines = line.strip().split(maxsplit=1)
+ key = lines[0]
+ line = lines[1] if len(lines) &gt; 1 else &quot;&quot;
+ line = line.strip()
+ if data_type == &quot;source&quot;:
+ if os.path.exists(line):
+ waveform, _ = librosa.load(line, sr=16000)
+ sample_num = len(waveform)
+ context_len = int(sample_num * 1000 / 16000 / 10)
+ else:
+ print(&quot;source file not found: {}&quot;.format(line))
+ continue
+ else:
+ context_len = len(line.split()) if &quot; &quot; in line else len(line)
+ res[key] = {data_type: line, f&quot;{data_type}_len&quot;: context_len}
+ return res
+</code></pre></details></div>
+<div class="api-detail" id="e553"><span class="dtag badge-f">function</span><h2><code>main_hydra(cfg)</code></h2><div class="dmod">funasr.datasets.scp2jsonl · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/datasets/audio_datasets/scp2jsonl.py#L115" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Main hydra.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>cfg</code> — Configuration overrides.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code>def main_hydra(cfg: DictConfig):
+
+ &quot;&quot;&quot;Main hydra.
+
+ Args:
+ cfg: Configuration overrides.
+ &quot;&quot;&quot;
+ kwargs = OmegaConf.to_container(cfg, resolve=True)
+ print(kwargs)
+
+ scp_file_list = kwargs.get(
+ &quot;scp_file_list&quot;,
+ (&quot;/Users/zhifu/funasr1.0/test_local/wav.scp&quot;, &quot;/Users/zhifu/funasr1.0/test_local/text.txt&quot;),
+ )
+ if isinstance(scp_file_list, str):
+ scp_file_list = eval(scp_file_list)
+ data_type_list = kwargs.get(&quot;data_type_list&quot;, (&quot;source&quot;, &quot;target&quot;))
+ jsonl_file_out = kwargs.get(
+ &quot;jsonl_file_out&quot;, &quot;/Users/zhifu/funasr1.0/test_local/audio_datasets.jsonl&quot;
+ )
+ gen_jsonl_from_wav_text_list(
+ scp_file_list, data_type_list=data_type_list, jsonl_file_out=jsonl_file_out
+ )
+</code></pre></details></div>
+<div class="api-detail" id="e554"><span class="dtag badge-f">function</span><h2><code>gen_jsonl_from_wav_text_list(path, data_type_list, jsonl_file_out, **kwargs)</code></h2><div class="dmod">funasr.datasets.scp2len · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/datasets/audio_datasets/scp2len.py#L13" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Gen jsonl from wav text list.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>path</code> — TODO.</li>
+<li><code>data_type_list</code> — TODO.</li>
+<li><code>jsonl_file_out</code> — TODO.</li>
+<li><code>**kwargs</code> — Additional keyword arguments.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code>def gen_jsonl_from_wav_text_list(
+ path, data_type_list=(&quot;source&quot;,), jsonl_file_out: str = None, **kwargs
+):
+ &quot;&quot;&quot;Gen jsonl from wav text list.
+
+ Args:
+ path: TODO.
+ data_type_list: TODO.
+ jsonl_file_out: TODO.
+ **kwargs: Additional keyword arguments.
+ &quot;&quot;&quot;
+ try:
+ rank = dist.get_rank()
+ world_size = dist.get_world_size()
+ except:
+ rank = 0
+ world_size = 1
+
+ cpu_cores = os.cpu_count() or 1
+ print(f&quot;convert wav.scp text to jsonl, ncpu: {cpu_cores}&quot;)
+ if rank == 0:
+ json_dict = {}
+ # for data_type, data_file in zip(data_type_list, path):
+ data_type = data_type_list[0]
+ data_file = path
+ json_dict[data_type] = {}
+ with open(data_file, &quot;r&quot;) as f:
+
+ data_file_lists = f.readlines()
+ print(&quot;&quot;)
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/datasets/audio_datasets/scp2len.py#L13" target="_blank" class="src-more">View full source on GitHub →</a></details></div>
+<div class="api-detail" id="e555"><span class="dtag badge-f">function</span><h2><code>parse_context_length(data_list, data_type, id)</code></h2><div class="dmod">funasr.datasets.scp2len · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/datasets/audio_datasets/scp2len.py#L85" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Parse context length.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>data_list</code> — TODO.</li>
+<li><code>data_type</code> — TODO.</li>
+<li><code>id</code> — TODO.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code>def parse_context_length(data_list: list, data_type: str, id=0):
+ &quot;&quot;&quot;Parse context length.
+
+ Args:
+ data_list: TODO.
+ data_type: TODO.
+ id: TODO.
+ &quot;&quot;&quot;
+ pbar = tqdm(total=len(data_list), dynamic_ncols=True)
+ res = {}
+ for i, line in enumerate(data_list):
+ pbar.update(1)
+ pbar.set_description(f&quot;cpu: {id}&quot;)
+ lines = line.strip().split(maxsplit=1)
+ key = lines[0]
+ line = lines[1] if len(lines) &gt; 1 else &quot;&quot;
+ line = line.strip()
+ if os.path.exists(line):
+ waveform, _ = librosa.load(line, sr=16000)
+ sample_num = len(waveform)
+ context_len = int(sample_num / 16000 * 1000 / 10)
+ else:
+ context_len = len(line.split()) if &quot; &quot; in line else len(line)
+ res[key] = {data_type: line, f&quot;{data_type}_len&quot;: context_len}
+ return res
+</code></pre></details></div>
+<div class="api-detail" id="e556"><span class="dtag badge-f">function</span><h2><code>main_hydra(cfg)</code></h2><div class="dmod">funasr.datasets.scp2len · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/datasets/audio_datasets/scp2len.py#L113" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Main hydra.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>cfg</code> — Configuration overrides.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code>def main_hydra(cfg: DictConfig):
+
+ &quot;&quot;&quot;Main hydra.
+
+ Args:
+ cfg: Configuration overrides.
+ &quot;&quot;&quot;
+ kwargs = OmegaConf.to_container(cfg, resolve=True)
+ print(kwargs)
+
+ scp_file_list = kwargs.get(&quot;scp_file_list&quot;, &quot;/Users/zhifu/funasr1.0/data/list/train_wav.scp&quot;)
+ # if isinstance(scp_file_list, str):
+ # scp_file_list = eval(scp_file_list)
+ data_type_list = kwargs.get(&quot;data_type_list&quot;, (&quot;source&quot;,))
+ jsonl_file_out = kwargs.get(&quot;jsonl_file_out&quot;, &quot;/Users/zhifu/funasr1.0/data/list/wav_len.txt&quot;)
+ gen_jsonl_from_wav_text_list(
+ scp_file_list, data_type_list=data_type_list, jsonl_file_out=jsonl_file_out
+ )
+</code></pre></details></div>
+<div class="api-detail" id="e557"><span class="dtag badge-f">function</span><h2><code>gen_jsonl_from_wav_text_list(path, data_type_list, jsonl_file_out, model_dir, **kwargs)</code></h2><div class="dmod">funasr.datasets.sensevoice2jsonl · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/datasets/audio_datasets/sensevoice2jsonl.py#L15" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Gen jsonl from wav text list.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>path</code> — TODO.</li>
+<li><code>data_type_list</code> — TODO.</li>
+<li><code>jsonl_file_out</code> — TODO.</li>
+<li><code>model_dir</code> — TODO.</li>
+<li><code>**kwargs</code> — Additional keyword arguments.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code>def gen_jsonl_from_wav_text_list(
+ path, data_type_list=(&quot;source&quot;, &quot;target&quot;), jsonl_file_out: str = None, model_dir: str = &quot;iic/SenseVoiceSmall&quot;, **kwargs
+):
+ &quot;&quot;&quot;Gen jsonl from wav text list.
+
+ Args:
+ path: TODO.
+ data_type_list: TODO.
+ jsonl_file_out: TODO.
+ model_dir: TODO.
+ **kwargs: Additional keyword arguments.
+ &quot;&quot;&quot;
+ try:
+ rank = dist.get_rank()
+ world_size = dist.get_world_size()
+ except:
+ rank = 0
+ world_size = 1
+
+ cpu_cores = os.cpu_count() or 1
+ print(f&quot;convert wav.scp text to jsonl, ncpu: {cpu_cores}&quot;)
+ if rank == 0:
+ json_dict = {}
+ for data_type, data_file in zip(data_type_list, path):
+ json_dict[data_type] = {}
+ with open(data_file, &quot;r&quot;) as f:
+
+ data_file_lists = f.readlines()
+ lines_for_each_th = (len(data_file_lists) - 1) // cpu_cores + 1
+ task_num = cpu_cores if len(data_file_lists) &gt; cpu_cores else 1
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/datasets/audio_datasets/sensevoice2jsonl.py#L15" target="_blank" class="src-more">View full source on GitHub →</a></details></div>
+<div class="api-detail" id="e558"><span class="dtag badge-f">function</span><h2><code>contains_punctuation(s)</code></h2><div class="dmod">funasr.datasets.sensevoice2jsonl · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/datasets/audio_datasets/sensevoice2jsonl.py#L131" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Contains punctuation.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>s</code> — TODO.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code>def contains_punctuation(s):
+ &quot;&quot;&quot;Contains punctuation.
+
+ Args:
+ s: TODO.
+ &quot;&quot;&quot;
+ punctuations = (
+ string.punctuation +
+ &#x27;,。、;:?!&quot;&quot;&#x27;&#x27;()【】《》〈〉「」『』〔〕[]{}~·…—–&#x27;
+ )
+ return any(char in punctuations for char in s)
+</code></pre></details></div>
+<div class="api-detail" id="e559"><span class="dtag badge-f">function</span><h2><code>parse_context_length(data_list, data_type, id)</code></h2><div class="dmod">funasr.datasets.sensevoice2jsonl · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/datasets/audio_datasets/sensevoice2jsonl.py#L143" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Parse context length.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>data_list</code> — TODO.</li>
+<li><code>data_type</code> — TODO.</li>
+<li><code>id</code> — TODO.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code>def parse_context_length(data_list: list, data_type: str, id=0):
+ &quot;&quot;&quot;Parse context length.
+
+ Args:
+ data_list: TODO.
+ data_type: TODO.
+ id: TODO.
+ &quot;&quot;&quot;
+ pbar = tqdm(total=len(data_list), dynamic_ncols=True)
+ res = {}
+ for i, line in enumerate(data_list):
+ pbar.update(1)
+ pbar.set_description(f&quot;cpu: {id}&quot;)
+ lines = line.strip().split(maxsplit=1)
+ key = lines[0]
+ line = lines[1] if len(lines) &gt; 1 else &quot;&quot;
+ line = line.strip()
+ if os.path.exists(line):
+ waveform, _ = librosa.load(line, sr=16000)
+ sample_num = len(waveform)
+ context_len = int(sample_num / 16000 * 1000 / 10)
+ else:
+ context_len = len(line.split()) if &quot; &quot; in line else len(line)
+ if data_type == &quot;source&quot;:
+ res[key] = {data_type: line, f&quot;{data_type}_len&quot;: context_len}
+ elif data_type == &quot;target&quot;:
+ punc = contains_punctuation(line)
+ if punc:
+ with_or_wo_itn = &quot;&lt;|withitn|&gt;&quot;
+ else:
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/datasets/audio_datasets/sensevoice2jsonl.py#L143" target="_blank" class="src-more">View full source on GitHub →</a></details></div>
+<div class="api-detail" id="e560"><span class="dtag badge-f">function</span><h2><code>main_hydra(cfg)</code></h2><div class="dmod">funasr.datasets.sensevoice2jsonl · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/datasets/audio_datasets/sensevoice2jsonl.py#L181" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Main hydra.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>cfg</code> — Configuration overrides.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code>def main_hydra(cfg: DictConfig):
+
+ &quot;&quot;&quot;Main hydra.
+
+ Args:
+ cfg: Configuration overrides.
+ &quot;&quot;&quot;
+ kwargs = OmegaConf.to_container(cfg, resolve=True)
+ print(kwargs)
+
+ scp_file_list = kwargs.get(
+ &quot;scp_file_list&quot;,
+ (&quot;/Users/zhifu/funasr1.0/test_local/wav.scp&quot;, &quot;/Users/zhifu/funasr1.0/test_local/text.txt&quot;),
+ )
+ if isinstance(scp_file_list, str):
+ scp_file_list = eval(scp_file_list)
+ data_type_list = kwargs.get(&quot;data_type_list&quot;, (&quot;source&quot;, &quot;target&quot;))
+ jsonl_file_out = kwargs.get(
+ &quot;jsonl_file_out&quot;, &quot;/Users/zhifu/funasr1.0/test_local/audio_datasets.jsonl&quot;
+ )
+ model_dir = kwargs.get(&quot;model_dir&quot;, &quot;iic/SenseVoiceSmall&quot;)
+ gen_jsonl_from_wav_text_list(
+ scp_file_list, data_type_list=data_type_list, jsonl_file_out=jsonl_file_out, model_dir=model_dir
+ )
+</code></pre></details></div>
+<div class="api-detail" id="e561"><span class="dtag badge-f">function</span><h2><code>gen_scp_from_jsonl(jsonl_file, jsonl_file_out, ncpu)</code></h2><div class="dmod">funasr.datasets.update_jsonl · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/datasets/audio_datasets/update_jsonl.py#L15" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Gen scp from jsonl.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>jsonl_file</code> — TODO.</li>
+<li><code>jsonl_file_out</code> — TODO.</li>
+<li><code>ncpu</code> — TODO.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code>def gen_scp_from_jsonl(jsonl_file, jsonl_file_out, ncpu):
+ &quot;&quot;&quot;Gen scp from jsonl.
+
+ Args:
+ jsonl_file: TODO.
+ jsonl_file_out: TODO.
+ ncpu: TODO.
+ &quot;&quot;&quot;
+ jsonl_file_out_f = open(jsonl_file_out, &quot;w&quot;)
+ with open(jsonl_file, encoding=&quot;utf-8&quot;) as fin:
+ lines = fin.readlines()
+
+ num_total = len(lines)
+ if ncpu &gt; 1:
+ # 使用ThreadPoolExecutor限制并发线程数
+ with ThreadPoolExecutor(max_workers=ncpu) as executor:
+ # 提交任务到线程池
+ futures = {executor.submit(update_data, lines, i) for i in tqdm(range(num_total))}
+
+ # 等待所有任务完成,这会阻塞直到所有提交的任务完成
+ for future in concurrent.futures.as_completed(futures):
+ # 这里可以添加额外的逻辑来处理完成的任务,但在这个例子中我们只是等待
+ pass
+ else:
+ for i in range(num_total):
+ update_data(lines, i)
+ logging.info(&quot;All audio durations have been processed.&quot;)
+
+ for line in lines:
+
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/datasets/audio_datasets/update_jsonl.py#L15" target="_blank" class="src-more">View full source on GitHub →</a></details></div>
+<div class="api-detail" id="e562"><span class="dtag badge-f">function</span><h2><code>update_data(lines, i)</code></h2><div class="dmod">funasr.datasets.update_jsonl · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/datasets/audio_datasets/update_jsonl.py#L51" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Update data.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>lines</code> — TODO.</li>
+<li><code>i</code> — TODO.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code>def update_data(lines, i):
+ &quot;&quot;&quot;Update data.
+
+ Args:
+ lines: TODO.
+ i: TODO.
+ &quot;&quot;&quot;
+ line = lines[i]
+ data = json.loads(line.strip())
+
+ wav_path = data[&quot;source&quot;].replace(&quot;/cpfs01&quot;, &quot;/cpfs_speech/data&quot;)
+ if os.path.exists(wav_path):
+ waveform, _ = librosa.load(wav_path, sr=16000)
+ sample_num = len(waveform)
+ source_len = int(sample_num / 16000 * 1000 / 10)
+ source_len_old = data[&quot;source_len&quot;]
+ # if (source_len_old - source_len) &gt; 100 or (source_len - source_len_old) &gt; 100:
+ # logging.info(f&quot;old: {source_len_old}, new: {source_len}, wav: {wav_path}&quot;)
+ data[&quot;source_len&quot;] = source_len
+ data[&quot;source&quot;] = wav_path
+ jsonl_line = json.dumps(data, ensure_ascii=False)
+ lines[i] = jsonl_line
+</code></pre></details></div>
+<div class="api-detail" id="e563"><span class="dtag badge-f">function</span><h2><code>update_wav_len(jsonl_file_list_in, jsonl_file_out_dir, ncpu)</code></h2><div class="dmod">funasr.datasets.update_jsonl · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/datasets/audio_datasets/update_jsonl.py#L75" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Update wav len.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>jsonl_file_list_in</code> — TODO.</li>
+<li><code>jsonl_file_out_dir</code> — TODO.</li>
+<li><code>ncpu</code> — TODO.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code>def update_wav_len(jsonl_file_list_in, jsonl_file_out_dir, ncpu=1):
+
+ &quot;&quot;&quot;Update wav len.
+
+ Args:
+ jsonl_file_list_in: TODO.
+ jsonl_file_out_dir: TODO.
+ ncpu: TODO.
+ &quot;&quot;&quot;
+ os.makedirs(jsonl_file_out_dir, exist_ok=True)
+ with open(jsonl_file_list_in, &quot;r&quot;) as f:
+ data_file_lists = f.readlines()
+
+ for i, jsonl in enumerate(data_file_lists):
+ filename_with_extension = os.path.basename(jsonl.strip())
+ jsonl_file_out = os.path.join(jsonl_file_out_dir, filename_with_extension)
+ logging.info(f&quot;{i}/{len(data_file_lists)}, jsonl: {jsonl}, {jsonl_file_out}&quot;)
+
+ gen_scp_from_jsonl(jsonl.strip(), jsonl_file_out, ncpu)
+</code></pre></details></div>
+<div class="api-detail" id="e564"><span class="dtag badge-f">function</span><h2><code>main_hydra(cfg)</code></h2><div class="dmod">funasr.datasets.update_jsonl · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/datasets/audio_datasets/update_jsonl.py#L97" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Main hydra.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>cfg</code> — Configuration overrides.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code>def main_hydra(cfg: DictConfig):
+
+ &quot;&quot;&quot;Main hydra.
+
+ Args:
+ cfg: Configuration overrides.
+ &quot;&quot;&quot;
+ kwargs = OmegaConf.to_container(cfg, resolve=True)
+ logging.info(kwargs)
+
+ jsonl_file_list_in = kwargs.get(
+ &quot;jsonl_file_list_in&quot;, &quot;/Users/zhifu/funasr1.0/data/list/data_jsonl.list&quot;
+ )
+ jsonl_file_out_dir = kwargs.get(&quot;jsonl_file_out_dir&quot;, &quot;/Users/zhifu/funasr1.0/data_tmp&quot;)
+ ncpu = kwargs.get(&quot;ncpu&quot;, 1)
+ update_wav_len(jsonl_file_list_in, jsonl_file_out_dir, ncpu)
+</code></pre></details></div>
+<div class="api-detail" id="e565"><span class="dtag badge-f">function</span><h2><code>DataloaderMapStyle(frontend, tokenizer, **kwargs)</code></h2><div class="dmod">funasr.datasets.dataloader_entry · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/datasets/dataloader_entry.py#L8" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Dataloadermapstyle.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>frontend</code> — Audio frontend for feature extraction.</li>
+<li><code>tokenizer</code> — Tokenizer instance for text encoding/decoding.</li>
+<li><code>**kwargs</code> — Additional keyword arguments.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code>def DataloaderMapStyle(frontend=None, tokenizer=None, **kwargs):
+ # dataset
+ &quot;&quot;&quot;Dataloadermapstyle.
+
+ Args:
+ frontend: Audio frontend for feature extraction.
+ tokenizer: Tokenizer instance for text encoding/decoding.
+ **kwargs: Additional keyword arguments.
+ &quot;&quot;&quot;
+ logging.info(&quot;Build dataloader&quot;)
+ dataset_class = tables.dataset_classes.get(kwargs.get(&quot;dataset&quot;, &quot;AudioDataset&quot;))
+ dataset_tr = dataset_class(
+ kwargs.get(&quot;train_data_set_list&quot;),
+ frontend=frontend,
+ tokenizer=tokenizer,
+ is_training=True,
+ **kwargs.get(&quot;dataset_conf&quot;),
+ )
+ dataset_val = dataset_class(
+ kwargs.get(&quot;valid_data_set_list&quot;),
+ frontend=frontend,
+ tokenizer=tokenizer,
+ is_training=False,
+ **kwargs.get(&quot;dataset_conf&quot;),
+ )
+
+ # dataloader
+ batch_sampler = kwargs[&quot;dataset_conf&quot;].get(&quot;batch_sampler&quot;, &quot;BatchSampler&quot;)
+ batch_sampler_val = None
+ if batch_sampler is not None:
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/datasets/dataloader_entry.py#L8" target="_blank" class="src-more">View full source on GitHub →</a></details></div>
+<div class="api-detail" id="e566"><span class="dtag badge-c">class</span><h2>DataloaderMapStyle</h2><div class="dmod">funasr.datasets.dataloader_entry · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/datasets/dataloader_entry.py#L55" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p class="muted">No documentation yet.</p></div><details class="src-block"><summary>📄 Source code</summary><pre><code>class DataloaderMapStyle:
+ def __init__(self, frontend=None, tokenizer=None, **kwargs):
+ # dataset
+ &quot;&quot;&quot;Initialize DataloaderMapStyle.
+
+ Args:
+ frontend: Audio frontend for feature extraction.
+ tokenizer: Tokenizer instance for text encoding/decoding.
+ **kwargs: Additional keyword arguments.
+ &quot;&quot;&quot;
+ logging.info(&quot;Build dataloader&quot;)
+
+ dataset_class = tables.dataset_classes.get(kwargs.get(&quot;dataset&quot;, &quot;AudioDataset&quot;))
+ dataset_tr = None
+ # split dataset
+ self.data_split_num = kwargs[&quot;dataset_conf&quot;].get(&quot;data_split_num&quot;, 1)
+ if self.data_split_num == 1:
+ dataset_tr = dataset_class(
+ kwargs.get(&quot;train_data_set_list&quot;),
+ frontend=frontend,
+ tokenizer=tokenizer,
+ is_training=True,
+ **kwargs.get(&quot;dataset_conf&quot;),
+ )
+ dataset_val = dataset_class(
+ kwargs.get(&quot;valid_data_set_list&quot;),
+ frontend=frontend,
+ tokenizer=tokenizer,
+ is_training=False,
+ **kwargs.get(&quot;dataset_conf&quot;),
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/datasets/dataloader_entry.py#L55" target="_blank" class="src-more">View full source on GitHub →</a></details><h3 class="mt">Methods</h3><div class="mblk"><code>.build_iter(epoch, data_split_i, start_step, **kwargs)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/datasets/dataloader_entry.py#L96" target="_blank" class="gh-link">L96</a><div class="mdoc"><p>Build iter.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>epoch</code> — TODO.</li>
+<li><code>data_split_i</code> — TODO.</li>
+<li><code>start_step</code> — TODO.</li>
+<li><code>**kwargs</code> — Additional keyword arguments.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def build_iter(self, epoch=0, data_split_i=0, start_step=0, **kwargs):
+
+ # reload dataset slice
+ &quot;&quot;&quot;Build iter.
+
+ Args:
+ epoch: TODO.
+ data_split_i: TODO.
+ start_step: TODO.
+ **kwargs: Additional keyword arguments.
+ &quot;&quot;&quot;
+ if self.data_split_num &gt; 1:
+ del self.dataset_tr
+ self.dataset_tr = self.dataset_class(
+ self.kwargs.get(&quot;train_data_set_list&quot;),
+ frontend=self.frontend,
+ tokenizer=self.tokenizer,
+ is_training=True,
+ **self.kwargs.get(&quot;dataset_conf&quot;),
+ data_split_i=data_split_i,
+ )
+
+ # dataloader
+ batch_sampler = self.kwargs[&quot;dataset_conf&quot;].get(&quot;batch_sampler&quot;, &quot;BatchSampler&quot;)
+ batch_sampler_val = None
+ if batch_sampler is not None:
+ batch_sampler_class = tables.batch_sampler_classes.get(batch_sampler)
+ batch_sampler = batch_sampler_class(
+ self.dataset_tr, start_step=start_step, **self.kwargs.get(&quot;dataset_conf&quot;)
+ )
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/datasets/dataloader_entry.py#L96" target="_blank" class="src-more">View full source →</a></details></div></div>
+<div class="api-detail" id="e567"><span class="dtag badge-m">method</span><h2><code>DataloaderMapStyle.build_iter(epoch, data_split_i, start_step, **kwargs)</code></h2><div class="dmod">funasr.datasets.dataloader_entry.DataloaderMapStyle · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/datasets/dataloader_entry.py#L96" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Build iter.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>epoch</code> — TODO.</li>
+<li><code>data_split_i</code> — TODO.</li>
+<li><code>start_step</code> — TODO.</li>
+<li><code>**kwargs</code> — Additional keyword arguments.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def build_iter(self, epoch=0, data_split_i=0, start_step=0, **kwargs):
+
+ # reload dataset slice
+ &quot;&quot;&quot;Build iter.
+
+ Args:
+ epoch: TODO.
+ data_split_i: TODO.
+ start_step: TODO.
+ **kwargs: Additional keyword arguments.
+ &quot;&quot;&quot;
+ if self.data_split_num &gt; 1:
+ del self.dataset_tr
+ self.dataset_tr = self.dataset_class(
+ self.kwargs.get(&quot;train_data_set_list&quot;),
+ frontend=self.frontend,
+ tokenizer=self.tokenizer,
+ is_training=True,
+ **self.kwargs.get(&quot;dataset_conf&quot;),
+ data_split_i=data_split_i,
+ )
+
+ # dataloader
+ batch_sampler = self.kwargs[&quot;dataset_conf&quot;].get(&quot;batch_sampler&quot;, &quot;BatchSampler&quot;)
+ batch_sampler_val = None
+ if batch_sampler is not None:
+ batch_sampler_class = tables.batch_sampler_classes.get(batch_sampler)
+ batch_sampler = batch_sampler_class(
+ self.dataset_tr, start_step=start_step, **self.kwargs.get(&quot;dataset_conf&quot;)
+ )
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/datasets/dataloader_entry.py#L96" target="_blank" class="src-more">View full source on GitHub →</a></details></div>
+<div class="api-detail" id="e568"><span class="dtag badge-f">function</span><h2><code>DataloaderIterable(frontend, tokenizer, **kwargs)</code></h2><div class="dmod">funasr.datasets.dataloader_entry · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/datasets/dataloader_entry.py#L143" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Dataloaderiterable.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>frontend</code> — Audio frontend for feature extraction.</li>
+<li><code>tokenizer</code> — Tokenizer instance for text encoding/decoding.</li>
+<li><code>**kwargs</code> — Additional keyword arguments.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code>def DataloaderIterable(frontend=None, tokenizer=None, **kwargs):
+ &quot;&quot;&quot;Dataloaderiterable.
+
+ Args:
+ frontend: Audio frontend for feature extraction.
+ tokenizer: Tokenizer instance for text encoding/decoding.
+ **kwargs: Additional keyword arguments.
+ &quot;&quot;&quot;
+ logging.info(&quot;Build dataloader&quot;)
+ dataset_class = tables.dataset_classes.get(kwargs.get(&quot;dataset&quot;, &quot;LargeDataset&quot;))
+ dataset_tr = dataset_class(
+ kwargs.get(&quot;train_data_set_list&quot;),
+ frontend=frontend,
+ tokenizer=tokenizer,
+ is_training=True,
+ **kwargs.get(&quot;dataset_conf&quot;),
+ )
+ dataset_val = dataset_class(
+ kwargs.get(&quot;valid_data_set_list&quot;),
+ frontend=frontend,
+ tokenizer=tokenizer,
+ is_training=False,
+ **kwargs.get(&quot;dataset_conf&quot;),
+ )
+
+ return dataset_tr, dataset_val
+</code></pre></details></div>
+<div class="api-detail" id="e569"><span class="dtag badge-c">class</span><h2>LabelSmoothingLoss</h2><div class="dmod">funasr.losses.label_smoothing_loss · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/losses/label_smoothing_loss.py#L14" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><ul>
+<li><code>Label</code> — smoothing loss.</li>
+</ul>
+<p>:param int size: the number of class</p>
+<p>:param int padding_idx: ignored class id</p>
+<p>:param float smoothing: smoothing rate (0.0 means the conventional CE)</p>
+<p>:param bool normalize_length: normalize loss by sequence length if True</p>
+<p>:param torch.nn.Module criterion: loss function to be smoothed</p>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code>class LabelSmoothingLoss(nn.Module):
+ &quot;&quot;&quot;Label-smoothing loss.
+
+ :param int size: the number of class
+ :param int padding_idx: ignored class id
+ :param float smoothing: smoothing rate (0.0 means the conventional CE)
+ :param bool normalize_length: normalize loss by sequence length if True
+ :param torch.nn.Module criterion: loss function to be smoothed
+ &quot;&quot;&quot;
+
+ def __init__(
+ self,
+ size,
+ padding_idx,
+ smoothing,
+ normalize_length=False,
+ criterion=nn.KLDivLoss(reduction=&quot;none&quot;),
+ ):
+ &quot;&quot;&quot;Construct an LabelSmoothingLoss object.&quot;&quot;&quot;
+ super(LabelSmoothingLoss, self).__init__()
+ self.criterion = criterion
+ self.padding_idx = padding_idx
+ self.confidence = 1.0 - smoothing
+ self.smoothing = smoothing
+ self.size = size
+ self.true_dist = None
+ self.normalize_length = normalize_length
+
+ def forward(self, x, target):
+ &quot;&quot;&quot;Compute loss between x and target.
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/losses/label_smoothing_loss.py#L14" target="_blank" class="src-more">View full source on GitHub →</a></details><h3 class="mt">Methods</h3><div class="mblk"><code>.forward(x, target)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/losses/label_smoothing_loss.py#L42" target="_blank" class="gh-link">L42</a><div class="mdoc"><p>Compute loss between x and target.</p>
+<p>:param torch.Tensor x: prediction (batch, seqlen, class)</p>
+<p>:param torch.Tensor target:</p>
+<p>target signal masked with self.padding_id (batch, seqlen)</p>
+<p>:return: scalar float value</p>
+<p>:rtype torch.Tensor</p>
+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def forward(self, x, target):
+ &quot;&quot;&quot;Compute loss between x and target.
+
+ :param torch.Tensor x: prediction (batch, seqlen, class)
+ :param torch.Tensor target:
+ target signal masked with self.padding_id (batch, seqlen)
+ :return: scalar float value
+ :rtype torch.Tensor
+ &quot;&quot;&quot;
+ assert x.size(2) == self.size
+ batch_size = x.size(0)
+ x = x.contiguous().view(-1, self.size)
+ target = target.contiguous().view(-1)
+ with torch.no_grad():
+ true_dist = x.clone()
+ true_dist.fill_(self.smoothing / (self.size - 1))
+ ignore = target == self.padding_idx # (B,)
+ total = len(target) - ignore.sum().item()
+ target = target.masked_fill(ignore, 0) # avoid -1 index
+ true_dist.scatter_(1, target.unsqueeze(1), self.confidence)
+ kl = self.criterion(torch.log_softmax(x, dim=1), true_dist)
+ denom = total if self.normalize_length else batch_size
+ return kl.masked_fill(ignore.unsqueeze(1), 0).sum() / denom
+</code></pre></details></div></div>
+<div class="api-detail" id="e570"><span class="dtag badge-m">method</span><h2><code>LabelSmoothingLoss.forward(x, target)</code></h2><div class="dmod">funasr.losses.label_smoothing_loss.LabelSmoothingLoss · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/losses/label_smoothing_loss.py#L42" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Compute loss between x and target.</p>
+<p>:param torch.Tensor x: prediction (batch, seqlen, class)</p>
+<p>:param torch.Tensor target:</p>
+<p>target signal masked with self.padding_id (batch, seqlen)</p>
+<p>:return: scalar float value</p>
+<p>:rtype torch.Tensor</p>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def forward(self, x, target):
+ &quot;&quot;&quot;Compute loss between x and target.
+
+ :param torch.Tensor x: prediction (batch, seqlen, class)
+ :param torch.Tensor target:
+ target signal masked with self.padding_id (batch, seqlen)
+ :return: scalar float value
+ :rtype torch.Tensor
+ &quot;&quot;&quot;
+ assert x.size(2) == self.size
+ batch_size = x.size(0)
+ x = x.contiguous().view(-1, self.size)
+ target = target.contiguous().view(-1)
+ with torch.no_grad():
+ true_dist = x.clone()
+ true_dist.fill_(self.smoothing / (self.size - 1))
+ ignore = target == self.padding_idx # (B,)
+ total = len(target) - ignore.sum().item()
+ target = target.masked_fill(ignore, 0) # avoid -1 index
+ true_dist.scatter_(1, target.unsqueeze(1), self.confidence)
+ kl = self.criterion(torch.log_softmax(x, dim=1), true_dist)
+ denom = total if self.normalize_length else batch_size
+ return kl.masked_fill(ignore.unsqueeze(1), 0).sum() / denom
+</code></pre></details></div>
+<div class="api-detail" id="e571"><span class="dtag badge-c">class</span><h2>SequenceBinaryCrossEntropy</h2><div class="dmod">funasr.losses.label_smoothing_loss · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/losses/label_smoothing_loss.py#L67" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p class="muted">No documentation yet.</p></div><details class="src-block"><summary>📄 Source code</summary><pre><code>class SequenceBinaryCrossEntropy(nn.Module):
+ def __init__(self, normalize_length=False, criterion=nn.BCEWithLogitsLoss(reduction=&quot;none&quot;)):
+ &quot;&quot;&quot;Initialize SequenceBinaryCrossEntropy.
+
+ Args:
+ normalize_length: TODO.
+ criterion: TODO.
+ &quot;&quot;&quot;
+ super().__init__()
+ self.normalize_length = normalize_length
+ self.criterion = criterion
+
+ def forward(self, pred, label, lengths):
+ &quot;&quot;&quot;Forward pass for training.
+
+ Args:
+ pred: TODO.
+ label: TODO.
+ lengths: TODO.
+ &quot;&quot;&quot;
+ pad_mask = make_pad_mask(lengths, maxlen=pred.shape[1]).to(pred.device)
+ loss = self.criterion(pred, label)
+ denom = (~pad_mask).sum() if self.normalize_length else pred.shape[0]
+ return loss.masked_fill(pad_mask.unsqueeze(-1), 0).sum() / denom
+</code></pre></details><h3 class="mt">Methods</h3><div class="mblk"><code>.forward(pred, label, lengths)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/losses/label_smoothing_loss.py#L79" target="_blank" class="gh-link">L79</a><div class="mdoc"><p>Forward pass for training.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>pred</code> — TODO.</li>
+<li><code>label</code> — TODO.</li>
+<li><code>lengths</code> — TODO.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def forward(self, pred, label, lengths):
+ &quot;&quot;&quot;Forward pass for training.
+
+ Args:
+ pred: TODO.
+ label: TODO.
+ lengths: TODO.
+ &quot;&quot;&quot;
+ pad_mask = make_pad_mask(lengths, maxlen=pred.shape[1]).to(pred.device)
+ loss = self.criterion(pred, label)
+ denom = (~pad_mask).sum() if self.normalize_length else pred.shape[0]
+ return loss.masked_fill(pad_mask.unsqueeze(-1), 0).sum() / denom
+</code></pre></details></div></div>
+<div class="api-detail" id="e572"><span class="dtag badge-m">method</span><h2><code>SequenceBinaryCrossEntropy.forward(pred, label, lengths)</code></h2><div class="dmod">funasr.losses.label_smoothing_loss.SequenceBinaryCrossEntropy · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/losses/label_smoothing_loss.py#L79" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Forward pass for training.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>pred</code> — TODO.</li>
+<li><code>label</code> — TODO.</li>
+<li><code>lengths</code> — TODO.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def forward(self, pred, label, lengths):
+ &quot;&quot;&quot;Forward pass for training.
+
+ Args:
+ pred: TODO.
+ label: TODO.
+ lengths: TODO.
+ &quot;&quot;&quot;
+ pad_mask = make_pad_mask(lengths, maxlen=pred.shape[1]).to(pred.device)
+ loss = self.criterion(pred, label)
+ denom = (~pad_mask).sum() if self.normalize_length else pred.shape[0]
+ return loss.masked_fill(pad_mask.unsqueeze(-1), 0).sum() / denom
+</code></pre></details></div>
+<div class="api-detail" id="e573"><span class="dtag badge-c">class</span><h2>NllLoss</h2><div class="dmod">funasr.losses.label_smoothing_loss · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/losses/label_smoothing_loss.py#L93" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Nll loss.</p>
+<p>:param int size: the number of class</p>
+<p>:param int padding_idx: ignored class id</p>
+<p>:param bool normalize_length: normalize loss by sequence length if True</p>
+<p>:param torch.nn.Module criterion: loss function</p>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code>class NllLoss(nn.Module):
+ &quot;&quot;&quot;Nll loss.
+
+ :param int size: the number of class
+ :param int padding_idx: ignored class id
+ :param bool normalize_length: normalize loss by sequence length if True
+ :param torch.nn.Module criterion: loss function
+ &quot;&quot;&quot;
+
+ def __init__(
+ self,
+ size,
+ padding_idx,
+ normalize_length=False,
+ criterion=nn.NLLLoss(reduction=&quot;none&quot;),
+ ):
+ &quot;&quot;&quot;Construct an NllLoss object.&quot;&quot;&quot;
+ super(NllLoss, self).__init__()
+ self.criterion = criterion
+ self.padding_idx = padding_idx
+ self.size = size
+ self.true_dist = None
+ self.normalize_length = normalize_length
+
+ def forward(self, x, target):
+ &quot;&quot;&quot;Compute loss between x and target.
+
+ :param torch.Tensor x: prediction (batch, seqlen, class)
+ :param torch.Tensor target:
+ target signal masked with self.padding_id (batch, seqlen)
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/losses/label_smoothing_loss.py#L93" target="_blank" class="src-more">View full source on GitHub →</a></details><h3 class="mt">Methods</h3><div class="mblk"><code>.forward(x, target)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/losses/label_smoothing_loss.py#L117" target="_blank" class="gh-link">L117</a><div class="mdoc"><p>Compute loss between x and target.</p>
+<p>:param torch.Tensor x: prediction (batch, seqlen, class)</p>
+<p>:param torch.Tensor target:</p>
+<p>target signal masked with self.padding_id (batch, seqlen)</p>
+<p>:return: scalar float value</p>
+<p>:rtype torch.Tensor</p>
+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def forward(self, x, target):
+ &quot;&quot;&quot;Compute loss between x and target.
+
+ :param torch.Tensor x: prediction (batch, seqlen, class)
+ :param torch.Tensor target:
+ target signal masked with self.padding_id (batch, seqlen)
+ :return: scalar float value
+ :rtype torch.Tensor
+ &quot;&quot;&quot;
+ assert x.size(2) == self.size
+ batch_size = x.size(0)
+ x = x.view(-1, self.size)
+ target = target.view(-1)
+ with torch.no_grad():
+ ignore = target == self.padding_idx # (B,)
+ total = len(target) - ignore.sum().item()
+ target = target.masked_fill(ignore, 0) # avoid -1 index
+ kl = self.criterion(x, target)
+ denom = total if self.normalize_length else batch_size
+ return kl.masked_fill(ignore, 0).sum() / denom
+</code></pre></details></div></div>
+<div class="api-detail" id="e574"><span class="dtag badge-m">method</span><h2><code>NllLoss.forward(x, target)</code></h2><div class="dmod">funasr.losses.label_smoothing_loss.NllLoss · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/losses/label_smoothing_loss.py#L117" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Compute loss between x and target.</p>
+<p>:param torch.Tensor x: prediction (batch, seqlen, class)</p>
+<p>:param torch.Tensor target:</p>
+<p>target signal masked with self.padding_id (batch, seqlen)</p>
+<p>:return: scalar float value</p>
+<p>:rtype torch.Tensor</p>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def forward(self, x, target):
+ &quot;&quot;&quot;Compute loss between x and target.
+
+ :param torch.Tensor x: prediction (batch, seqlen, class)
+ :param torch.Tensor target:
+ target signal masked with self.padding_id (batch, seqlen)
+ :return: scalar float value
+ :rtype torch.Tensor
+ &quot;&quot;&quot;
+ assert x.size(2) == self.size
+ batch_size = x.size(0)
+ x = x.view(-1, self.size)
+ target = target.view(-1)
+ with torch.no_grad():
+ ignore = target == self.padding_idx # (B,)
+ total = len(target) - ignore.sum().item()
+ target = target.masked_fill(ignore, 0) # avoid -1 index
+ kl = self.criterion(x, target)
+ denom = total if self.normalize_length else batch_size
+ return kl.masked_fill(ignore, 0).sum() / denom
+</code></pre></details></div>
+<div class="api-detail" id="e575"><span class="dtag badge-c">class</span><h2>CustomLambdaLR</h2><div class="dmod">funasr.schedulers.lambdalr_cus · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/schedulers/lambdalr_cus.py#L19" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p class="muted">No documentation yet.</p></div><details class="src-block"><summary>📄 Source code</summary><pre><code>class CustomLambdaLR(_LRScheduler):
+ def __init__(
+ self,
+ optimizer,
+ warmup_steps: int = 25000,
+ total_steps: int = 500000,
+ last_epoch=-1,
+ verbose=False,
+ ):
+ &quot;&quot;&quot;Initialize CustomLambdaLR.
+
+ Args:
+ optimizer: TODO.
+ warmup_steps: TODO.
+ total_steps: TODO.
+ last_epoch: TODO.
+ verbose: TODO.
+ &quot;&quot;&quot;
+ self.warmup_steps = warmup_steps
+ self.total_steps = total_steps
+ super().__init__(optimizer, last_epoch, verbose)
+
+ def get_lr(self):
+
+ &quot;&quot;&quot;Get lr.&quot;&quot;&quot;
+ step = self.last_epoch + 1
+ if step &lt; self.warmup_steps:
+ lr_scale = step / self.warmup_steps
+ else:
+ lr_scale = max(
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/schedulers/lambdalr_cus.py#L19" target="_blank" class="src-more">View full source on GitHub →</a></details><h3 class="mt">Methods</h3><div class="mblk"><code>.get_lr()</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/schedulers/lambdalr_cus.py#L41" target="_blank" class="gh-link">L41</a><div class="mdoc"><p>Get lr.</p>
+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def get_lr(self):
+
+ &quot;&quot;&quot;Get lr.&quot;&quot;&quot;
+ step = self.last_epoch + 1
+ if step &lt; self.warmup_steps:
+ lr_scale = step / self.warmup_steps
+ else:
+ lr_scale = max(
+ 0.0, 1 - (step - self.warmup_steps) / (self.total_steps - self.warmup_steps)
+ )
+ return [base_lr * lr_scale for base_lr in self.base_lrs]
+</code></pre></details></div></div>
+<div class="api-detail" id="e576"><span class="dtag badge-m">method</span><h2><code>CustomLambdaLR.get_lr()</code></h2><div class="dmod">funasr.schedulers.lambdalr_cus.CustomLambdaLR · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/schedulers/lambdalr_cus.py#L41" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Get lr.</p>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def get_lr(self):
+
+ &quot;&quot;&quot;Get lr.&quot;&quot;&quot;
+ step = self.last_epoch + 1
+ if step &lt; self.warmup_steps:
+ lr_scale = step / self.warmup_steps
+ else:
+ lr_scale = max(
+ 0.0, 1 - (step - self.warmup_steps) / (self.total_steps - self.warmup_steps)
+ )
+ return [base_lr * lr_scale for base_lr in self.base_lrs]
+</code></pre></details></div>
+<div class="api-detail" id="e577"><span class="dtag badge-c">class</span><h2>NoamLR</h2><div class="dmod">funasr.schedulers.noam_lr · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/schedulers/noam_lr.py#L12" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>The LR scheduler proposed by Noam</p>
+<p>Ref:</p>
+<p>&quot;Attention Is All You Need&quot;, https://arxiv.org/pdf/1706.03762.pdf</p>
+<ul>
+<li><code>FIXME(kamo)</code> — PyTorch doesn&#x27;t provide _LRScheduler as public class,</li>
+</ul>
+<p>thus the behaviour isn&#x27;t guaranteed at forward PyTorch version.</p>
+<ul>
+<li><code>NOTE(kamo)</code> — The &quot;model_size&quot; in original implementation is derived from</li>
+</ul>
+<p>the model, but in this implementation, this parameter is a constant value.</p>
+<p>You need to change it if the model is changed.</p>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code>class NoamLR(_LRScheduler, AbsBatchStepScheduler):
+ &quot;&quot;&quot;The LR scheduler proposed by Noam
+
+ Ref:
+ &quot;Attention Is All You Need&quot;, https://arxiv.org/pdf/1706.03762.pdf
+
+ FIXME(kamo): PyTorch doesn&#x27;t provide _LRScheduler as public class,
+ thus the behaviour isn&#x27;t guaranteed at forward PyTorch version.
+
+ NOTE(kamo): The &quot;model_size&quot; in original implementation is derived from
+ the model, but in this implementation, this parameter is a constant value.
+ You need to change it if the model is changed.
+
+ &quot;&quot;&quot;
+
+ def __init__(
+ self,
+ optimizer: torch.optim.Optimizer,
+ model_size: Union[int, float] = 320,
+ warmup_steps: Union[int, float] = 25000,
+ last_epoch: int = -1,
+ ):
+ &quot;&quot;&quot;Initialize NoamLR.
+
+ Args:
+ optimizer: TODO.
+ model_size: Size/dimension parameter.
+ warmup_steps: TODO.
+ last_epoch: TODO.
+ &quot;&quot;&quot;
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/schedulers/noam_lr.py#L12" target="_blank" class="src-more">View full source on GitHub →</a></details><h3 class="mt">Methods</h3><div class="mblk"><code>.lr_for_WarmupLR(lr)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/schedulers/noam_lr.py#L56" target="_blank" class="gh-link">L56</a><div class="mdoc"><p>Lr for warmuplr.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>lr</code> — TODO.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def lr_for_WarmupLR(self, lr: float) -&gt; float:
+ &quot;&quot;&quot;Lr for warmuplr.
+
+ Args:
+ lr: TODO.
+ &quot;&quot;&quot;
+ return lr / self.model_size**0.5 / self.warmup_steps**0.5
+</code></pre></details></div><div class="mblk"><code>.get_lr()</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/schedulers/noam_lr.py#L71" target="_blank" class="gh-link">L71</a><div class="mdoc"><p>Get lr.</p>
+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def get_lr(self):
+ &quot;&quot;&quot;Get lr.&quot;&quot;&quot;
+ step_num = self.last_epoch + 1
+ return [
+ lr * self.model_size**-0.5 * min(step_num**-0.5, step_num * self.warmup_steps**-1.5)
+ for lr in self.base_lrs
+ ]
+</code></pre></details></div></div>
+<div class="api-detail" id="e578"><span class="dtag badge-m">method</span><h2><code>NoamLR.lr_for_WarmupLR(lr)</code></h2><div class="dmod">funasr.schedulers.noam_lr.NoamLR · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/schedulers/noam_lr.py#L56" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Lr for warmuplr.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>lr</code> — TODO.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def lr_for_WarmupLR(self, lr: float) -&gt; float:
+ &quot;&quot;&quot;Lr for warmuplr.
+
+ Args:
+ lr: TODO.
+ &quot;&quot;&quot;
+ return lr / self.model_size**0.5 / self.warmup_steps**0.5
+</code></pre></details></div>
+<div class="api-detail" id="e579"><span class="dtag badge-m">method</span><h2><code>NoamLR.get_lr()</code></h2><div class="dmod">funasr.schedulers.noam_lr.NoamLR · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/schedulers/noam_lr.py#L71" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Get lr.</p>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def get_lr(self):
+ &quot;&quot;&quot;Get lr.&quot;&quot;&quot;
+ step_num = self.last_epoch + 1
+ return [
+ lr * self.model_size**-0.5 * min(step_num**-0.5, step_num * self.warmup_steps**-1.5)
+ for lr in self.base_lrs
+ ]
+</code></pre></details></div>
+<div class="api-detail" id="e580"><span class="dtag badge-c">class</span><h2>TriStageLR</h2><div class="dmod">funasr.schedulers.tri_stage_scheduler · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/schedulers/tri_stage_scheduler.py#L15" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p class="muted">No documentation yet.</p></div><details class="src-block"><summary>📄 Source code</summary><pre><code>class TriStageLR(_LRScheduler, AbsBatchStepScheduler):
+ def __init__(
+ self,
+ optimizer: torch.optim.Optimizer,
+ last_epoch: int = -1,
+ phase_ratio: Optional[List[float]] = None,
+ init_lr_scale: float = 0.01,
+ final_lr_scale: float = 0.01,
+ ):
+ &quot;&quot;&quot;Initialize TriStageLR.
+
+ Args:
+ optimizer: TODO.
+ last_epoch: TODO.
+ phase_ratio: TODO.
+ init_lr_scale: TODO.
+ final_lr_scale: TODO.
+ &quot;&quot;&quot;
+ self.optimizer = optimizer
+ self.last_epoch = last_epoch
+ self.phase_ratio = phase_ratio
+ self.init_lr_scale = init_lr_scale
+ self.final_lr_scale = final_lr_scale
+ self.optimizer_lr = self.optimizer.defaults[&quot;lr&quot;]
+
+ def init_tri_stage_scheudler(self, max_update):
+ &quot;&quot;&quot;Init tri stage scheudler.
+
+ Args:
+ max_update: TODO.
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/schedulers/tri_stage_scheduler.py#L15" target="_blank" class="src-more">View full source on GitHub →</a></details><h3 class="mt">Methods</h3><div class="mblk"><code>.init_tri_stage_scheudler(max_update)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/schedulers/tri_stage_scheduler.py#L40" target="_blank" class="gh-link">L40</a><div class="mdoc"><p>Init tri stage scheudler.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>max_update</code> — TODO.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def init_tri_stage_scheudler(self, max_update):
+ &quot;&quot;&quot;Init tri stage scheudler.
+
+ Args:
+ max_update: TODO.
+ &quot;&quot;&quot;
+ self.max_update = max_update
+ self.peak_lr = self.optimizer_lr
+ self.init_lr = self.init_lr_scale * self.optimizer_lr
+ self.final_lr = self.final_lr_scale * self.optimizer_lr
+
+ assert self.max_update &gt; 0
+ assert sum(self.phase_ratio) == 1, &quot;phase ratios must add up to 1&quot;
+ assert len(self.phase_ratio) == 3
+ self.warmup_steps = int(self.max_update * self.phase_ratio[0])
+ self.hold_steps = int(self.max_update * self.phase_ratio[1])
+ self.decay_steps = int(self.max_update * self.phase_ratio[2])
+
+ self.warmup_rate = (
+ (self.peak_lr - self.init_lr) / self.warmup_steps if self.warmup_steps != 0 else 0
+ )
+ self.decay_factor = -math.log(self.final_lr_scale) / self.decay_steps
+
+ # initial learning rate
+ self.lr = self.init_lr
+
+ # __init__() must be invoked before setting field
+ # because step() is also invoked in __init__()
+ self.set_optimizer_lr(self.lr)
+ super().__init__(self.optimizer, self.last_epoch)
+</code></pre></details></div><div class="mblk"><code>.step_update(num_updates)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/schedulers/tri_stage_scheduler.py#L96" target="_blank" class="gh-link">L96</a><div class="mdoc"><p>Update the learning rate after each update.</p>
+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def step_update(self, num_updates):
+ &quot;&quot;&quot;Update the learning rate after each update.&quot;&quot;&quot;
+ stage, steps_in_stage = self._decide_stage(num_updates)
+ if stage == 0:
+ self.lr = self.init_lr + self.warmup_rate * steps_in_stage
+ elif stage == 1:
+ self.lr = self.peak_lr
+ elif stage == 2:
+ self.lr = self.peak_lr * math.exp(-self.decay_factor * steps_in_stage)
+ elif stage == 3:
+ self.lr = self.final_lr
+ else:
+ raise ValueError(&quot;Undefined stage&quot;)
+ self.set_optimizer_lr(self.lr)
+</code></pre></details></div><div class="mblk"><code>.set_optimizer_lr(lr)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/schedulers/tri_stage_scheduler.py#L111" target="_blank" class="gh-link">L111</a><div class="mdoc"><p>Set optimizer lr.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>lr</code> — TODO.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def set_optimizer_lr(self, lr):
+ &quot;&quot;&quot;Set optimizer lr.
+
+ Args:
+ lr: TODO.
+ &quot;&quot;&quot;
+ for param_group in self.optimizer.param_groups:
+ param_group[&quot;lr&quot;] = lr
+</code></pre></details></div><div class="mblk"><code>.get_lr()</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/schedulers/tri_stage_scheduler.py#L120" target="_blank" class="gh-link">L120</a><div class="mdoc"><p>Get lr.</p>
+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def get_lr(self):
+ &quot;&quot;&quot;Get lr.&quot;&quot;&quot;
+ step_num = self.last_epoch + 1
+ self.step_update(step_num)
+ return [self.lr]
+</code></pre></details></div></div>
+<div class="api-detail" id="e581"><span class="dtag badge-m">method</span><h2><code>TriStageLR.init_tri_stage_scheudler(max_update)</code></h2><div class="dmod">funasr.schedulers.tri_stage_scheduler.TriStageLR · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/schedulers/tri_stage_scheduler.py#L40" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Init tri stage scheudler.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>max_update</code> — TODO.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def init_tri_stage_scheudler(self, max_update):
+ &quot;&quot;&quot;Init tri stage scheudler.
+
+ Args:
+ max_update: TODO.
+ &quot;&quot;&quot;
+ self.max_update = max_update
+ self.peak_lr = self.optimizer_lr
+ self.init_lr = self.init_lr_scale * self.optimizer_lr
+ self.final_lr = self.final_lr_scale * self.optimizer_lr
+
+ assert self.max_update &gt; 0
+ assert sum(self.phase_ratio) == 1, &quot;phase ratios must add up to 1&quot;
+ assert len(self.phase_ratio) == 3
+ self.warmup_steps = int(self.max_update * self.phase_ratio[0])
+ self.hold_steps = int(self.max_update * self.phase_ratio[1])
+ self.decay_steps = int(self.max_update * self.phase_ratio[2])
+
+ self.warmup_rate = (
+ (self.peak_lr - self.init_lr) / self.warmup_steps if self.warmup_steps != 0 else 0
+ )
+ self.decay_factor = -math.log(self.final_lr_scale) / self.decay_steps
+
+ # initial learning rate
+ self.lr = self.init_lr
+
+ # __init__() must be invoked before setting field
+ # because step() is also invoked in __init__()
+ self.set_optimizer_lr(self.lr)
+ super().__init__(self.optimizer, self.last_epoch)
+</code></pre></details></div>
+<div class="api-detail" id="e582"><span class="dtag badge-m">method</span><h2><code>TriStageLR.step_update(num_updates)</code></h2><div class="dmod">funasr.schedulers.tri_stage_scheduler.TriStageLR · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/schedulers/tri_stage_scheduler.py#L96" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Update the learning rate after each update.</p>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def step_update(self, num_updates):
+ &quot;&quot;&quot;Update the learning rate after each update.&quot;&quot;&quot;
+ stage, steps_in_stage = self._decide_stage(num_updates)
+ if stage == 0:
+ self.lr = self.init_lr + self.warmup_rate * steps_in_stage
+ elif stage == 1:
+ self.lr = self.peak_lr
+ elif stage == 2:
+ self.lr = self.peak_lr * math.exp(-self.decay_factor * steps_in_stage)
+ elif stage == 3:
+ self.lr = self.final_lr
+ else:
+ raise ValueError(&quot;Undefined stage&quot;)
+ self.set_optimizer_lr(self.lr)
+</code></pre></details></div>
+<div class="api-detail" id="e583"><span class="dtag badge-m">method</span><h2><code>TriStageLR.set_optimizer_lr(lr)</code></h2><div class="dmod">funasr.schedulers.tri_stage_scheduler.TriStageLR · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/schedulers/tri_stage_scheduler.py#L111" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Set optimizer lr.</p>
+<h4>Args:</h4>
+<ul>
+<li><code>lr</code> — TODO.</li>
+</ul>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def set_optimizer_lr(self, lr):
+ &quot;&quot;&quot;Set optimizer lr.
+
+ Args:
+ lr: TODO.
+ &quot;&quot;&quot;
+ for param_group in self.optimizer.param_groups:
+ param_group[&quot;lr&quot;] = lr
+</code></pre></details></div>
+<div class="api-detail" id="e584"><span class="dtag badge-m">method</span><h2><code>TriStageLR.get_lr()</code></h2><div class="dmod">funasr.schedulers.tri_stage_scheduler.TriStageLR · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/schedulers/tri_stage_scheduler.py#L120" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Get lr.</p>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def get_lr(self):
+ &quot;&quot;&quot;Get lr.&quot;&quot;&quot;
+ step_num = self.last_epoch + 1
+ self.step_update(step_num)
+ return [self.lr]
+</code></pre></details></div>
+<div class="api-detail" id="e585"><span class="dtag badge-c">class</span><h2>WarmupLR</h2><div class="dmod">funasr.schedulers.warmup_lr · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/schedulers/warmup_lr.py#L11" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>The WarmupLR scheduler</p>
+<p>This scheduler is almost same as NoamLR Scheduler except for following difference:</p>
+<p>NoamLR:</p>
+<p>lr = optimizer.lr * model_size ** -0.5</p>
+<p>* min(step ** -0.5, step * warmup_step ** -1.5)</p>
+<p>WarmupLR:</p>
+<p>lr = optimizer.lr * warmup_step ** 0.5</p>
+<p>* min(step ** -0.5, step * warmup_step ** -1.5)</p>
+<p>Note that the maximum lr equals to optimizer.lr in this scheduler.</p>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code>class WarmupLR(_LRScheduler, AbsBatchStepScheduler):
+ &quot;&quot;&quot;The WarmupLR scheduler
+
+ This scheduler is almost same as NoamLR Scheduler except for following difference:
+
+ NoamLR:
+ lr = optimizer.lr * model_size ** -0.5
+ * min(step ** -0.5, step * warmup_step ** -1.5)
+ WarmupLR:
+ lr = optimizer.lr * warmup_step ** 0.5
+ * min(step ** -0.5, step * warmup_step ** -1.5)
+
+ Note that the maximum lr equals to optimizer.lr in this scheduler.
+
+ &quot;&quot;&quot;
+
+ def __init__(
+ self,
+ optimizer: torch.optim.Optimizer,
+ warmup_steps: Union[int, float] = 25000,
+ last_epoch: int = -1,
+ ):
+ &quot;&quot;&quot;Initialize WarmupLR.
+
+ Args:
+ optimizer: TODO.
+ warmup_steps: TODO.
+ last_epoch: TODO.
+ &quot;&quot;&quot;
+ self.warmup_steps = warmup_steps
+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/schedulers/warmup_lr.py#L11" target="_blank" class="src-more">View full source on GitHub →</a></details><h3 class="mt">Methods</h3><div class="mblk"><code>.get_lr()</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/schedulers/warmup_lr.py#L50" target="_blank" class="gh-link">L50</a><div class="mdoc"><p>Get lr.</p>
+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def get_lr(self):
+ &quot;&quot;&quot;Get lr.&quot;&quot;&quot;
+ step_num = self.last_epoch + 1
+ return [
+ lr * self.warmup_steps**0.5 * min(step_num**-0.5, step_num * self.warmup_steps**-1.5)
+ for lr in self.base_lrs
+ ]
+</code></pre></details></div></div>
+<div class="api-detail" id="e586"><span class="dtag badge-m">method</span><h2><code>WarmupLR.get_lr()</code></h2><div class="dmod">funasr.schedulers.warmup_lr.WarmupLR · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/schedulers/warmup_lr.py#L50" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Get lr.</p>
+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def get_lr(self):
+ &quot;&quot;&quot;Get lr.&quot;&quot;&quot;
+ step_num = self.last_epoch + 1
+ return [
+ lr * self.warmup_steps**0.5 * min(step_num**-0.5, step_num * self.warmup_steps**-1.5)
+ for lr in self.base_lrs
+ ]
+</code></pre></details></div>
+
+</div></div>
+<script>
+function toggleL1(el){el.closest('.l1').classList.toggle('open')}
+function toggleL2(el){el.closest('.l2').classList.toggle('open')}
+function showEntry(id){
+document.querySelectorAll('.api-detail').forEach(e=>e.classList.remove('active'));
+document.querySelectorAll('.l3-item').forEach(e=>e.classList.remove('active'));
+document.getElementById('api-welcome').style.display='none';
+var t=document.getElementById(id);if(t)t.classList.add('active');
+var l=document.querySelector('[data-target="'+id+'"]');if(l)l.classList.add('active');
+document.querySelector('.api-content').scrollTop=0;
+}
+function filterSidebar(q){
+q=q.toLowerCase();
+document.querySelectorAll('.l3-item').forEach(e=>e.classList.toggle('hidden',q&&!e.textContent.toLowerCase().includes(q)));
+if(q)document.querySelectorAll('.l1,.l2').forEach(e=>e.classList.add('open'));
+}
+if(window.location.hash)showEntry(window.location.hash.slice(1));
+</script></body></html>
\ No newline at end of file
diff --git a/training.html b/training.html
new file mode 100644
index 000000000..86d44b152
--- /dev/null
+++ b/training.html
@@ -0,0 +1,348 @@
+<!DOCTYPE html>
+<html lang="en"><head><meta charset="UTF-8"><meta name="viewport" content="width=device-width,initial-scale=1.0">
+<title>FunASR Training Guide</title>
+<link href="https://fonts.googleapis.com/css2?family=Inter:wght@400;500;600;700;800&family=JetBrains+Mono:wght@400;500&display=swap" rel="stylesheet">
+<link rel="stylesheet" href="style.css">
+<style>
+.content{padding:48px 0 80px}.container.narrow{max-width:820px;margin:0 auto;padding:0 24px}
+h1{font-size:2.2rem;font-weight:800;letter-spacing:-0.5px;margin-bottom:8px}
+h1+p{color:var(--muted);margin-bottom:32px;font-size:1.05rem}
+h2{font-size:1.4rem;font-weight:700;margin:56px 0 16px;padding-bottom:10px;border-bottom:2px solid var(--line)}
+h3{font-size:1.1rem;font-weight:600;margin:32px 0 12px}
+h4{font-size:0.95rem;font-weight:600;margin:24px 0 8px;color:var(--ink-soft)}
+p{margin-bottom:12px;line-height:1.75;font-size:0.92rem}
+ul,ol{margin:12px 0 12px 24px}li{margin-bottom:8px;line-height:1.6;font-size:0.9rem}
+pre{background:var(--code-bg,#101828);color:#e2e8f0;border-radius:8px;padding:18px 22px;overflow-x:auto;margin:14px 0 22px;font-size:0.82rem;line-height:1.55}
+code{font-family:'JetBrains Mono',monospace}
+p code,li code,td code{background:#f1f5f9;padding:2px 6px;border-radius:4px;font-size:0.82rem;color:#b91c1c}
+.note{background:#eff6ff;border-left:4px solid #2563eb;padding:14px 18px;border-radius:0 8px 8px 0;margin:16px 0;font-size:0.88rem}
+.warning{background:#fffbeb;border-left:4px solid #f59e0b;padding:14px 18px;border-radius:0 8px 8px 0;margin:16px 0;font-size:0.88rem}
+.tip{background:#f0fdf4;border-left:4px solid #059669;padding:14px 18px;border-radius:0 8px 8px 0;margin:16px 0;font-size:0.88rem}
+table{width:100%;border-collapse:collapse;margin:16px 0;font-size:0.88rem}
+th,td{padding:10px 14px;text-align:left;border-bottom:1px solid var(--line)}
+th{background:#f8fafc;font-weight:600;font-size:0.78rem;text-transform:uppercase;letter-spacing:0.3px;color:var(--muted)}
+.toc-grid{display:grid;grid-template-columns:repeat(auto-fit,minmax(180px,1fr));gap:8px;margin:20px 0 40px}
+.toc-grid a{display:block;padding:10px 14px;background:var(--page,#f7f9fc);border:1px solid var(--line);border-radius:8px;text-decoration:none;color:var(--ink-soft);font-size:0.82rem;font-weight:500;transition:all 0.15s}
+.toc-grid a:hover{border-color:#2563eb;color:#2563eb}
+.output-box{background:#f8fafc;border:1px solid var(--line);border-radius:8px;padding:14px 18px;margin:12px 0;font-family:'JetBrains Mono',monospace;font-size:0.76rem;line-height:1.6;overflow-x:auto}
+</style>
+</head><body>
+<nav class="nav"><div class="container">
+<a href="index.html" class="nav-logo">FunASR</a>
+<div class="nav-links">
+<a href="index.html">Home</a>
+<a href="tutorial.html">Tutorial</a>
+<a href="training.html" class="active">Training</a>
+<a href="model-registration.html">Develop</a>
+<a href="api.html">API</a>
+</div>
+<a href="https://github.com/modelscope/FunASR" class="nav-github">GitHub</a>
+</div></nav>
+
+<div class="content"><div class="container narrow">
+<h1>Training & Fine-tuning</h1>
+<p>Fine-tune pretrained models on your own data using FunASR's training framework.</p>
+
+<div class="toc-grid">
+<a href="#overview">Overview</a>
+<a href="#data">Data Preparation</a>
+<a href="#paraformer">Fine-tune Paraformer</a>
+<a href="#sensevoice">Fine-tune SenseVoice</a>
+<a href="#nano">Fine-tune Fun-ASR-Nano</a>
+<a href="#params">Parameter Reference</a>
+<a href="#multi-gpu">Multi-GPU Training</a>
+<a href="#deepspeed">DeepSpeed</a>
+<a href="#monitor">Monitoring</a>
+<a href="#inference-after">Use Fine-tuned Model</a>
+<a href="#tips">Tips & Troubleshooting</a>
+</div>
+
+<!-- Overview -->
+<h2 id="overview">Overview</h2>
+<p>FunASR's training framework supports:</p>
+<ul>
+<li><strong>Fine-tuning</strong> any pretrained model on custom domain data</li>
+<li><strong>Multi-GPU</strong> training with PyTorch DDP (single/multi-node)</li>
+<li><strong>DeepSpeed</strong> ZeRO Stage 1/2/3 for large model training</li>
+<li><strong>Dynamic batching</strong> by token count or example count</li>
+<li><strong>Checkpoint averaging</strong> for best performance</li>
+<li><strong>Resume training</strong> from interruption</li>
+</ul>
+
+<p>The training entry point is <code>funasr-train-ds</code> (or <code>funasr/bin/train_ds.py</code>), launched via <code>torchrun</code> for distributed training.</p>
+
+<!-- Data -->
+<h2 id="data">Data Preparation</h2>
+
+<h3>Standard Format (Paraformer, SenseVoice)</h3>
+<p>Training data uses JSONL format — one JSON object per line:</p>
+<pre>{"key": "utt001", "source": "/path/to/audio.wav", "source_len": 90, "target": "这是转写文本", "target_len": 6}
+{"key": "utt002", "source": "/path/to/audio2.wav", "source_len": 150, "target": "hello world", "target_len": 2}</pre>
+
+<table>
+<tr><th>Field</th><th>Type</th><th>Description</th></tr>
+<tr><td><code>key</code></td><td>str</td><td>Unique utterance ID</td></tr>
+<tr><td><code>source</code></td><td>str</td><td>Audio file path (local path or URL)</td></tr>
+<tr><td><code>source_len</code></td><td>int</td><td>Audio length in fbank frames (1 frame = 10ms)</td></tr>
+<tr><td><code>target</code></td><td>str</td><td>Transcription text</td></tr>
+<tr><td><code>target_len</code></td><td>int</td><td>Number of text tokens</td></tr>
+</table>
+
+<h4>Generate from wav.scp + text.txt</h4>
+<p>If you have Kaldi-style data files, convert them:</p>
+<pre># train_wav.scp (tab-separated: id \t path)
+utt001 /data/audio/001.wav
+utt002 /data/audio/002.wav
+
+# train_text.txt (tab-separated: id \t text)
+utt001 这是转写文本
+utt002 hello world</pre>
+
+<pre># Convert to jsonl
+scp2jsonl \
+ ++scp_file_list='["/data/list/train_wav.scp", "/data/list/train_text.txt"]' \
+ ++data_type_list='["source", "target"]' \
+ ++jsonl_file_out="/data/list/train.jsonl"</pre>
+
+<h3>ChatML Format (Fun-ASR-Nano)</h3>
+<p>Fun-ASR-Nano uses ChatML conversation format:</p>
+<pre>{"messages": [
+ {"role": "system", "content": "You are a helpful assistant."},
+ {"role": "user", "content": "语音转写:<|startofspeech|>!/path/to/audio.wav<|endofspeech|>"},
+ {"role": "assistant", "content": "几点了?"}
+], "speech_length": 145, "text_length": 3}</pre>
+
+<table>
+<tr><th>Field</th><th>Description</th></tr>
+<tr><td><code>messages[0]</code></td><td>System prompt (fixed: "You are a helpful assistant.")</td></tr>
+<tr><td><code>messages[1]</code></td><td>User: prompt + audio path wrapped in <code>&lt;|startofspeech|&gt;!...&lt;|endofspeech|&gt;</code></td></tr>
+<tr><td><code>messages[2]</code></td><td>Assistant: transcription text</td></tr>
+<tr><td><code>speech_length</code></td><td>Number of fbank frames (10ms each)</td></tr>
+<tr><td><code>text_length</code></td><td>Number of tokens (tokenized by Qwen3-0.6B)</td></tr>
+</table>
+
+<div class="tip"><strong>Prompt variations:</strong><br>
+• Chinese: <code>语音转写:</code><br>
+• English: <code>Speech transcription:</code><br>
+• Cross-language: <code>语音转写成英文:</code><br>
+• No ITN: <code>语音转写,不进行文本规整:</code></div>
+
+<p>Convert from wav.scp + text.txt:</p>
+<pre>python tools/scp2jsonl.py \
+ ++scp_file=data/train_wav.scp \
+ ++transcript_file=data/train_text.txt \
+ ++jsonl_file=data/train_example.jsonl</pre>
+
+<!-- Paraformer -->
+<h2 id="paraformer">Fine-tune Paraformer</h2>
+<pre>cd examples/industrial_data_pretraining/paraformer
+bash finetune.sh</pre>
+
+<p>Or customize the key parameters:</p>
+<pre>export CUDA_VISIBLE_DEVICES="0,1"
+gpu_num=2
+
+torchrun --nproc_per_node $gpu_num \
+ funasr/bin/train_ds.py \
+ ++model="iic/speech_paraformer-large_asr_nat-zh-cn-16k-common-vocab8404-pytorch" \
+ ++train_data_set_list="data/train.jsonl" \
+ ++valid_data_set_list="data/val.jsonl" \
+ ++dataset_conf.batch_size=6000 \
+ ++dataset_conf.batch_type="token" \
+ ++dataset_conf.num_workers=4 \
+ ++train_conf.max_epoch=50 \
+ ++train_conf.validate_interval=2000 \
+ ++train_conf.save_checkpoint_interval=2000 \
+ ++train_conf.keep_nbest_models=20 \
+ ++train_conf.avg_nbest_model=10 \
+ ++optim_conf.lr=0.0002 \
+ ++output_dir="./outputs"</pre>
+
+<!-- SenseVoice -->
+<h2 id="sensevoice">Fine-tune SenseVoice</h2>
+<pre>cd examples/industrial_data_pretraining/sense_voice
+bash finetune.sh</pre>
+
+<p>Same data format as Paraformer (source/target JSONL). Key difference: SenseVoice uses its own dataset class internally.</p>
+
+<!-- Fun-ASR-Nano -->
+<h2 id="nano">Fine-tune Fun-ASR-Nano</h2>
+<pre>cd examples/industrial_data_pretraining/fun_asr_nano
+bash finetune.sh</pre>
+
+<p>Key differences from Paraformer:</p>
+<ul>
+<li>Uses ChatML data format (see above)</li>
+<li><code>++trust_remote_code=true</code> required</li>
+<li>Supports <strong>selective freezing</strong>: freeze encoder/adaptor while training LLM decoder</li>
+</ul>
+
+<pre># Freeze encoder + adaptor, only train LLM (recommended for domain adaptation)
+++audio_encoder_conf.freeze=true
+++audio_adaptor_conf.freeze=true
+++llm_conf.freeze=false
+
+# Full fine-tune (all parameters)
+++audio_encoder_conf.freeze=false
+++audio_adaptor_conf.freeze=false
+++llm_conf.freeze=false</pre>
+
+<div class="note"><strong>Recommended strategy:</strong> Start with LLM-only fine-tuning (faster, less data needed). If results are insufficient, unfreeze adaptor. Only unfreeze encoder with very large datasets (>1000h).</div>
+
+<!-- Parameters -->
+<h2 id="params">Parameter Reference</h2>
+
+<h3>Dataset Parameters</h3>
+<table>
+<tr><th>Parameter</th><th>Default</th><th>Description</th></tr>
+<tr><td><code>dataset_conf.batch_type</code></td><td>"token"</td><td><code>"token"</code>: dynamic batch by total tokens. <code>"example"</code>: fixed batch count.</td></tr>
+<tr><td><code>dataset_conf.batch_size</code></td><td>6000</td><td>Token mode: total frames per batch. Example mode: number of samples.</td></tr>
+<tr><td><code>dataset_conf.sort_size</code></td><td>1024</td><td>Buffer size for length-based sorting (improves padding efficiency).</td></tr>
+<tr><td><code>dataset_conf.num_workers</code></td><td>4</td><td>Data loading threads.</td></tr>
+<tr><td><code>dataset_conf.data_split_num</code></td><td>1</td><td>Split data into N groups for large-scale training (reduces memory).</td></tr>
+<tr><td><code>dataset_conf.max_token_length</code></td><td>—</td><td>Filter: skip samples longer than this (in frames/tokens).</td></tr>
+<tr><td><code>dataset_conf.min_token_length</code></td><td>—</td><td>Filter: skip samples shorter than this.</td></tr>
+</table>
+
+<h3>Training Parameters</h3>
+<table>
+<tr><th>Parameter</th><th>Default</th><th>Description</th></tr>
+<tr><td><code>train_conf.max_epoch</code></td><td>50</td><td>Total training epochs.</td></tr>
+<tr><td><code>train_conf.log_interval</code></td><td>1</td><td>Print loss every N steps.</td></tr>
+<tr><td><code>train_conf.validate_interval</code></td><td>2000</td><td>Run validation every N steps.</td></tr>
+<tr><td><code>train_conf.save_checkpoint_interval</code></td><td>2000</td><td>Save model every N steps.</td></tr>
+<tr><td><code>train_conf.keep_nbest_models</code></td><td>20</td><td>Keep top N models (by validation accuracy).</td></tr>
+<tr><td><code>train_conf.avg_nbest_model</code></td><td>10</td><td>Average top N models for final checkpoint.</td></tr>
+<tr><td><code>train_conf.resume</code></td><td>true</td><td>Resume from last checkpoint if exists.</td></tr>
+<tr><td><code>train_conf.use_deepspeed</code></td><td>false</td><td>Enable DeepSpeed ZeRO optimization.</td></tr>
+<tr><td><code>optim_conf.lr</code></td><td>0.0002</td><td>Learning rate.</td></tr>
+</table>
+
+<!-- Multi-GPU -->
+<h2 id="multi-gpu">Multi-GPU Training</h2>
+
+<h3>Single Machine, Multiple GPUs</h3>
+<pre>export CUDA_VISIBLE_DEVICES="0,1,2,3"
+gpu_num=4
+
+torchrun --nnodes 1 --nproc_per_node $gpu_num \
+ funasr/bin/train_ds.py ${train_args}</pre>
+
+<h3>Multiple Machines</h3>
+<pre># Machine 1 (master, IP=192.168.1.1)
+torchrun --nnodes 2 --node_rank 0 --nproc_per_node 4 \
+ --master_addr=192.168.1.1 --master_port=12345 \
+ funasr/bin/train_ds.py ${train_args}
+
+# Machine 2
+torchrun --nnodes 2 --node_rank 1 --nproc_per_node 4 \
+ --master_addr=192.168.1.1 --master_port=12345 \
+ funasr/bin/train_ds.py ${train_args}</pre>
+
+<!-- DeepSpeed -->
+<h2 id="deepspeed">DeepSpeed</h2>
+<p>For large models (Fun-ASR-Nano 800M params), enable DeepSpeed ZeRO:</p>
+<pre>++train_conf.use_deepspeed=true
+++train_conf.deepspeed_config=./deepspeed_conf/ds_stage1.json</pre>
+
+<p>Stage 1 config (recommended starting point):</p>
+<pre>{
+ "train_micro_batch_size_per_gpu": 1,
+ "gradient_accumulation_steps": 1,
+ "bf16": {"enabled": true},
+ "zero_optimization": {
+ "stage": 1,
+ "reduce_bucket_size": 5e8,
+ "allgather_bucket_size": 5e8
+ }
+}</pre>
+
+<div class="note"><strong>When to use which stage:</strong><br>
+• Stage 1: Optimizer state partitioned. Good for most cases.<br>
+• Stage 2: + Gradient partitioned. For larger models.<br>
+• Stage 3: + Parameter partitioned. Maximum memory savings but slower.</div>
+
+<!-- Monitor -->
+<h2 id="monitor">Monitoring Training</h2>
+
+<h4>Log file</h4>
+<pre>tail -f outputs/log.txt
+
+# Example output:
+# train, rank: 0, epoch: 0/50, step: 6990, (loss_avg_rank: 0.327),
+# (acc_avg_epoch: 0.795), (lr: 1.165e-04),
+# GPU memory: usage: 3.8GB, peak: 18.3GB</pre>
+
+<p>Key metrics to watch:</p>
+<ul>
+<li><code>loss_avg_epoch</code>: should decrease over time</li>
+<li><code>acc_avg_epoch</code>: should increase (most important metric)</li>
+<li><code>lr</code>: learning rate at current step</li>
+<li><code>GPU memory</code>: peak should not exceed your GPU VRAM</li>
+</ul>
+
+<h4>TensorBoard</h4>
+<pre>tensorboard --logdir outputs/log/tensorboard
+# Open http://localhost:6006</pre>
+
+<!-- Use Fine-tuned Model -->
+<h2 id="inference-after">Use Your Fine-tuned Model</h2>
+
+<h4>If outputs/ has configuration.json</h4>
+<pre>from funasr import AutoModel
+model = AutoModel(model="./outputs")
+res = model.generate(input="test.wav")
+print(res[0]["text"])</pre>
+
+<h4>If no configuration.json</h4>
+<pre>funasr ++model="./outputs" \
+ ++config-path="./outputs" \
+ ++config-name="config.yaml" \
+ ++init_param="./outputs/model.pt" \
+ ++input="test.wav"</pre>
+
+<!-- Tips -->
+<h2 id="tips">Tips & Troubleshooting</h2>
+
+<h4>OOM during training</h4>
+<ol>
+<li>Reduce <code>dataset_conf.batch_size</code></li>
+<li>Add <code>dataset_conf.max_token_length=2000</code> to filter long utterances</li>
+<li>Enable DeepSpeed (partitions optimizer states)</li>
+<li>Reduce <code>dataset_conf.num_workers</code></li>
+</ol>
+
+<h4>Training loss stuck / NaN gradients</h4>
+<ul>
+<li>Reduce learning rate (try 0.00005)</li>
+<li>Check data quality — corrupted audio files cause NaN</li>
+<li>For Fun-ASR-Nano: start with encoder frozen</li>
+</ul>
+
+<h4>Validation accuracy not improving</h4>
+<ul>
+<li>Increase training data (min ~10h for fine-tuning)</li>
+<li>Check domain match — model may not generalize to very different domains</li>
+<li>Try unfreezing more layers gradually</li>
+</ul>
+
+<h4>Large-scale data (>10,000 hours)</h4>
+<p>Use data splitting to avoid memory issues:</p>
+<pre># Split data into chunks, load 2 at a time
+++dataset_conf.data_split_num=256
+# data.list contains paths to split jsonl files:
+# data/train.0.jsonl
+# data/train.1.jsonl
+# ...
+++train_data_set_list="data/data.list"</pre>
+
+<h4>Resume after crash</h4>
+<p>Set <code>++train_conf.resume=true</code> (default). Training automatically restarts from the latest checkpoint in <code>output_dir</code>.</p>
+
+</div></div>
+
+<footer>
+<p>FunASR &middot; Tongyi Lab, Alibaba Group</p>
+<p><a href="index.html">Home</a> &middot; <a href="tutorial.html">Tutorial</a> &middot; <a href="model-registration.html">Develop</a> &middot; <a href="api.html">API</a> &middot; <a href="https://github.com/modelscope/FunASR">GitHub</a></p>
+</footer>
+</body></html>