Open-source speech recognition toolkit for training, inference, streaming ASR, VAD, punctuation, speaker diarization pipelines, and OpenAI-compatible/MCP serving.
Python asraudiochineseemotion-recognitionfunasrmcp-servermultilingual-asropenai-compatible-apiparaformerpunctuationpytorchreal-time-asrspeaker-diarizationspeech-recognitionspeech-to-textstreaming-asrtranscriptionvllmvoice-activity-detectionwhisper-alternative
docs: add Training page with complete fine-tuning guide (data prep, multi-GPU, DeepSpeed, monitoring)
游雁abc132e7bb8b
diff --git a/gh-pages-output/api.html b/gh-pages-output/api.htmlnew file mode 100644index 000000000..dc7bcbffb--- /dev/null+++ b/gh-pages-output/api.html@@ -0,0 +1,27935 @@+<!DOCTYPE html>+<html lang="en"><head><meta charset="UTF-8"><meta name="viewport" content="width=device-width,initial-scale=1.0">+<title>FunASR API</title>+<link href="https://fonts.googleapis.com/css2?family=Inter:wght@400;500;600;700&family=JetBrains+Mono:wght@400;500&display=swap" rel="stylesheet">+<link rel="stylesheet" href="style.css">+<style>+body{display:flex;flex-direction:column;min-height:100vh;background:var(--surface)}+.api-layout{display:flex;flex:1}+.api-sidebar{width:260px;min-width:260px;height:calc(100vh - 57px);position:sticky;top:57px;overflow-y:auto;border-right:1px solid var(--line);background:var(--page);padding:8px 0}+.sb-search{margin:4px 8px 10px;padding:7px 10px;width:calc(100% - 16px);border:1px solid var(--line);border-radius:5px;font-size:0.78rem}+.sb-search:focus{outline:none;border-color:var(--primary)}+.l1{border-bottom:1px solid var(--line)}.l1-title{padding:9px 12px;font-size:0.78rem;font-weight:700;color:var(--ink-soft);cursor:pointer;display:flex;align-items:center;gap:6px}+.l1-title:hover{background:var(--surface-2)}.l1-title .cnt{font-size:0.65rem;background:var(--surface-2);color:var(--muted);padding:1px 5px;border-radius:8px;margin-left:auto}+.l1-title .arr{font-size:0.6rem;color:var(--muted);transition:transform 0.2s}.l1-children{max-height:0;overflow:hidden;transition:max-height 0.3s}+.l1.open>.l1-children{max-height:20000px}.l1.open>.l1-title .arr{transform:rotate(90deg)}+.l2-title{padding:4px 12px 4px 22px;font-size:0.74rem;font-weight:600;color:var(--muted);cursor:pointer;display:flex;align-items:center}+.l2-title:hover{background:var(--surface-2);color:var(--ink-soft)}.l2-title .arr{margin-left:auto;font-size:0.55rem;transition:transform 0.2s}+.l2-children{max-height:0;overflow:hidden;transition:max-height 0.25s}.l2.open>.l2-children{max-height:20000px}.l2.open>.l2-title .arr{transform:rotate(90deg)}+.l3-item{display:flex;align-items:center;gap:5px;padding:2px 12px 2px 34px;font-size:0.73rem;color:var(--ink-soft);text-decoration:none;cursor:pointer;border-left:2px solid transparent}+.l3-item:hover{background:var(--surface-2)}.l3-item.active{background:var(--primary-soft);color:var(--primary-dark);border-left-color:var(--primary);font-weight:600}+.l3m{padding-left:44px;font-size:0.7rem}+.mb{font-size:0.5rem;font-weight:700;width:13px;height:13px;display:inline-flex;align-items:center;justify-content:center;border-radius:2px;flex-shrink:0}+.badge-c{background:#dbeafe;color:#1d4ed8}.badge-m{background:#d1fae5;color:#059669}.badge-f{background:#fef3c7;color:#b45309}+.api-content{flex:1;padding:32px 40px;overflow-y:auto;max-width:800px}+.api-detail{display:none}.api-detail.active{display:block}+.api-welcome{color:var(--muted);padding:60px 20px;text-align:center}.api-welcome h2{color:var(--ink-soft);margin-bottom:8px}+.dtag{font-size:0.63rem;font-weight:700;padding:2px 7px;border-radius:3px;text-transform:uppercase}+h2{font-size:1.1rem;margin:8px 0}h2 code{font-family:'JetBrains Mono',monospace;font-size:0.92rem;background:none;color:var(--ink);padding:0}+.dmod{font-size:0.7rem;color:var(--muted);font-family:monospace;margin-bottom:18px;padding-bottom:12px;border-bottom:1px solid var(--line)}+.dmod a{color:var(--primary);text-decoration:none;font-size:0.7rem}.dmod a:hover{text-decoration:underline}+.ddoc{line-height:1.7}.ddoc p{margin-bottom:8px;font-size:0.87rem}.ddoc h4{font-size:0.82rem;font-weight:700;margin:16px 0 4px;color:var(--ink)}+.ddoc ul{list-style:none;padding:0;margin:0 0 12px}.ddoc li{padding:3px 0 3px 12px;border-left:2px solid var(--line);font-size:0.83rem;color:var(--ink-soft)}+.ddoc li code{background:var(--surface-2);padding:1px 4px;border-radius:3px;font-size:0.77rem;color:#b91c1c}+.ddoc li.sub{padding-left:24px;font-size:0.79rem;color:var(--muted);border-left-color:transparent}+.mt{font-size:0.85rem;font-weight:700;margin:24px 0 10px;padding-top:14px;border-top:1px solid var(--line)}+.mblk{margin:8px 0;padding:10px 14px;background:var(--page);border-radius:6px;border:1px solid var(--line)}+.mblk code{font-family:'JetBrains Mono',monospace;font-size:0.84rem}+.mdoc p{font-size:0.83rem}.mdoc h4{font-size:0.78rem}+.gh-link{font-size:0.68rem;color:var(--muted);text-decoration:none;margin-left:8px}.gh-link:hover{color:var(--primary)}+.src-block{margin:12px 0;border:1px solid var(--line);border-radius:6px;overflow:hidden}+.src-block summary{padding:8px 14px;font-size:0.78rem;cursor:pointer;color:var(--muted);background:var(--surface-2)}+.src-block summary:hover{color:var(--ink-soft)}+.src-block pre{margin:0;border-radius:0;font-size:0.75rem;max-height:400px;overflow-y:auto}+.src-more{display:block;padding:8px 14px;font-size:0.75rem;color:var(--primary);text-decoration:none;border-top:1px solid var(--line)}+.src-more:hover{background:var(--surface-2)}+.muted{color:var(--muted-2);font-style:italic}+.hidden{display:none!important}+</style></head><body>+<nav class="nav"><div class="container">+<a href="index.html" class="nav-logo">FunASR</a>+<div class="nav-links"><a href="index.html">Home</a><a href="tutorial.html">Tutorial</a><a href="model-registration.html">Develop</a><a href="api.html" class="active">API</a></div>+<a href="https://github.com/modelscope/FunASR" class="nav-github">GitHub</a>+</div></nav>+<div class="api-layout">+<div class="api-sidebar">+<input type="text" class="sb-search" placeholder="Search..." oninput="filterSidebar(this.value)">+<div class="l1"><div class="l1-title" onclick="toggleL1(this)"><span>funasr.auto</span><span class="cnt">1</span><span class="arr">▸</span></div><div class="l1-children">+<div class="l2"><div class="l2-title" onclick="toggleL2(this)"><span>auto_model</span><span class="arr">▸</span></div><div class="l2-children">+<a class="l3-item" data-target="e1" onclick="showEntry('e1')"><span class="mb badge-f">F</span>is_npu_available</a>+<a class="l3-item" data-target="e2" onclick="showEntry('e2')"><span class="mb badge-f">F</span>prepare_data_iterator</a>+<a class="l3-item" data-target="e3" onclick="showEntry('e3')"><span class="mb badge-c">C</span>AutoModel</a>+<a class="l3-item l3m" data-target="e4" onclick="showEntry('e4')"><span class="mb badge-m">M</span>.build_model()</a>+<a class="l3-item l3m" data-target="e5" onclick="showEntry('e5')"><span class="mb badge-m">M</span>.generate()</a>+<a class="l3-item l3m" data-target="e6" onclick="showEntry('e6')"><span class="mb badge-m">M</span>.inference()</a>+<a class="l3-item l3m" data-target="e7" onclick="showEntry('e7')"><span class="mb badge-m">M</span>.inference_with_vad()</a>+<a class="l3-item l3m" data-target="e8" onclick="showEntry('e8')"><span class="mb badge-m">M</span>.export()</a>+</div></div>+</div></div>+<div class="l1"><div class="l1-title" onclick="toggleL1(this)"><span>funasr.register</span><span class="cnt">1</span><span class="arr">▸</span></div><div class="l1-children">+<div class="l2"><div class="l2-title" onclick="toggleL2(this)"><span>register</span><span class="arr">▸</span></div><div class="l2-children">+<a class="l3-item" data-target="e9" onclick="showEntry('e9')"><span class="mb badge-c">C</span>RegisterTables</a>+<a class="l3-item l3m" data-target="e10" onclick="showEntry('e10')"><span class="mb badge-m">M</span>.print()</a>+<a class="l3-item l3m" data-target="e11" onclick="showEntry('e11')"><span class="mb badge-m">M</span>.register()</a>+</div></div>+</div></div>+<div class="l1"><div class="l1-title" onclick="toggleL1(this)"><span>funasr.models</span><span class="cnt">39</span><span class="arr">▸</span></div><div class="l1-children">+<div class="l2"><div class="l2-title" onclick="toggleL2(this)"><span>bat</span><span class="arr">▸</span></div><div class="l2-children">+<a class="l3-item" data-target="e12" onclick="showEntry('e12')"><span class="mb badge-c">C</span>BAT</a>+</div></div>+<div class="l2"><div class="l2-title" onclick="toggleL2(this)"><span>bicif_paraformer</span><span class="arr">▸</span></div><div class="l2-children">+<a class="l3-item" data-target="e13" onclick="showEntry('e13')"><span class="mb badge-c">C</span>BiCifParaformer</a>+<a class="l3-item l3m" data-target="e14" onclick="showEntry('e14')"><span class="mb badge-m">M</span>.calc_predictor()</a>+<a class="l3-item l3m" data-target="e15" onclick="showEntry('e15')"><span class="mb badge-m">M</span>.calc_predictor_timestamp()</a>+<a class="l3-item l3m" data-target="e16" onclick="showEntry('e16')"><span class="mb badge-m">M</span>.forward()</a>+<a class="l3-item l3m" data-target="e17" onclick="showEntry('e17')"><span class="mb badge-m">M</span>.inference()</a>+<a class="l3-item l3m" data-target="e18" onclick="showEntry('e18')"><span class="mb badge-m">M</span>.export()</a>+</div></div>+<div class="l2"><div class="l2-title" onclick="toggleL2(this)"><span>branchformer</span><span class="arr">▸</span></div><div class="l2-children">+<a class="l3-item" data-target="e19" onclick="showEntry('e19')"><span class="mb badge-c">C</span>Branchformer</a>+</div></div>+<div class="l2"><div class="l2-title" onclick="toggleL2(this)"><span>campplus</span><span class="arr">▸</span></div><div class="l2-children">+<a class="l3-item" data-target="e20" onclick="showEntry('e20')"><span class="mb badge-c">C</span>CAMPPlus</a>+<a class="l3-item l3m" data-target="e21" onclick="showEntry('e21')"><span class="mb badge-m">M</span>.forward()</a>+<a class="l3-item l3m" data-target="e22" onclick="showEntry('e22')"><span class="mb badge-m">M</span>.inference()</a>+</div></div>+<div class="l2"><div class="l2-title" onclick="toggleL2(this)"><span>conformer</span><span class="arr">▸</span></div><div class="l2-children">+<a class="l3-item" data-target="e23" onclick="showEntry('e23')"><span class="mb badge-c">C</span>Conformer</a>+</div></div>+<div class="l2"><div class="l2-title" onclick="toggleL2(this)"><span>conformer_rwkv</span><span class="arr">▸</span></div><div class="l2-children">+<a class="l3-item" data-target="e24" onclick="showEntry('e24')"><span class="mb badge-c">C</span>Conformer</a>+</div></div>+<div class="l2"><div class="l2-title" onclick="toggleL2(this)"><span>contextual_paraformer</span><span class="arr">▸</span></div><div class="l2-children">+<a class="l3-item" data-target="e25" onclick="showEntry('e25')"><span class="mb badge-c">C</span>ContextualParaformer</a>+<a class="l3-item l3m" data-target="e26" onclick="showEntry('e26')"><span class="mb badge-m">M</span>.forward()</a>+<a class="l3-item l3m" data-target="e27" onclick="showEntry('e27')"><span class="mb badge-m">M</span>.sampler()</a>+<a class="l3-item l3m" data-target="e28" onclick="showEntry('e28')"><span class="mb badge-m">M</span>.cal_decoder_with_predictor()</a>+<a class="l3-item l3m" data-target="e29" onclick="showEntry('e29')"><span class="mb badge-m">M</span>.inference()</a>+<a class="l3-item l3m" data-target="e30" onclick="showEntry('e30')"><span class="mb badge-m">M</span>.generate_hotwords_list()</a>+<a class="l3-item l3m" data-target="e31" onclick="showEntry('e31')"><span class="mb badge-m">M</span>.export()</a>+</div></div>+<div class="l2"><div class="l2-title" onclick="toggleL2(this)"><span>ct_transformer</span><span class="arr">▸</span></div><div class="l2-children">+<a class="l3-item" data-target="e32" onclick="showEntry('e32')"><span class="mb badge-c">C</span>CTTransformer</a>+<a class="l3-item l3m" data-target="e33" onclick="showEntry('e33')"><span class="mb badge-m">M</span>.punc_forward()</a>+<a class="l3-item l3m" data-target="e34" onclick="showEntry('e34')"><span class="mb badge-m">M</span>.with_vad()</a>+<a class="l3-item l3m" data-target="e35" onclick="showEntry('e35')"><span class="mb badge-m">M</span>.score()</a>+<a class="l3-item l3m" data-target="e36" onclick="showEntry('e36')"><span class="mb badge-m">M</span>.batch_score()</a>+<a class="l3-item l3m" data-target="e37" onclick="showEntry('e37')"><span class="mb badge-m">M</span>.nll()</a>+<a class="l3-item l3m" data-target="e38" onclick="showEntry('e38')"><span class="mb badge-m">M</span>.forward()</a>+<a class="l3-item l3m" data-target="e39" onclick="showEntry('e39')"><span class="mb badge-m">M</span>.inference()</a>+<a class="l3-item l3m" data-target="e40" onclick="showEntry('e40')"><span class="mb badge-m">M</span>.export()</a>+</div></div>+<div class="l2"><div class="l2-title" onclick="toggleL2(this)"><span>ct_transformer_streaming</span><span class="arr">▸</span></div><div class="l2-children">+<a class="l3-item" data-target="e41" onclick="showEntry('e41')"><span class="mb badge-c">C</span>CTTransformerStreaming</a>+<a class="l3-item l3m" data-target="e42" onclick="showEntry('e42')"><span class="mb badge-m">M</span>.punc_forward()</a>+<a class="l3-item l3m" data-target="e43" onclick="showEntry('e43')"><span class="mb badge-m">M</span>.with_vad()</a>+<a class="l3-item l3m" data-target="e44" onclick="showEntry('e44')"><span class="mb badge-m">M</span>.inference()</a>+<a class="l3-item l3m" data-target="e45" onclick="showEntry('e45')"><span class="mb badge-m">M</span>.export()</a>+</div></div>+<div class="l2"><div class="l2-title" onclick="toggleL2(this)"><span>ctc</span><span class="arr">▸</span></div><div class="l2-children">+<a class="l3-item" data-target="e46" onclick="showEntry('e46')"><span class="mb badge-c">C</span>Transformer</a>+<a class="l3-item l3m" data-target="e47" onclick="showEntry('e47')"><span class="mb badge-m">M</span>.forward()</a>+<a class="l3-item l3m" data-target="e48" onclick="showEntry('e48')"><span class="mb badge-m">M</span>.encode()</a>+<a class="l3-item l3m" data-target="e49" onclick="showEntry('e49')"><span class="mb badge-m">M</span>.inference()</a>+</div></div>+<div class="l2"><div class="l2-title" onclick="toggleL2(this)"><span>e_branchformer</span><span class="arr">▸</span></div><div class="l2-children">+<a class="l3-item" data-target="e50" onclick="showEntry('e50')"><span class="mb badge-c">C</span>EBranchformer</a>+</div></div>+<div class="l2"><div class="l2-title" onclick="toggleL2(this)"><span>e_paraformer</span><span class="arr">▸</span></div><div class="l2-children">+<a class="l3-item" data-target="e51" onclick="showEntry('e51')"><span class="mb badge-c">C</span>EParaformer</a>+<a class="l3-item l3m" data-target="e52" onclick="showEntry('e52')"><span class="mb badge-m">M</span>.forward()</a>+<a class="l3-item l3m" data-target="e53" onclick="showEntry('e53')"><span class="mb badge-m">M</span>.encode()</a>+<a class="l3-item l3m" data-target="e54" onclick="showEntry('e54')"><span class="mb badge-m">M</span>.calc_predictor()</a>+<a class="l3-item l3m" data-target="e55" onclick="showEntry('e55')"><span class="mb badge-m">M</span>.cal_decoder_with_predictor()</a>+<a class="l3-item l3m" data-target="e56" onclick="showEntry('e56')"><span class="mb badge-m">M</span>.sampler()</a>+<a class="l3-item l3m" data-target="e57" onclick="showEntry('e57')"><span class="mb badge-m">M</span>.sampler_with_grad()</a>+<a class="l3-item l3m" data-target="e58" onclick="showEntry('e58')"><span class="mb badge-m">M</span>.init_beam_search()</a>+<a class="l3-item l3m" data-target="e59" onclick="showEntry('e59')"><span class="mb badge-m">M</span>.inference()</a>+<a class="l3-item l3m" data-target="e60" onclick="showEntry('e60')"><span class="mb badge-m">M</span>.export()</a>+</div></div>+<div class="l2"><div class="l2-title" onclick="toggleL2(this)"><span>emotion2vec</span><span class="arr">▸</span></div><div class="l2-children">+<a class="l3-item" data-target="e61" onclick="showEntry('e61')"><span class="mb badge-c">C</span>Emotion2vec</a>+<a class="l3-item l3m" data-target="e62" onclick="showEntry('e62')"><span class="mb badge-m">M</span>.forward()</a>+<a class="l3-item l3m" data-target="e63" onclick="showEntry('e63')"><span class="mb badge-m">M</span>.extract_features()</a>+<a class="l3-item l3m" data-target="e64" onclick="showEntry('e64')"><span class="mb badge-m">M</span>.inference()</a>+<a class="l3-item l3m" data-target="e65" onclick="showEntry('e65')"><span class="mb badge-m">M</span>.export()</a>+</div></div>+<div class="l2"><div class="l2-title" onclick="toggleL2(this)"><span>eres2net</span><span class="arr">▸</span></div><div class="l2-children">+<a class="l3-item" data-target="e66" onclick="showEntry('e66')"><span class="mb badge-c">C</span>ERes2NetV2SV</a>+<a class="l3-item l3m" data-target="e67" onclick="showEntry('e67')"><span class="mb badge-m">M</span>.forward()</a>+<a class="l3-item l3m" data-target="e68" onclick="showEntry('e68')"><span class="mb badge-m">M</span>.inference()</a>+</div></div>+<div class="l2"><div class="l2-title" onclick="toggleL2(this)"><span>fsmn_kws</span><span class="arr">▸</span></div><div class="l2-children">+<a class="l3-item" data-target="e69" onclick="showEntry('e69')"><span class="mb badge-c">C</span>FsmnKWS</a>+<a class="l3-item l3m" data-target="e70" onclick="showEntry('e70')"><span class="mb badge-m">M</span>.forward()</a>+<a class="l3-item l3m" data-target="e71" onclick="showEntry('e71')"><span class="mb badge-m">M</span>.encode()</a>+<a class="l3-item l3m" data-target="e72" onclick="showEntry('e72')"><span class="mb badge-m">M</span>.inference()</a>+<a class="l3-item" data-target="e73" onclick="showEntry('e73')"><span class="mb badge-c">C</span>FsmnKWSConvert</a>+<a class="l3-item l3m" data-target="e74" onclick="showEntry('e74')"><span class="mb badge-m">M</span>.to_kaldi_net()</a>+<a class="l3-item l3m" data-target="e75" onclick="showEntry('e75')"><span class="mb badge-m">M</span>.to_pytorch_net()</a>+</div></div>+<div class="l2"><div class="l2-title" onclick="toggleL2(this)"><span>fsmn_kws_mt</span><span class="arr">▸</span></div><div class="l2-children">+<a class="l3-item" data-target="e76" onclick="showEntry('e76')"><span class="mb badge-c">C</span>FsmnKWSMT</a>+<a class="l3-item l3m" data-target="e77" onclick="showEntry('e77')"><span class="mb badge-m">M</span>.forward()</a>+<a class="l3-item l3m" data-target="e78" onclick="showEntry('e78')"><span class="mb badge-m">M</span>.encode()</a>+<a class="l3-item l3m" data-target="e79" onclick="showEntry('e79')"><span class="mb badge-m">M</span>.inference()</a>+<a class="l3-item" data-target="e80" onclick="showEntry('e80')"><span class="mb badge-c">C</span>FsmnKWSMTConvert</a>+<a class="l3-item l3m" data-target="e81" onclick="showEntry('e81')"><span class="mb badge-m">M</span>.to_kaldi_net()</a>+<a class="l3-item l3m" data-target="e82" onclick="showEntry('e82')"><span class="mb badge-m">M</span>.to_kaldi_net2()</a>+<a class="l3-item l3m" data-target="e83" onclick="showEntry('e83')"><span class="mb badge-m">M</span>.to_pytorch_net()</a>+</div></div>+<div class="l2"><div class="l2-title" onclick="toggleL2(this)"><span>fsmn_vad_streaming</span><span class="arr">▸</span></div><div class="l2-children">+<a class="l3-item" data-target="e84" onclick="showEntry('e84')"><span class="mb badge-c">C</span>FsmnVADStreaming</a>+<a class="l3-item l3m" data-target="e85" onclick="showEntry('e85')"><span class="mb badge-m">M</span>.ResetDetection()</a>+<a class="l3-item l3m" data-target="e86" onclick="showEntry('e86')"><span class="mb badge-m">M</span>.ComputeDecibel()</a>+<a class="l3-item l3m" data-target="e87" onclick="showEntry('e87')"><span class="mb badge-m">M</span>.ComputeScores()</a>+<a class="l3-item l3m" data-target="e88" onclick="showEntry('e88')"><span class="mb badge-m">M</span>.PopDataBufTillFrame()</a>+<a class="l3-item l3m" data-target="e89" onclick="showEntry('e89')"><span class="mb badge-m">M</span>.PopDataToOutputBuf()</a>+<a class="l3-item l3m" data-target="e90" onclick="showEntry('e90')"><span class="mb badge-m">M</span>.OnSilenceDetected()</a>+<a class="l3-item l3m" data-target="e91" onclick="showEntry('e91')"><span class="mb badge-m">M</span>.OnVoiceDetected()</a>+<a class="l3-item l3m" data-target="e92" onclick="showEntry('e92')"><span class="mb badge-m">M</span>.OnVoiceStart()</a>+<a class="l3-item l3m" data-target="e93" onclick="showEntry('e93')"><span class="mb badge-m">M</span>.OnVoiceEnd()</a>+<a class="l3-item l3m" data-target="e94" onclick="showEntry('e94')"><span class="mb badge-m">M</span>.MaybeOnVoiceEndIfLastFrame()</a>+<a class="l3-item l3m" data-target="e95" onclick="showEntry('e95')"><span class="mb badge-m">M</span>.GetLatency()</a>+<a class="l3-item l3m" data-target="e96" onclick="showEntry('e96')"><span class="mb badge-m">M</span>.LatencyFrmNumAtStartPoint()</a>+<a class="l3-item l3m" data-target="e97" onclick="showEntry('e97')"><span class="mb badge-m">M</span>.GetFrameState()</a>+<a class="l3-item l3m" data-target="e98" onclick="showEntry('e98')"><span class="mb badge-m">M</span>.forward()</a>+<a class="l3-item l3m" data-target="e99" onclick="showEntry('e99')"><span class="mb badge-m">M</span>.init_cache()</a>+<a class="l3-item l3m" data-target="e100" onclick="showEntry('e100')"><span class="mb badge-m">M</span>.inference()</a>+<a class="l3-item l3m" data-target="e101" onclick="showEntry('e101')"><span class="mb badge-m">M</span>.export()</a>+<a class="l3-item l3m" data-target="e102" onclick="showEntry('e102')"><span class="mb badge-m">M</span>.DetectCommonFrames()</a>+<a class="l3-item l3m" data-target="e103" onclick="showEntry('e103')"><span class="mb badge-m">M</span>.DetectLastFrames()</a>+<a class="l3-item l3m" data-target="e104" onclick="showEntry('e104')"><span class="mb badge-m">M</span>.DetectOneFrame()</a>+</div></div>+<div class="l2"><div class="l2-title" onclick="toggleL2(this)"><span>fun_asr_nano</span><span class="arr">▸</span></div><div class="l2-children">+<a class="l3-item" data-target="e105" onclick="showEntry('e105')"><span class="mb badge-c">C</span>FunASRNano</a>+<a class="l3-item l3m" data-target="e106" onclick="showEntry('e106')"><span class="mb badge-m">M</span>.forward()</a>+<a class="l3-item l3m" data-target="e107" onclick="showEntry('e107')"><span class="mb badge-m">M</span>.forward_export()</a>+<a class="l3-item l3m" data-target="e108" onclick="showEntry('e108')"><span class="mb badge-m">M</span>.encode()</a>+<a class="l3-item l3m" data-target="e109" onclick="showEntry('e109')"><span class="mb badge-m">M</span>.data_template()</a>+<a class="l3-item l3m" data-target="e110" onclick="showEntry('e110')"><span class="mb badge-m">M</span>.data_load_speech()</a>+<a class="l3-item l3m" data-target="e111" onclick="showEntry('e111')"><span class="mb badge-m">M</span>.inference_prepare()</a>+<a class="l3-item l3m" data-target="e112" onclick="showEntry('e112')"><span class="mb badge-m">M</span>.get_prompt()</a>+<a class="l3-item l3m" data-target="e113" onclick="showEntry('e113')"><span class="mb badge-m">M</span>.generate_chatml()</a>+<a class="l3-item l3m" data-target="e114" onclick="showEntry('e114')"><span class="mb badge-m">M</span>.inference()</a>+<a class="l3-item l3m" data-target="e115" onclick="showEntry('e115')"><span class="mb badge-m">M</span>.inference_llm()</a>+<a class="l3-item l3m" data-target="e116" onclick="showEntry('e116')"><span class="mb badge-m">M</span>.from_pretrained()</a>+</div></div>+<div class="l2"><div class="l2-title" onclick="toggleL2(this)"><span>glm_asr</span><span class="arr">▸</span></div><div class="l2-children">+<a class="l3-item" data-target="e117" onclick="showEntry('e117')"><span class="mb badge-c">C</span>GLMASR</a>+<a class="l3-item l3m" data-target="e118" onclick="showEntry('e118')"><span class="mb badge-m">M</span>.forward()</a>+<a class="l3-item l3m" data-target="e119" onclick="showEntry('e119')"><span class="mb badge-m">M</span>.inference()</a>+</div></div>+<div class="l2"><div class="l2-title" onclick="toggleL2(this)"><span>lcbnet</span><span class="arr">▸</span></div><div class="l2-children">+<a class="l3-item" data-target="e120" onclick="showEntry('e120')"><span class="mb badge-c">C</span>LCBNet</a>+<a class="l3-item l3m" data-target="e121" onclick="showEntry('e121')"><span class="mb badge-m">M</span>.forward()</a>+<a class="l3-item l3m" data-target="e122" onclick="showEntry('e122')"><span class="mb badge-m">M</span>.encode()</a>+<a class="l3-item l3m" data-target="e123" onclick="showEntry('e123')"><span class="mb badge-m">M</span>.init_beam_search()</a>+<a class="l3-item l3m" data-target="e124" onclick="showEntry('e124')"><span class="mb badge-m">M</span>.inference()</a>+</div></div>+<div class="l2"><div class="l2-title" onclick="toggleL2(this)"><span>llm_asr</span><span class="arr">▸</span></div><div class="l2-children">+<a class="l3-item" data-target="e125" onclick="showEntry('e125')"><span class="mb badge-c">C</span>LLMASR</a>+<a class="l3-item l3m" data-target="e126" onclick="showEntry('e126')"><span class="mb badge-m">M</span>.forward()</a>+<a class="l3-item l3m" data-target="e127" onclick="showEntry('e127')"><span class="mb badge-m">M</span>.encode()</a>+<a class="l3-item l3m" data-target="e128" onclick="showEntry('e128')"><span class="mb badge-m">M</span>.inference()</a>+<a class="l3-item" data-target="e129" onclick="showEntry('e129')"><span class="mb badge-c">C</span>LLMASR2</a>+<a class="l3-item l3m" data-target="e130" onclick="showEntry('e130')"><span class="mb badge-m">M</span>.forward()</a>+<a class="l3-item l3m" data-target="e131" onclick="showEntry('e131')"><span class="mb badge-m">M</span>.encode()</a>+<a class="l3-item l3m" data-target="e132" onclick="showEntry('e132')"><span class="mb badge-m">M</span>.data_template()</a>+<a class="l3-item l3m" data-target="e133" onclick="showEntry('e133')"><span class="mb badge-m">M</span>.data_load_speech()</a>+<a class="l3-item l3m" data-target="e134" onclick="showEntry('e134')"><span class="mb badge-m">M</span>.inference()</a>+<a class="l3-item" data-target="e135" onclick="showEntry('e135')"><span class="mb badge-c">C</span>LLMASR3</a>+<a class="l3-item l3m" data-target="e136" onclick="showEntry('e136')"><span class="mb badge-m">M</span>.encode()</a>+<a class="l3-item" data-target="e137" onclick="showEntry('e137')"><span class="mb badge-c">C</span>LLMASR4</a>+<a class="l3-item l3m" data-target="e138" onclick="showEntry('e138')"><span class="mb badge-m">M</span>.forward()</a>+<a class="l3-item l3m" data-target="e139" onclick="showEntry('e139')"><span class="mb badge-m">M</span>.encode()</a>+<a class="l3-item l3m" data-target="e140" onclick="showEntry('e140')"><span class="mb badge-m">M</span>.data_template()</a>+<a class="l3-item l3m" data-target="e141" onclick="showEntry('e141')"><span class="mb badge-m">M</span>.data_load_speech()</a>+<a class="l3-item l3m" data-target="e142" onclick="showEntry('e142')"><span class="mb badge-m">M</span>.inference_prepare()</a>+<a class="l3-item l3m" data-target="e143" onclick="showEntry('e143')"><span class="mb badge-m">M</span>.inference()</a>+</div></div>+<div class="l2"><div class="l2-title" onclick="toggleL2(this)"><span>llm_asr_nar</span><span class="arr">▸</span></div><div class="l2-children">+<a class="l3-item" data-target="e144" onclick="showEntry('e144')"><span class="mb badge-c">C</span>LLMASRNAR</a>+<a class="l3-item l3m" data-target="e145" onclick="showEntry('e145')"><span class="mb badge-m">M</span>.forward()</a>+<a class="l3-item l3m" data-target="e146" onclick="showEntry('e146')"><span class="mb badge-m">M</span>.encode()</a>+<a class="l3-item l3m" data-target="e147" onclick="showEntry('e147')"><span class="mb badge-m">M</span>.inference()</a>+<a class="l3-item" data-target="e148" onclick="showEntry('e148')"><span class="mb badge-c">C</span>LLMASRNARPrompt</a>+<a class="l3-item l3m" data-target="e149" onclick="showEntry('e149')"><span class="mb badge-m">M</span>.forward()</a>+<a class="l3-item l3m" data-target="e150" onclick="showEntry('e150')"><span class="mb badge-m">M</span>.encode()</a>+<a class="l3-item l3m" data-target="e151" onclick="showEntry('e151')"><span class="mb badge-m">M</span>.inference()</a>+</div></div>+<div class="l2"><div class="l2-title" onclick="toggleL2(this)"><span>monotonic_aligner</span><span class="arr">▸</span></div><div class="l2-children">+<a class="l3-item" data-target="e152" onclick="showEntry('e152')"><span class="mb badge-c">C</span>MonotonicAligner</a>+<a class="l3-item l3m" data-target="e153" onclick="showEntry('e153')"><span class="mb badge-m">M</span>.forward()</a>+<a class="l3-item l3m" data-target="e154" onclick="showEntry('e154')"><span class="mb badge-m">M</span>.calc_predictor_timestamp()</a>+<a class="l3-item l3m" data-target="e155" onclick="showEntry('e155')"><span class="mb badge-m">M</span>.encode()</a>+<a class="l3-item l3m" data-target="e156" onclick="showEntry('e156')"><span class="mb badge-m">M</span>.inference()</a>+</div></div>+<div class="l2"><div class="l2-title" onclick="toggleL2(this)"><span>paraformer</span><span class="arr">▸</span></div><div class="l2-children">+<a class="l3-item" data-target="e157" onclick="showEntry('e157')"><span class="mb badge-c">C</span>Paraformer</a>+<a class="l3-item l3m" data-target="e158" onclick="showEntry('e158')"><span class="mb badge-m">M</span>.forward()</a>+<a class="l3-item l3m" data-target="e159" onclick="showEntry('e159')"><span class="mb badge-m">M</span>.encode()</a>+<a class="l3-item l3m" data-target="e160" onclick="showEntry('e160')"><span class="mb badge-m">M</span>.calc_predictor()</a>+<a class="l3-item l3m" data-target="e161" onclick="showEntry('e161')"><span class="mb badge-m">M</span>.cal_decoder_with_predictor()</a>+<a class="l3-item l3m" data-target="e162" onclick="showEntry('e162')"><span class="mb badge-m">M</span>.sampler()</a>+<a class="l3-item l3m" data-target="e163" onclick="showEntry('e163')"><span class="mb badge-m">M</span>.init_beam_search()</a>+<a class="l3-item l3m" data-target="e164" onclick="showEntry('e164')"><span class="mb badge-m">M</span>.inference()</a>+<a class="l3-item l3m" data-target="e165" onclick="showEntry('e165')"><span class="mb badge-m">M</span>.export()</a>+</div></div>+<div class="l2"><div class="l2-title" onclick="toggleL2(this)"><span>paraformer_streaming</span><span class="arr">▸</span></div><div class="l2-children">+<a class="l3-item" data-target="e166" onclick="showEntry('e166')"><span class="mb badge-c">C</span>ParaformerStreaming</a>+<a class="l3-item l3m" data-target="e167" onclick="showEntry('e167')"><span class="mb badge-m">M</span>.forward()</a>+<a class="l3-item l3m" data-target="e168" onclick="showEntry('e168')"><span class="mb badge-m">M</span>.encode_chunk()</a>+<a class="l3-item l3m" data-target="e169" onclick="showEntry('e169')"><span class="mb badge-m">M</span>.sampler()</a>+<a class="l3-item l3m" data-target="e170" onclick="showEntry('e170')"><span class="mb badge-m">M</span>.calc_predictor()</a>+<a class="l3-item l3m" data-target="e171" onclick="showEntry('e171')"><span class="mb badge-m">M</span>.calc_predictor_chunk()</a>+<a class="l3-item l3m" data-target="e172" onclick="showEntry('e172')"><span class="mb badge-m">M</span>.cal_decoder_with_predictor()</a>+<a class="l3-item l3m" data-target="e173" onclick="showEntry('e173')"><span class="mb badge-m">M</span>.cal_decoder_with_predictor_chunk()</a>+<a class="l3-item l3m" data-target="e174" onclick="showEntry('e174')"><span class="mb badge-m">M</span>.init_cache()</a>+<a class="l3-item l3m" data-target="e175" onclick="showEntry('e175')"><span class="mb badge-m">M</span>.generate_chunk()</a>+<a class="l3-item l3m" data-target="e176" onclick="showEntry('e176')"><span class="mb badge-m">M</span>.inference()</a>+<a class="l3-item l3m" data-target="e177" onclick="showEntry('e177')"><span class="mb badge-m">M</span>.export()</a>+</div></div>+<div class="l2"><div class="l2-title" onclick="toggleL2(this)"><span>paraformer_v2_community</span><span class="arr">▸</span></div><div class="l2-children">+<a class="l3-item" data-target="e178" onclick="showEntry('e178')"><span class="mb badge-c">C</span>Paraformer</a>+<a class="l3-item l3m" data-target="e179" onclick="showEntry('e179')"><span class="mb badge-m">M</span>.forward()</a>+<a class="l3-item l3m" data-target="e180" onclick="showEntry('e180')"><span class="mb badge-m">M</span>.encode()</a>+<a class="l3-item l3m" data-target="e181" onclick="showEntry('e181')"><span class="mb badge-m">M</span>.map_alignment_to_target_index()</a>+<a class="l3-item l3m" data-target="e182" onclick="showEntry('e182')"><span class="mb badge-m">M</span>.force_align()</a>+<a class="l3-item l3m" data-target="e183" onclick="showEntry('e183')"><span class="mb badge-m">M</span>.average_repeats_training()</a>+<a class="l3-item l3m" data-target="e184" onclick="showEntry('e184')"><span class="mb badge-m">M</span>.average_repeats_inference()</a>+<a class="l3-item l3m" data-target="e185" onclick="showEntry('e185')"><span class="mb badge-m">M</span>.inference()</a>+<a class="l3-item l3m" data-target="e186" onclick="showEntry('e186')"><span class="mb badge-m">M</span>.export()</a>+</div></div>+<div class="l2"><div class="l2-title" onclick="toggleL2(this)"><span>qwen3_asr</span><span class="arr">▸</span></div><div class="l2-children">+<a class="l3-item" data-target="e187" onclick="showEntry('e187')"><span class="mb badge-c">C</span>Qwen3ASR</a>+<a class="l3-item l3m" data-target="e188" onclick="showEntry('e188')"><span class="mb badge-m">M</span>.forward()</a>+<a class="l3-item l3m" data-target="e189" onclick="showEntry('e189')"><span class="mb badge-m">M</span>.inference()</a>+</div></div>+<div class="l2"><div class="l2-title" onclick="toggleL2(this)"><span>qwen_audio</span><span class="arr">▸</span></div><div class="l2-children">+<a class="l3-item" data-target="e190" onclick="showEntry('e190')"><span class="mb badge-c">C</span>QwenAudioWarp</a>+<a class="l3-item l3m" data-target="e191" onclick="showEntry('e191')"><span class="mb badge-m">M</span>.forward()</a>+<a class="l3-item l3m" data-target="e192" onclick="showEntry('e192')"><span class="mb badge-m">M</span>.inference()</a>+<a class="l3-item" data-target="e193" onclick="showEntry('e193')"><span class="mb badge-c">C</span>QwenAudioChatWarp</a>+<a class="l3-item l3m" data-target="e194" onclick="showEntry('e194')"><span class="mb badge-m">M</span>.forward()</a>+<a class="l3-item l3m" data-target="e195" onclick="showEntry('e195')"><span class="mb badge-m">M</span>.inference()</a>+</div></div>+<div class="l2"><div class="l2-title" onclick="toggleL2(this)"><span>sanm</span><span class="arr">▸</span></div><div class="l2-children">+<a class="l3-item" data-target="e196" onclick="showEntry('e196')"><span class="mb badge-c">C</span>SANM</a>+</div></div>+<div class="l2"><div class="l2-title" onclick="toggleL2(this)"><span>sanm_kws</span><span class="arr">▸</span></div><div class="l2-children">+<a class="l3-item" data-target="e197" onclick="showEntry('e197')"><span class="mb badge-c">C</span>SanmKWS</a>+<a class="l3-item l3m" data-target="e198" onclick="showEntry('e198')"><span class="mb badge-m">M</span>.forward()</a>+<a class="l3-item l3m" data-target="e199" onclick="showEntry('e199')"><span class="mb badge-m">M</span>.encode()</a>+<a class="l3-item l3m" data-target="e200" onclick="showEntry('e200')"><span class="mb badge-m">M</span>.inference()</a>+<a class="l3-item l3m" data-target="e201" onclick="showEntry('e201')"><span class="mb badge-m">M</span>.export()</a>+</div></div>+<div class="l2"><div class="l2-title" onclick="toggleL2(this)"><span>sanm_kws_streaming</span><span class="arr">▸</span></div><div class="l2-children">+<a class="l3-item" data-target="e202" onclick="showEntry('e202')"><span class="mb badge-c">C</span>SanmKWSStreaming</a>+<a class="l3-item l3m" data-target="e203" onclick="showEntry('e203')"><span class="mb badge-m">M</span>.forward()</a>+<a class="l3-item l3m" data-target="e204" onclick="showEntry('e204')"><span class="mb badge-m">M</span>.encode_chunk()</a>+<a class="l3-item l3m" data-target="e205" onclick="showEntry('e205')"><span class="mb badge-m">M</span>.init_cache()</a>+<a class="l3-item l3m" data-target="e206" onclick="showEntry('e206')"><span class="mb badge-m">M</span>.generate_chunk()</a>+<a class="l3-item l3m" data-target="e207" onclick="showEntry('e207')"><span class="mb badge-m">M</span>.inference()</a>+<a class="l3-item l3m" data-target="e208" onclick="showEntry('e208')"><span class="mb badge-m">M</span>.export()</a>+</div></div>+<div class="l2"><div class="l2-title" onclick="toggleL2(this)"><span>scama</span><span class="arr">▸</span></div><div class="l2-children">+<a class="l3-item" data-target="e209" onclick="showEntry('e209')"><span class="mb badge-c">C</span>SCAMA</a>+<a class="l3-item l3m" data-target="e210" onclick="showEntry('e210')"><span class="mb badge-m">M</span>.forward()</a>+<a class="l3-item l3m" data-target="e211" onclick="showEntry('e211')"><span class="mb badge-m">M</span>.encode()</a>+<a class="l3-item l3m" data-target="e212" onclick="showEntry('e212')"><span class="mb badge-m">M</span>.encode_chunk()</a>+<a class="l3-item l3m" data-target="e213" onclick="showEntry('e213')"><span class="mb badge-m">M</span>.calc_predictor_chunk()</a>+<a class="l3-item l3m" data-target="e214" onclick="showEntry('e214')"><span class="mb badge-m">M</span>.calc_predictor_mask()</a>+<a class="l3-item l3m" data-target="e215" onclick="showEntry('e215')"><span class="mb badge-m">M</span>.init_beam_search()</a>+<a class="l3-item l3m" data-target="e216" onclick="showEntry('e216')"><span class="mb badge-m">M</span>.generate_chunk()</a>+<a class="l3-item l3m" data-target="e217" onclick="showEntry('e217')"><span class="mb badge-m">M</span>.init_cache()</a>+<a class="l3-item l3m" data-target="e218" onclick="showEntry('e218')"><span class="mb badge-m">M</span>.inference()</a>+</div></div>+<div class="l2"><div class="l2-title" onclick="toggleL2(this)"><span>seaco_paraformer</span><span class="arr">▸</span></div><div class="l2-children">+<a class="l3-item" data-target="e219" onclick="showEntry('e219')"><span class="mb badge-c">C</span>SeacoParaformer</a>+<a class="l3-item l3m" data-target="e220" onclick="showEntry('e220')"><span class="mb badge-m">M</span>.forward()</a>+<a class="l3-item l3m" data-target="e221" onclick="showEntry('e221')"><span class="mb badge-m">M</span>.calc_predictor()</a>+<a class="l3-item l3m" data-target="e222" onclick="showEntry('e222')"><span class="mb badge-m">M</span>.inference()</a>+<a class="l3-item l3m" data-target="e223" onclick="showEntry('e223')"><span class="mb badge-m">M</span>.generate_hotwords_list()</a>+<a class="l3-item l3m" data-target="e224" onclick="showEntry('e224')"><span class="mb badge-m">M</span>.export()</a>+<a class="l3-item" data-target="e225" onclick="showEntry('e225')"><span class="mb badge-f">F</span>load_seg_dict</a>+</div></div>+<div class="l2"><div class="l2-title" onclick="toggleL2(this)"><span>sense_voice</span><span class="arr">▸</span></div><div class="l2-children">+<a class="l3-item" data-target="e226" onclick="showEntry('e226')"><span class="mb badge-f">F</span>sequence_mask</a>+<a class="l3-item" data-target="e227" onclick="showEntry('e227')"><span class="mb badge-c">C</span>SenseVoiceEncoderSmall</a>+<a class="l3-item l3m" data-target="e228" onclick="showEntry('e228')"><span class="mb badge-m">M</span>.output_size()</a>+<a class="l3-item l3m" data-target="e229" onclick="showEntry('e229')"><span class="mb badge-m">M</span>.forward()</a>+<a class="l3-item" data-target="e230" onclick="showEntry('e230')"><span class="mb badge-c">C</span>SenseVoiceSmall</a>+<a class="l3-item l3m" data-target="e231" onclick="showEntry('e231')"><span class="mb badge-m">M</span>.from_pretrained()</a>+<a class="l3-item l3m" data-target="e232" onclick="showEntry('e232')"><span class="mb badge-m">M</span>.forward()</a>+<a class="l3-item l3m" data-target="e233" onclick="showEntry('e233')"><span class="mb badge-m">M</span>.encode()</a>+<a class="l3-item l3m" data-target="e234" onclick="showEntry('e234')"><span class="mb badge-m">M</span>.inference()</a>+<a class="l3-item l3m" data-target="e235" onclick="showEntry('e235')"><span class="mb badge-m">M</span>.post()</a>+<a class="l3-item l3m" data-target="e236" onclick="showEntry('e236')"><span class="mb badge-m">M</span>.export()</a>+</div></div>+<div class="l2"><div class="l2-title" onclick="toggleL2(this)"><span>transducer</span><span class="arr">▸</span></div><div class="l2-children">+<a class="l3-item" data-target="e237" onclick="showEntry('e237')"><span class="mb badge-c">C</span>Transducer</a>+<a class="l3-item l3m" data-target="e238" onclick="showEntry('e238')"><span class="mb badge-m">M</span>.forward()</a>+<a class="l3-item l3m" data-target="e239" onclick="showEntry('e239')"><span class="mb badge-m">M</span>.encode()</a>+<a class="l3-item l3m" data-target="e240" onclick="showEntry('e240')"><span class="mb badge-m">M</span>.init_beam_search()</a>+<a class="l3-item l3m" data-target="e241" onclick="showEntry('e241')"><span class="mb badge-m">M</span>.inference()</a>+</div></div>+<div class="l2"><div class="l2-title" onclick="toggleL2(this)"><span>transformer</span><span class="arr">▸</span></div><div class="l2-children">+<a class="l3-item" data-target="e242" onclick="showEntry('e242')"><span class="mb badge-c">C</span>Transformer</a>+<a class="l3-item l3m" data-target="e243" onclick="showEntry('e243')"><span class="mb badge-m">M</span>.forward()</a>+<a class="l3-item l3m" data-target="e244" onclick="showEntry('e244')"><span class="mb badge-m">M</span>.encode()</a>+<a class="l3-item l3m" data-target="e245" onclick="showEntry('e245')"><span class="mb badge-m">M</span>.init_beam_search()</a>+<a class="l3-item l3m" data-target="e246" onclick="showEntry('e246')"><span class="mb badge-m">M</span>.inference()</a>+</div></div>+<div class="l2"><div class="l2-title" onclick="toggleL2(this)"><span>uniasr</span><span class="arr">▸</span></div><div class="l2-children">+<a class="l3-item" data-target="e247" onclick="showEntry('e247')"><span class="mb badge-c">C</span>UniASR</a>+<a class="l3-item l3m" data-target="e248" onclick="showEntry('e248')"><span class="mb badge-m">M</span>.forward()</a>+<a class="l3-item l3m" data-target="e249" onclick="showEntry('e249')"><span class="mb badge-m">M</span>.collect_feats()</a>+<a class="l3-item l3m" data-target="e250" onclick="showEntry('e250')"><span class="mb badge-m">M</span>.encode()</a>+<a class="l3-item l3m" data-target="e251" onclick="showEntry('e251')"><span class="mb badge-m">M</span>.encode2()</a>+<a class="l3-item l3m" data-target="e252" onclick="showEntry('e252')"><span class="mb badge-m">M</span>.nll()</a>+<a class="l3-item l3m" data-target="e253" onclick="showEntry('e253')"><span class="mb badge-m">M</span>.batchify_nll()</a>+<a class="l3-item l3m" data-target="e254" onclick="showEntry('e254')"><span class="mb badge-m">M</span>.calc_predictor_mask()</a>+<a class="l3-item l3m" data-target="e255" onclick="showEntry('e255')"><span class="mb badge-m">M</span>.calc_predictor_mask2()</a>+<a class="l3-item l3m" data-target="e256" onclick="showEntry('e256')"><span class="mb badge-m">M</span>.init_beam_search()</a>+<a class="l3-item l3m" data-target="e257" onclick="showEntry('e257')"><span class="mb badge-m">M</span>.inference()</a>+</div></div>+<div class="l2"><div class="l2-title" onclick="toggleL2(this)"><span>whisper</span><span class="arr">▸</span></div><div class="l2-children">+<a class="l3-item" data-target="e258" onclick="showEntry('e258')"><span class="mb badge-c">C</span>WhisperWarp</a>+<a class="l3-item l3m" data-target="e259" onclick="showEntry('e259')"><span class="mb badge-m">M</span>.forward()</a>+<a class="l3-item l3m" data-target="e260" onclick="showEntry('e260')"><span class="mb badge-m">M</span>.inference()</a>+</div></div>+<div class="l2"><div class="l2-title" onclick="toggleL2(this)"><span>whisper_lid</span><span class="arr">▸</span></div><div class="l2-children">+<a class="l3-item" data-target="e261" onclick="showEntry('e261')"><span class="mb badge-c">C</span>OpenAIWhisperModel</a>+<a class="l3-item l3m" data-target="e262" onclick="showEntry('e262')"><span class="mb badge-m">M</span>.forward()</a>+<a class="l3-item l3m" data-target="e263" onclick="showEntry('e263')"><span class="mb badge-m">M</span>.encode()</a>+<a class="l3-item l3m" data-target="e264" onclick="showEntry('e264')"><span class="mb badge-m">M</span>.init_beam_search()</a>+<a class="l3-item l3m" data-target="e265" onclick="showEntry('e265')"><span class="mb badge-m">M</span>.inference()</a>+<a class="l3-item" data-target="e266" onclick="showEntry('e266')"><span class="mb badge-c">C</span>OpenAIWhisperLIDModel</a>+<a class="l3-item l3m" data-target="e267" onclick="showEntry('e267')"><span class="mb badge-m">M</span>.forward()</a>+<a class="l3-item l3m" data-target="e268" onclick="showEntry('e268')"><span class="mb badge-m">M</span>.encode()</a>+<a class="l3-item l3m" data-target="e269" onclick="showEntry('e269')"><span class="mb badge-m">M</span>.inference()</a>+</div></div>+</div></div>+<div class="l1"><div class="l1-title" onclick="toggleL1(this)"><span>funasr.frontends</span><span class="cnt">7</span><span class="arr">▸</span></div><div class="l1-children">+<div class="l2"><div class="l2-title" onclick="toggleL2(this)"><span>default</span><span class="arr">▸</span></div><div class="l2-children">+<a class="l3-item" data-target="e270" onclick="showEntry('e270')"><span class="mb badge-c">C</span>DefaultFrontend</a>+<a class="l3-item l3m" data-target="e271" onclick="showEntry('e271')"><span class="mb badge-m">M</span>.output_size()</a>+<a class="l3-item l3m" data-target="e272" onclick="showEntry('e272')"><span class="mb badge-m">M</span>.forward()</a>+<a class="l3-item" data-target="e273" onclick="showEntry('e273')"><span class="mb badge-c">C</span>MultiChannelFrontend</a>+<a class="l3-item l3m" data-target="e274" onclick="showEntry('e274')"><span class="mb badge-m">M</span>.output_size()</a>+<a class="l3-item l3m" data-target="e275" onclick="showEntry('e275')"><span class="mb badge-m">M</span>.forward()</a>+</div></div>+<div class="l2"><div class="l2-title" onclick="toggleL2(this)"><span>eend_ola_feature</span><span class="arr">▸</span></div><div class="l2-children">+<a class="l3-item" data-target="e276" onclick="showEntry('e276')"><span class="mb badge-f">F</span>transform</a>+<a class="l3-item" data-target="e277" onclick="showEntry('e277')"><span class="mb badge-f">F</span>subsample</a>+<a class="l3-item" data-target="e278" onclick="showEntry('e278')"><span class="mb badge-f">F</span>splice</a>+<a class="l3-item" data-target="e279" onclick="showEntry('e279')"><span class="mb badge-f">F</span>stft</a>+</div></div>+<div class="l2"><div class="l2-title" onclick="toggleL2(this)"><span>fused</span><span class="arr">▸</span></div><div class="l2-children">+<a class="l3-item" data-target="e280" onclick="showEntry('e280')"><span class="mb badge-c">C</span>FusedFrontends</a>+<a class="l3-item l3m" data-target="e281" onclick="showEntry('e281')"><span class="mb badge-m">M</span>.output_size()</a>+<a class="l3-item l3m" data-target="e282" onclick="showEntry('e282')"><span class="mb badge-m">M</span>.forward()</a>+</div></div>+<div class="l2"><div class="l2-title" onclick="toggleL2(this)"><span>s3prl</span><span class="arr">▸</span></div><div class="l2-children">+<a class="l3-item" data-target="e283" onclick="showEntry('e283')"><span class="mb badge-f">F</span>base_s3prl_setup</a>+<a class="l3-item" data-target="e284" onclick="showEntry('e284')"><span class="mb badge-c">C</span>S3prlFrontend</a>+<a class="l3-item l3m" data-target="e285" onclick="showEntry('e285')"><span class="mb badge-m">M</span>.output_size()</a>+<a class="l3-item l3m" data-target="e286" onclick="showEntry('e286')"><span class="mb badge-m">M</span>.forward()</a>+<a class="l3-item l3m" data-target="e287" onclick="showEntry('e287')"><span class="mb badge-m">M</span>.reload_pretrained_parameters()</a>+</div></div>+<div class="l2"><div class="l2-title" onclick="toggleL2(this)"><span>wav_frontend</span><span class="arr">▸</span></div><div class="l2-children">+<a class="l3-item" data-target="e288" onclick="showEntry('e288')"><span class="mb badge-f">F</span>load_cmvn</a>+<a class="l3-item" data-target="e289" onclick="showEntry('e289')"><span class="mb badge-f">F</span>apply_cmvn</a>+<a class="l3-item" data-target="e290" onclick="showEntry('e290')"><span class="mb badge-f">F</span>apply_lfr</a>+<a class="l3-item" data-target="e291" onclick="showEntry('e291')"><span class="mb badge-c">C</span>WavFrontend</a>+<a class="l3-item l3m" data-target="e292" onclick="showEntry('e292')"><span class="mb badge-m">M</span>.output_size()</a>+<a class="l3-item l3m" data-target="e293" onclick="showEntry('e293')"><span class="mb badge-m">M</span>.forward()</a>+<a class="l3-item l3m" data-target="e294" onclick="showEntry('e294')"><span class="mb badge-m">M</span>.forward_fbank()</a>+<a class="l3-item l3m" data-target="e295" onclick="showEntry('e295')"><span class="mb badge-m">M</span>.forward_lfr_cmvn()</a>+<a class="l3-item" data-target="e296" onclick="showEntry('e296')"><span class="mb badge-c">C</span>WavFrontendOnline</a>+<a class="l3-item l3m" data-target="e297" onclick="showEntry('e297')"><span class="mb badge-m">M</span>.output_size()</a>+<a class="l3-item l3m" data-target="e298" onclick="showEntry('e298')"><span class="mb badge-m">M</span>.apply_cmvn()</a>+<a class="l3-item l3m" data-target="e299" onclick="showEntry('e299')"><span class="mb badge-m">M</span>.apply_lfr()</a>+<a class="l3-item l3m" data-target="e300" onclick="showEntry('e300')"><span class="mb badge-m">M</span>.compute_frame_num()</a>+<a class="l3-item l3m" data-target="e301" onclick="showEntry('e301')"><span class="mb badge-m">M</span>.forward_fbank()</a>+<a class="l3-item l3m" data-target="e302" onclick="showEntry('e302')"><span class="mb badge-m">M</span>.forward_lfr_cmvn()</a>+<a class="l3-item l3m" data-target="e303" onclick="showEntry('e303')"><span class="mb badge-m">M</span>.forward()</a>+<a class="l3-item l3m" data-target="e304" onclick="showEntry('e304')"><span class="mb badge-m">M</span>.init_cache()</a>+<a class="l3-item" data-target="e305" onclick="showEntry('e305')"><span class="mb badge-c">C</span>WavFrontendMel23</a>+<a class="l3-item l3m" data-target="e306" onclick="showEntry('e306')"><span class="mb badge-m">M</span>.output_size()</a>+<a class="l3-item l3m" data-target="e307" onclick="showEntry('e307')"><span class="mb badge-m">M</span>.forward()</a>+</div></div>+<div class="l2"><div class="l2-title" onclick="toggleL2(this)"><span>whisper_frontend</span><span class="arr">▸</span></div><div class="l2-children">+<a class="l3-item" data-target="e308" onclick="showEntry('e308')"><span class="mb badge-c">C</span>WhisperFrontend</a>+<a class="l3-item l3m" data-target="e309" onclick="showEntry('e309')"><span class="mb badge-m">M</span>.output_size()</a>+<a class="l3-item l3m" data-target="e310" onclick="showEntry('e310')"><span class="mb badge-m">M</span>.log_mel_spectrogram()</a>+<a class="l3-item l3m" data-target="e311" onclick="showEntry('e311')"><span class="mb badge-m">M</span>.forward()</a>+</div></div>+<div class="l2"><div class="l2-title" onclick="toggleL2(this)"><span>windowing</span><span class="arr">▸</span></div><div class="l2-children">+<a class="l3-item" data-target="e312" onclick="showEntry('e312')"><span class="mb badge-c">C</span>SlidingWindow</a>+<a class="l3-item l3m" data-target="e313" onclick="showEntry('e313')"><span class="mb badge-m">M</span>.forward()</a>+<a class="l3-item l3m" data-target="e314" onclick="showEntry('e314')"><span class="mb badge-m">M</span>.output_size()</a>+</div></div>+</div></div>+<div class="l1"><div class="l1-title" onclick="toggleL1(this)"><span>funasr.tokenizer</span><span class="cnt">10</span><span class="arr">▸</span></div><div class="l1-children">+<div class="l2"><div class="l2-title" onclick="toggleL2(this)"><span>build_tokenizer</span><span class="arr">▸</span></div><div class="l2-children">+<a class="l3-item" data-target="e315" onclick="showEntry('e315')"><span class="mb badge-f">F</span>build_tokenizer</a>+</div></div>+<div class="l2"><div class="l2-title" onclick="toggleL2(this)"><span>char_tokenizer</span><span class="arr">▸</span></div><div class="l2-children">+<a class="l3-item" data-target="e316" onclick="showEntry('e316')"><span class="mb badge-c">C</span>CharTokenizer</a>+<a class="l3-item l3m" data-target="e317" onclick="showEntry('e317')"><span class="mb badge-m">M</span>.text2tokens()</a>+<a class="l3-item l3m" data-target="e318" onclick="showEntry('e318')"><span class="mb badge-m">M</span>.tokens2text()</a>+<a class="l3-item" data-target="e319" onclick="showEntry('e319')"><span class="mb badge-f">F</span>load_seg_dict</a>+<a class="l3-item" data-target="e320" onclick="showEntry('e320')"><span class="mb badge-f">F</span>seg_tokenize</a>+</div></div>+<div class="l2"><div class="l2-title" onclick="toggleL2(this)"><span>cleaner</span><span class="arr">▸</span></div><div class="l2-children">+<a class="l3-item" data-target="e321" onclick="showEntry('e321')"><span class="mb badge-c">C</span>TextCleaner</a>+</div></div>+<div class="l2"><div class="l2-title" onclick="toggleL2(this)"><span>hf_tokenizer</span><span class="arr">▸</span></div><div class="l2-children">+<a class="l3-item" data-target="e322" onclick="showEntry('e322')"><span class="mb badge-f">F</span>HuggingfaceTokenizer</a>+</div></div>+<div class="l2"><div class="l2-title" onclick="toggleL2(this)"><span>korean_cleaner</span><span class="arr">▸</span></div><div class="l2-children">+<a class="l3-item" data-target="e323" onclick="showEntry('e323')"><span class="mb badge-c">C</span>KoreanCleaner</a>+<a class="l3-item l3m" data-target="e324" onclick="showEntry('e324')"><span class="mb badge-m">M</span>.normalize_text()</a>+</div></div>+<div class="l2"><div class="l2-title" onclick="toggleL2(this)"><span>phoneme_tokenizer</span><span class="arr">▸</span></div><div class="l2-children">+<a class="l3-item" data-target="e325" onclick="showEntry('e325')"><span class="mb badge-f">F</span>split_by_space</a>+<a class="l3-item" data-target="e326" onclick="showEntry('e326')"><span class="mb badge-f">F</span>pyopenjtalk_g2p</a>+<a class="l3-item" data-target="e327" onclick="showEntry('e327')"><span class="mb badge-f">F</span>pyopenjtalk_g2p_accent</a>+<a class="l3-item" data-target="e328" onclick="showEntry('e328')"><span class="mb badge-f">F</span>pyopenjtalk_g2p_accent_with_pause</a>+<a class="l3-item" data-target="e329" onclick="showEntry('e329')"><span class="mb badge-f">F</span>pyopenjtalk_g2p_kana</a>+<a class="l3-item" data-target="e330" onclick="showEntry('e330')"><span class="mb badge-f">F</span>pyopenjtalk_g2p_prosody</a>+<a class="l3-item" data-target="e331" onclick="showEntry('e331')"><span class="mb badge-f">F</span>pypinyin_g2p</a>+<a class="l3-item" data-target="e332" onclick="showEntry('e332')"><span class="mb badge-f">F</span>pypinyin_g2p_phone</a>+<a class="l3-item" data-target="e333" onclick="showEntry('e333')"><span class="mb badge-c">C</span>G2p_en</a>+<a class="l3-item" data-target="e334" onclick="showEntry('e334')"><span class="mb badge-c">C</span>G2pk</a>+<a class="l3-item" data-target="e335" onclick="showEntry('e335')"><span class="mb badge-c">C</span>Jaso</a>+<a class="l3-item" data-target="e336" onclick="showEntry('e336')"><span class="mb badge-c">C</span>Phonemizer</a>+<a class="l3-item" data-target="e337" onclick="showEntry('e337')"><span class="mb badge-c">C</span>PhonemeTokenizer</a>+<a class="l3-item l3m" data-target="e338" onclick="showEntry('e338')"><span class="mb badge-m">M</span>.text2tokens()</a>+<a class="l3-item l3m" data-target="e339" onclick="showEntry('e339')"><span class="mb badge-m">M</span>.tokens2text()</a>+</div></div>+<div class="l2"><div class="l2-title" onclick="toggleL2(this)"><span>sentencepiece_tokenizer</span><span class="arr">▸</span></div><div class="l2-children">+<a class="l3-item" data-target="e340" onclick="showEntry('e340')"><span class="mb badge-c">C</span>SentencepiecesTokenizer</a>+<a class="l3-item l3m" data-target="e341" onclick="showEntry('e341')"><span class="mb badge-m">M</span>.text2tokens()</a>+<a class="l3-item l3m" data-target="e342" onclick="showEntry('e342')"><span class="mb badge-m">M</span>.tokens2text()</a>+<a class="l3-item l3m" data-target="e343" onclick="showEntry('e343')"><span class="mb badge-m">M</span>.encode()</a>+<a class="l3-item l3m" data-target="e344" onclick="showEntry('e344')"><span class="mb badge-m">M</span>.decode()</a>+<a class="l3-item l3m" data-target="e345" onclick="showEntry('e345')"><span class="mb badge-m">M</span>.get_vocab_size()</a>+<a class="l3-item l3m" data-target="e346" onclick="showEntry('e346')"><span class="mb badge-m">M</span>.ids2tokens()</a>+<a class="l3-item l3m" data-target="e347" onclick="showEntry('e347')"><span class="mb badge-m">M</span>.tokens2ids()</a>+</div></div>+<div class="l2"><div class="l2-title" onclick="toggleL2(this)"><span>token_id_converter</span><span class="arr">▸</span></div><div class="l2-children">+<a class="l3-item" data-target="e348" onclick="showEntry('e348')"><span class="mb badge-c">C</span>TokenIDConverter</a>+<a class="l3-item l3m" data-target="e349" onclick="showEntry('e349')"><span class="mb badge-m">M</span>.get_num_vocabulary_size()</a>+<a class="l3-item l3m" data-target="e350" onclick="showEntry('e350')"><span class="mb badge-m">M</span>.ids2tokens()</a>+<a class="l3-item l3m" data-target="e351" onclick="showEntry('e351')"><span class="mb badge-m">M</span>.tokens2ids()</a>+</div></div>+<div class="l2"><div class="l2-title" onclick="toggleL2(this)"><span>whisper_tokenizer</span><span class="arr">▸</span></div><div class="l2-children">+<a class="l3-item" data-target="e352" onclick="showEntry('e352')"><span class="mb badge-f">F</span>WhisperTokenizer</a>+<a class="l3-item" data-target="e353" onclick="showEntry('e353')"><span class="mb badge-f">F</span>SenseVoiceTokenizer</a>+</div></div>+<div class="l2"><div class="l2-title" onclick="toggleL2(this)"><span>word_tokenizer</span><span class="arr">▸</span></div><div class="l2-children">+<a class="l3-item" data-target="e354" onclick="showEntry('e354')"><span class="mb badge-c">C</span>WordTokenizer</a>+<a class="l3-item l3m" data-target="e355" onclick="showEntry('e355')"><span class="mb badge-m">M</span>.text2tokens()</a>+<a class="l3-item l3m" data-target="e356" onclick="showEntry('e356')"><span class="mb badge-m">M</span>.tokens2text()</a>+</div></div>+</div></div>+<div class="l1"><div class="l1-title" onclick="toggleL1(this)"><span>funasr.utils</span><span class="cnt">15</span><span class="arr">▸</span></div><div class="l1-children">+<div class="l2"><div class="l2-title" onclick="toggleL2(this)"><span>compute_det_ctc</span><span class="arr">▸</span></div><div class="l2-children">+<a class="l3-item" data-target="e357" onclick="showEntry('e357')"><span class="mb badge-c">C</span>thread_wrapper</a>+<a class="l3-item l3m" data-target="e358" onclick="showEntry('e358')"><span class="mb badge-m">M</span>.run()</a>+<a class="l3-item l3m" data-target="e359" onclick="showEntry('e359')"><span class="mb badge-m">M</span>.get_result()</a>+<a class="l3-item" data-target="e360" onclick="showEntry('e360')"><span class="mb badge-f">F</span>space_mixed_label</a>+<a class="l3-item" data-target="e361" onclick="showEntry('e361')"><span class="mb badge-f">F</span>read_lists</a>+<a class="l3-item" data-target="e362" onclick="showEntry('e362')"><span class="mb badge-f">F</span>make_pair</a>+<a class="l3-item" data-target="e363" onclick="showEntry('e363')"><span class="mb badge-f">F</span>count_duration</a>+<a class="l3-item" data-target="e364" onclick="showEntry('e364')"><span class="mb badge-f">F</span>load_data_and_score</a>+</div></div>+<div class="l2"><div class="l2-title" onclick="toggleL2(this)"><span>datadir_writer</span><span class="arr">▸</span></div><div class="l2-children">+<a class="l3-item" data-target="e365" onclick="showEntry('e365')"><span class="mb badge-c">C</span>DatadirWriter</a>+<a class="l3-item l3m" data-target="e366" onclick="showEntry('e366')"><span class="mb badge-m">M</span>.close()</a>+</div></div>+<div class="l2"><div class="l2-title" onclick="toggleL2(this)"><span>dynamic_import</span><span class="arr">▸</span></div><div class="l2-children">+<a class="l3-item" data-target="e367" onclick="showEntry('e367')"><span class="mb badge-f">F</span>load_module_from_path</a>+<a class="l3-item" data-target="e368" onclick="showEntry('e368')"><span class="mb badge-f">F</span>import_module_from_path</a>+</div></div>+<div class="l2"><div class="l2-title" onclick="toggleL2(this)"><span>export_utils</span><span class="arr">▸</span></div><div class="l2-children">+<a class="l3-item" data-target="e369" onclick="showEntry('e369')"><span class="mb badge-f">F</span>export</a>+</div></div>+<div class="l2"><div class="l2-title" onclick="toggleL2(this)"><span>install_model_requirements</span><span class="arr">▸</span></div><div class="l2-children">+<a class="l3-item" data-target="e370" onclick="showEntry('e370')"><span class="mb badge-f">F</span>install_requirements</a>+<a class="l3-item" data-target="e371" onclick="showEntry('e371')"><span class="mb badge-f">F</span>pip_install_r</a>+</div></div>+<div class="l2"><div class="l2-title" onclick="toggleL2(this)"><span>kws_utils</span><span class="arr">▸</span></div><div class="l2-children">+<a class="l3-item" data-target="e372" onclick="showEntry('e372')"><span class="mb badge-f">F</span>split_mixed_label</a>+<a class="l3-item" data-target="e373" onclick="showEntry('e373')"><span class="mb badge-f">F</span>query_token_set</a>+<a class="l3-item" data-target="e374" onclick="showEntry('e374')"><span class="mb badge-c">C</span>KwsCtcPrefixDecoder</a>+<a class="l3-item l3m" data-target="e375" onclick="showEntry('e375')"><span class="mb badge-m">M</span>.beam_search()</a>+<a class="l3-item l3m" data-target="e376" onclick="showEntry('e376')"><span class="mb badge-m">M</span>.is_sublist()</a>+<a class="l3-item l3m" data-target="e377" onclick="showEntry('e377')"><span class="mb badge-m">M</span>.decode()</a>+</div></div>+<div class="l2"><div class="l2-title" onclick="toggleL2(this)"><span>load_utils</span><span class="arr">▸</span></div><div class="l2-children">+<a class="l3-item" data-target="e378" onclick="showEntry('e378')"><span class="mb badge-f">F</span>is_ffmpeg_installed</a>+<a class="l3-item" data-target="e379" onclick="showEntry('e379')"><span class="mb badge-f">F</span>load_audio_text_image_video</a>+<a class="l3-item" data-target="e380" onclick="showEntry('e380')"><span class="mb badge-f">F</span>load_bytes</a>+<a class="l3-item" data-target="e381" onclick="showEntry('e381')"><span class="mb badge-f">F</span>validate_frame_rate</a>+<a class="l3-item" data-target="e382" onclick="showEntry('e382')"><span class="mb badge-f">F</span>extract_fbank</a>+</div></div>+<div class="l2"><div class="l2-title" onclick="toggleL2(this)"><span>misc</span><span class="arr">▸</span></div><div class="l2-children">+<a class="l3-item" data-target="e383" onclick="showEntry('e383')"><span class="mb badge-f">F</span>statistic_model_parameters</a>+<a class="l3-item" data-target="e384" onclick="showEntry('e384')"><span class="mb badge-f">F</span>int2vec</a>+<a class="l3-item" data-target="e385" onclick="showEntry('e385')"><span class="mb badge-f">F</span>seq2arr</a>+<a class="l3-item" data-target="e386" onclick="showEntry('e386')"><span class="mb badge-f">F</span>load_scp_as_dict</a>+<a class="l3-item" data-target="e387" onclick="showEntry('e387')"><span class="mb badge-f">F</span>load_scp_as_list</a>+<a class="l3-item" data-target="e388" onclick="showEntry('e388')"><span class="mb badge-f">F</span>deep_update</a>+<a class="l3-item" data-target="e389" onclick="showEntry('e389')"><span class="mb badge-f">F</span>prepare_model_dir</a>+<a class="l3-item" data-target="e390" onclick="showEntry('e390')"><span class="mb badge-f">F</span>extract_filename_without_extension</a>+<a class="l3-item" data-target="e391" onclick="showEntry('e391')"><span class="mb badge-f">F</span>smart_remove</a>+</div></div>+<div class="l2"><div class="l2-title" onclick="toggleL2(this)"><span>postprocess_utils</span><span class="arr">▸</span></div><div class="l2-children">+<a class="l3-item" data-target="e392" onclick="showEntry('e392')"><span class="mb badge-f">F</span>isChinese</a>+<a class="l3-item" data-target="e393" onclick="showEntry('e393')"><span class="mb badge-f">F</span>isAllChinese</a>+<a class="l3-item" data-target="e394" onclick="showEntry('e394')"><span class="mb badge-f">F</span>isAllAlpha</a>+<a class="l3-item" data-target="e395" onclick="showEntry('e395')"><span class="mb badge-f">F</span>abbr_dispose</a>+<a class="l3-item" data-target="e396" onclick="showEntry('e396')"><span class="mb badge-f">F</span>sentence_postprocess</a>+<a class="l3-item" data-target="e397" onclick="showEntry('e397')"><span class="mb badge-f">F</span>sentence_postprocess_sentencepiece</a>+<a class="l3-item" data-target="e398" onclick="showEntry('e398')"><span class="mb badge-f">F</span>format_str_v2</a>+<a class="l3-item" data-target="e399" onclick="showEntry('e399')"><span class="mb badge-f">F</span>rich_transcription_postprocess</a>+</div></div>+<div class="l2"><div class="l2-title" onclick="toggleL2(this)"><span>speaker_utils</span><span class="arr">▸</span></div><div class="l2-children">+<a class="l3-item" data-target="e400" onclick="showEntry('e400')"><span class="mb badge-f">F</span>check_audio_list</a>+<a class="l3-item" data-target="e401" onclick="showEntry('e401')"><span class="mb badge-f">F</span>sv_preprocess</a>+<a class="l3-item" data-target="e402" onclick="showEntry('e402')"><span class="mb badge-f">F</span>sv_chunk</a>+<a class="l3-item" data-target="e403" onclick="showEntry('e403')"><span class="mb badge-f">F</span>extract_feature</a>+<a class="l3-item" data-target="e404" onclick="showEntry('e404')"><span class="mb badge-f">F</span>postprocess</a>+<a class="l3-item" data-target="e405" onclick="showEntry('e405')"><span class="mb badge-f">F</span>correct_labels</a>+<a class="l3-item" data-target="e406" onclick="showEntry('e406')"><span class="mb badge-f">F</span>merge_seque</a>+<a class="l3-item" data-target="e407" onclick="showEntry('e407')"><span class="mb badge-f">F</span>smooth</a>+<a class="l3-item" data-target="e408" onclick="showEntry('e408')"><span class="mb badge-f">F</span>distribute_spk</a>+</div></div>+<div class="l2"><div class="l2-title" onclick="toggleL2(this)"><span>timestamp_tools</span><span class="arr">▸</span></div><div class="l2-children">+<a class="l3-item" data-target="e409" onclick="showEntry('e409')"><span class="mb badge-f">F</span>cif_wo_hidden</a>+<a class="l3-item" data-target="e410" onclick="showEntry('e410')"><span class="mb badge-f">F</span>ts_prediction_lfr6_standard</a>+<a class="l3-item" data-target="e411" onclick="showEntry('e411')"><span class="mb badge-f">F</span>timestamp_sentence</a>+<a class="l3-item" data-target="e412" onclick="showEntry('e412')"><span class="mb badge-f">F</span>timestamp_sentence_en</a>+</div></div>+<div class="l2"><div class="l2-title" onclick="toggleL2(this)"><span>torch_function</span><span class="arr">▸</span></div><div class="l2-children">+<a class="l3-item" data-target="e413" onclick="showEntry('e413')"><span class="mb badge-c">C</span>MakePadMask</a>+<a class="l3-item l3m" data-target="e414" onclick="showEntry('e414')"><span class="mb badge-m">M</span>.forward()</a>+<a class="l3-item" data-target="e415" onclick="showEntry('e415')"><span class="mb badge-c">C</span>sequence_mask</a>+<a class="l3-item l3m" data-target="e416" onclick="showEntry('e416')"><span class="mb badge-m">M</span>.forward()</a>+<a class="l3-item" data-target="e417" onclick="showEntry('e417')"><span class="mb badge-f">F</span>normalize</a>+<a class="l3-item" data-target="e418" onclick="showEntry('e418')"><span class="mb badge-f">F</span>subsequent_mask</a>+<a class="l3-item" data-target="e419" onclick="showEntry('e419')"><span class="mb badge-f">F</span>MakePadMask_test</a>+</div></div>+<div class="l2"><div class="l2-title" onclick="toggleL2(this)"><span>type_utils</span><span class="arr">▸</span></div><div class="l2-children">+<a class="l3-item" data-target="e420" onclick="showEntry('e420')"><span class="mb badge-f">F</span>str2bool</a>+<a class="l3-item" data-target="e421" onclick="showEntry('e421')"><span class="mb badge-f">F</span>remove_parenthesis</a>+<a class="l3-item" data-target="e422" onclick="showEntry('e422')"><span class="mb badge-f">F</span>remove_quotes</a>+<a class="l3-item" data-target="e423" onclick="showEntry('e423')"><span class="mb badge-f">F</span>int_or_none</a>+<a class="l3-item" data-target="e424" onclick="showEntry('e424')"><span class="mb badge-f">F</span>float_or_none</a>+<a class="l3-item" data-target="e425" onclick="showEntry('e425')"><span class="mb badge-f">F</span>humanfriendly_parse_size_or_none</a>+<a class="l3-item" data-target="e426" onclick="showEntry('e426')"><span class="mb badge-f">F</span>str_or_int</a>+<a class="l3-item" data-target="e427" onclick="showEntry('e427')"><span class="mb badge-f">F</span>str_or_none</a>+<a class="l3-item" data-target="e428" onclick="showEntry('e428')"><span class="mb badge-f">F</span>str2pair_str</a>+<a class="l3-item" data-target="e429" onclick="showEntry('e429')"><span class="mb badge-f">F</span>str2triple_str</a>+</div></div>+<div class="l2"><div class="l2-title" onclick="toggleL2(this)"><span>vad_utils</span><span class="arr">▸</span></div><div class="l2-children">+<a class="l3-item" data-target="e430" onclick="showEntry('e430')"><span class="mb badge-f">F</span>slice_padding_fbank</a>+<a class="l3-item" data-target="e431" onclick="showEntry('e431')"><span class="mb badge-f">F</span>slice_padding_audio_samples</a>+<a class="l3-item" data-target="e432" onclick="showEntry('e432')"><span class="mb badge-f">F</span>merge_vad</a>+</div></div>+<div class="l2"><div class="l2-title" onclick="toggleL2(this)"><span>version_checker</span><span class="arr">▸</span></div><div class="l2-children">+<a class="l3-item" data-target="e433" onclick="showEntry('e433')"><span class="mb badge-f">F</span>get_pypi_version</a>+<a class="l3-item" data-target="e434" onclick="showEntry('e434')"><span class="mb badge-f">F</span>check_for_update</a>+</div></div>+</div></div>+<div class="l1"><div class="l1-title" onclick="toggleL1(this)"><span>funasr.bin</span><span class="cnt">6</span><span class="arr">▸</span></div><div class="l1-children">+<div class="l2"><div class="l2-title" onclick="toggleL2(this)"><span>compute_audio_cmvn</span><span class="arr">▸</span></div><div class="l2-children">+<a class="l3-item" data-target="e435" onclick="showEntry('e435')"><span class="mb badge-f">F</span>main_hydra</a>+<a class="l3-item" data-target="e436" onclick="showEntry('e436')"><span class="mb badge-f">F</span>main</a>+</div></div>+<div class="l2"><div class="l2-title" onclick="toggleL2(this)"><span>export</span><span class="arr">▸</span></div><div class="l2-children">+<a class="l3-item" data-target="e437" onclick="showEntry('e437')"><span class="mb badge-f">F</span>main_hydra</a>+</div></div>+<div class="l2"><div class="l2-title" onclick="toggleL2(this)"><span>inference</span><span class="arr">▸</span></div><div class="l2-children">+<a class="l3-item" data-target="e438" onclick="showEntry('e438')"><span class="mb badge-f">F</span>main_hydra</a>+</div></div>+<div class="l2"><div class="l2-title" onclick="toggleL2(this)"><span>tokenize_text</span><span class="arr">▸</span></div><div class="l2-children">+<a class="l3-item" data-target="e439" onclick="showEntry('e439')"><span class="mb badge-f">F</span>field2slice</a>+<a class="l3-item" data-target="e440" onclick="showEntry('e440')"><span class="mb badge-f">F</span>tokenize</a>+<a class="l3-item" data-target="e441" onclick="showEntry('e441')"><span class="mb badge-f">F</span>get_parser</a>+<a class="l3-item" data-target="e442" onclick="showEntry('e442')"><span class="mb badge-f">F</span>main</a>+</div></div>+<div class="l2"><div class="l2-title" onclick="toggleL2(this)"><span>train</span><span class="arr">▸</span></div><div class="l2-children">+<a class="l3-item" data-target="e443" onclick="showEntry('e443')"><span class="mb badge-f">F</span>main_hydra</a>+<a class="l3-item" data-target="e444" onclick="showEntry('e444')"><span class="mb badge-f">F</span>main</a>+</div></div>+<div class="l2"><div class="l2-title" onclick="toggleL2(this)"><span>train_ds</span><span class="arr">▸</span></div><div class="l2-children">+<a class="l3-item" data-target="e445" onclick="showEntry('e445')"><span class="mb badge-f">F</span>main_hydra</a>+<a class="l3-item" data-target="e446" onclick="showEntry('e446')"><span class="mb badge-f">F</span>main</a>+</div></div>+</div></div>+<div class="l1"><div class="l1-title" onclick="toggleL1(this)"><span>funasr.download</span><span class="cnt">4</span><span class="arr">▸</span></div><div class="l1-children">+<div class="l2"><div class="l2-title" onclick="toggleL2(this)"><span>download_dataset_from_hub</span><span class="arr">▸</span></div><div class="l2-children">+<a class="l3-item" data-target="e447" onclick="showEntry('e447')"><span class="mb badge-f">F</span>download_dataset</a>+<a class="l3-item" data-target="e448" onclick="showEntry('e448')"><span class="mb badge-f">F</span>download_dataset_from_ms</a>+</div></div>+<div class="l2"><div class="l2-title" onclick="toggleL2(this)"><span>download_model_from_hub</span><span class="arr">▸</span></div><div class="l2-children">+<a class="l3-item" data-target="e449" onclick="showEntry('e449')"><span class="mb badge-f">F</span>download_model</a>+<a class="l3-item" data-target="e450" onclick="showEntry('e450')"><span class="mb badge-f">F</span>download_from_ms</a>+<a class="l3-item" data-target="e451" onclick="showEntry('e451')"><span class="mb badge-f">F</span>download_from_hf</a>+<a class="l3-item" data-target="e452" onclick="showEntry('e452')"><span class="mb badge-f">F</span>add_file_root_path</a>+<a class="l3-item" data-target="e453" onclick="showEntry('e453')"><span class="mb badge-f">F</span>get_or_download_model_dir</a>+<a class="l3-item" data-target="e454" onclick="showEntry('e454')"><span class="mb badge-f">F</span>get_or_download_model_dir_hf</a>+</div></div>+<div class="l2"><div class="l2-title" onclick="toggleL2(this)"><span>file</span><span class="arr">▸</span></div><div class="l2-children">+<a class="l3-item" data-target="e455" onclick="showEntry('e455')"><span class="mb badge-f">F</span>download_from_url</a>+<a class="l3-item" data-target="e456" onclick="showEntry('e456')"><span class="mb badge-c">C</span>Storage</a>+<a class="l3-item l3m" data-target="e457" onclick="showEntry('e457')"><span class="mb badge-m">M</span>.read()</a>+<a class="l3-item l3m" data-target="e458" onclick="showEntry('e458')"><span class="mb badge-m">M</span>.read_text()</a>+<a class="l3-item l3m" data-target="e459" onclick="showEntry('e459')"><span class="mb badge-m">M</span>.write()</a>+<a class="l3-item l3m" data-target="e460" onclick="showEntry('e460')"><span class="mb badge-m">M</span>.write_text()</a>+<a class="l3-item" data-target="e461" onclick="showEntry('e461')"><span class="mb badge-c">C</span>LocalStorage</a>+<a class="l3-item l3m" data-target="e462" onclick="showEntry('e462')"><span class="mb badge-m">M</span>.read()</a>+<a class="l3-item l3m" data-target="e463" onclick="showEntry('e463')"><span class="mb badge-m">M</span>.read_text()</a>+<a class="l3-item l3m" data-target="e464" onclick="showEntry('e464')"><span class="mb badge-m">M</span>.write()</a>+<a class="l3-item l3m" data-target="e465" onclick="showEntry('e465')"><span class="mb badge-m">M</span>.write_text()</a>+<a class="l3-item l3m" data-target="e466" onclick="showEntry('e466')"><span class="mb badge-m">M</span>.as_local_path()</a>+<a class="l3-item" data-target="e467" onclick="showEntry('e467')"><span class="mb badge-c">C</span>HTTPStorage</a>+<a class="l3-item l3m" data-target="e468" onclick="showEntry('e468')"><span class="mb badge-m">M</span>.read()</a>+<a class="l3-item l3m" data-target="e469" onclick="showEntry('e469')"><span class="mb badge-m">M</span>.read_text()</a>+<a class="l3-item l3m" data-target="e470" onclick="showEntry('e470')"><span class="mb badge-m">M</span>.as_local_path()</a>+<a class="l3-item l3m" data-target="e471" onclick="showEntry('e471')"><span class="mb badge-m">M</span>.write()</a>+<a class="l3-item l3m" data-target="e472" onclick="showEntry('e472')"><span class="mb badge-m">M</span>.write_text()</a>+<a class="l3-item" data-target="e473" onclick="showEntry('e473')"><span class="mb badge-c">C</span>OSSStorage</a>+<a class="l3-item l3m" data-target="e474" onclick="showEntry('e474')"><span class="mb badge-m">M</span>.read()</a>+<a class="l3-item l3m" data-target="e475" onclick="showEntry('e475')"><span class="mb badge-m">M</span>.read_text()</a>+<a class="l3-item l3m" data-target="e476" onclick="showEntry('e476')"><span class="mb badge-m">M</span>.as_local_path()</a>+<a class="l3-item l3m" data-target="e477" onclick="showEntry('e477')"><span class="mb badge-m">M</span>.write()</a>+<a class="l3-item l3m" data-target="e478" onclick="showEntry('e478')"><span class="mb badge-m">M</span>.write_text()</a>+<a class="l3-item" data-target="e479" onclick="showEntry('e479')"><span class="mb badge-c">C</span>File</a>+<a class="l3-item l3m" data-target="e480" onclick="showEntry('e480')"><span class="mb badge-m">M</span>.read()</a>+<a class="l3-item l3m" data-target="e481" onclick="showEntry('e481')"><span class="mb badge-m">M</span>.read_text()</a>+<a class="l3-item l3m" data-target="e482" onclick="showEntry('e482')"><span class="mb badge-m">M</span>.write()</a>+<a class="l3-item l3m" data-target="e483" onclick="showEntry('e483')"><span class="mb badge-m">M</span>.write_text()</a>+<a class="l3-item l3m" data-target="e484" onclick="showEntry('e484')"><span class="mb badge-m">M</span>.as_local_path()</a>+</div></div>+<div class="l2"><div class="l2-title" onclick="toggleL2(this)"><span>runtime_sdk_download_tool</span><span class="arr">▸</span></div><div class="l2-children">+<a class="l3-item" data-target="e485" onclick="showEntry('e485')"><span class="mb badge-f">F</span>main</a>+</div></div>+</div></div>+<div class="l1"><div class="l1-title" onclick="toggleL1(this)"><span>funasr.train_utils</span><span class="cnt">11</span><span class="arr">▸</span></div><div class="l1-children">+<div class="l2"><div class="l2-title" onclick="toggleL2(this)"><span>add_gradient_noise</span><span class="arr">▸</span></div><div class="l2-children">+<a class="l3-item" data-target="e486" onclick="showEntry('e486')"><span class="mb badge-f">F</span>add_gradient_noise</a>+</div></div>+<div class="l2"><div class="l2-title" onclick="toggleL2(this)"><span>average_nbest_models</span><span class="arr">▸</span></div><div class="l2-children">+<a class="l3-item" data-target="e487" onclick="showEntry('e487')"><span class="mb badge-f">F</span>average_checkpoints</a>+</div></div>+<div class="l2"><div class="l2-title" onclick="toggleL2(this)"><span>device_funcs</span><span class="arr">▸</span></div><div class="l2-children">+<a class="l3-item" data-target="e488" onclick="showEntry('e488')"><span class="mb badge-f">F</span>to_device</a>+<a class="l3-item" data-target="e489" onclick="showEntry('e489')"><span class="mb badge-f">F</span>force_gatherable</a>+</div></div>+<div class="l2"><div class="l2-title" onclick="toggleL2(this)"><span>forward_adaptor</span><span class="arr">▸</span></div><div class="l2-children">+<a class="l3-item" data-target="e490" onclick="showEntry('e490')"><span class="mb badge-c">C</span>ForwardAdaptor</a>+<a class="l3-item l3m" data-target="e491" onclick="showEntry('e491')"><span class="mb badge-m">M</span>.forward()</a>+</div></div>+<div class="l2"><div class="l2-title" onclick="toggleL2(this)"><span>initialize</span><span class="arr">▸</span></div><div class="l2-children">+<a class="l3-item" data-target="e492" onclick="showEntry('e492')"><span class="mb badge-f">F</span>initialize</a>+</div></div>+<div class="l2"><div class="l2-title" onclick="toggleL2(this)"><span>load_pretrained_model</span><span class="arr">▸</span></div><div class="l2-children">+<a class="l3-item" data-target="e493" onclick="showEntry('e493')"><span class="mb badge-f">F</span>load_pretrained_model</a>+</div></div>+<div class="l2"><div class="l2-title" onclick="toggleL2(this)"><span>model_summary</span><span class="arr">▸</span></div><div class="l2-children">+<a class="l3-item" data-target="e494" onclick="showEntry('e494')"><span class="mb badge-f">F</span>get_human_readable_count</a>+<a class="l3-item" data-target="e495" onclick="showEntry('e495')"><span class="mb badge-f">F</span>to_bytes</a>+<a class="l3-item" data-target="e496" onclick="showEntry('e496')"><span class="mb badge-f">F</span>model_summary</a>+</div></div>+<div class="l2"><div class="l2-title" onclick="toggleL2(this)"><span>recursive_op</span><span class="arr">▸</span></div><div class="l2-children">+<a class="l3-item" data-target="e497" onclick="showEntry('e497')"><span class="mb badge-f">F</span>recursive_sum</a>+<a class="l3-item" data-target="e498" onclick="showEntry('e498')"><span class="mb badge-f">F</span>recursive_divide</a>+<a class="l3-item" data-target="e499" onclick="showEntry('e499')"><span class="mb badge-f">F</span>recursive_average</a>+</div></div>+<div class="l2"><div class="l2-title" onclick="toggleL2(this)"><span>set_all_random_seed</span><span class="arr">▸</span></div><div class="l2-children">+<a class="l3-item" data-target="e500" onclick="showEntry('e500')"><span class="mb badge-f">F</span>set_all_random_seed</a>+</div></div>+<div class="l2"><div class="l2-title" onclick="toggleL2(this)"><span>trainer</span><span class="arr">▸</span></div><div class="l2-children">+<a class="l3-item" data-target="e501" onclick="showEntry('e501')"><span class="mb badge-f">F</span>maybe_autocast</a>+<a class="l3-item" data-target="e502" onclick="showEntry('e502')"><span class="mb badge-c">C</span>Trainer</a>+<a class="l3-item l3m" data-target="e503" onclick="showEntry('e503')"><span class="mb badge-m">M</span>.save_checkpoint()</a>+<a class="l3-item l3m" data-target="e504" onclick="showEntry('e504')"><span class="mb badge-m">M</span>.resume_checkpoint()</a>+<a class="l3-item l3m" data-target="e505" onclick="showEntry('e505')"><span class="mb badge-m">M</span>.train_epoch()</a>+<a class="l3-item l3m" data-target="e506" onclick="showEntry('e506')"><span class="mb badge-m">M</span>.validate_epoch()</a>+<a class="l3-item l3m" data-target="e507" onclick="showEntry('e507')"><span class="mb badge-m">M</span>.log()</a>+<a class="l3-item l3m" data-target="e508" onclick="showEntry('e508')"><span class="mb badge-m">M</span>.close()</a>+</div></div>+<div class="l2"><div class="l2-title" onclick="toggleL2(this)"><span>trainer_ds</span><span class="arr">▸</span></div><div class="l2-children">+<a class="l3-item" data-target="e509" onclick="showEntry('e509')"><span class="mb badge-f">F</span>maybe_autocast</a>+<a class="l3-item" data-target="e510" onclick="showEntry('e510')"><span class="mb badge-c">C</span>Trainer</a>+<a class="l3-item l3m" data-target="e511" onclick="showEntry('e511')"><span class="mb badge-m">M</span>.save_checkpoint()</a>+<a class="l3-item l3m" data-target="e512" onclick="showEntry('e512')"><span class="mb badge-m">M</span>.resume_checkpoint()</a>+<a class="l3-item l3m" data-target="e513" onclick="showEntry('e513')"><span class="mb badge-m">M</span>.train_epoch()</a>+<a class="l3-item l3m" data-target="e514" onclick="showEntry('e514')"><span class="mb badge-m">M</span>.forward_step()</a>+<a class="l3-item l3m" data-target="e515" onclick="showEntry('e515')"><span class="mb badge-m">M</span>.backward_step()</a>+<a class="l3-item l3m" data-target="e516" onclick="showEntry('e516')"><span class="mb badge-m">M</span>.update_step()</a>+<a class="l3-item l3m" data-target="e517" onclick="showEntry('e517')"><span class="mb badge-m">M</span>.validate_epoch()</a>+<a class="l3-item l3m" data-target="e518" onclick="showEntry('e518')"><span class="mb badge-m">M</span>.log()</a>+<a class="l3-item l3m" data-target="e519" onclick="showEntry('e519')"><span class="mb badge-m">M</span>.close()</a>+<a class="l3-item l3m" data-target="e520" onclick="showEntry('e520')"><span class="mb badge-m">M</span>.warp_model()</a>+<a class="l3-item l3m" data-target="e521" onclick="showEntry('e521')"><span class="mb badge-m">M</span>.warp_optim_scheduler()</a>+</div></div>+</div></div>+<div class="l1"><div class="l1-title" onclick="toggleL1(this)"><span>funasr.datasets</span><span class="cnt">11</span><span class="arr">▸</span></div><div class="l1-children">+<div class="l2"><div class="l2-title" onclick="toggleL2(this)"><span>datasets</span><span class="arr">▸</span></div><div class="l2-children">+<a class="l3-item" data-target="e522" onclick="showEntry('e522')"><span class="mb badge-c">C</span>AudioDataset</a>+<a class="l3-item l3m" data-target="e523" onclick="showEntry('e523')"><span class="mb badge-m">M</span>.get_source_len()</a>+<a class="l3-item l3m" data-target="e524" onclick="showEntry('e524')"><span class="mb badge-m">M</span>.get_target_len()</a>+<a class="l3-item l3m" data-target="e525" onclick="showEntry('e525')"><span class="mb badge-m">M</span>.collator()</a>+<a class="l3-item" data-target="e526" onclick="showEntry('e526')"><span class="mb badge-c">C</span>AudioDatasetHotword</a>+<a class="l3-item l3m" data-target="e527" onclick="showEntry('e527')"><span class="mb badge-m">M</span>.collator()</a>+</div></div>+<div class="l2"><div class="l2-title" onclick="toggleL2(this)"><span>espnet_samplers</span><span class="arr">▸</span></div><div class="l2-children">+<a class="l3-item" data-target="e528" onclick="showEntry('e528')"><span class="mb badge-f">F</span>EspnetStyleBatchSampler_fn</a>+<a class="l3-item" data-target="e529" onclick="showEntry('e529')"><span class="mb badge-c">C</span>EspnetStyleBatchSampler</a>+<a class="l3-item l3m" data-target="e530" onclick="showEntry('e530')"><span class="mb badge-m">M</span>.set_epoch()</a>+</div></div>+<div class="l2"><div class="l2-title" onclick="toggleL2(this)"><span>index_ds</span><span class="arr">▸</span></div><div class="l2-children">+<a class="l3-item" data-target="e531" onclick="showEntry('e531')"><span class="mb badge-c">C</span>IndexDSJsonlRankFull</a>+<a class="l3-item l3m" data-target="e532" onclick="showEntry('e532')"><span class="mb badge-m">M</span>.get_source_len()</a>+<a class="l3-item l3m" data-target="e533" onclick="showEntry('e533')"><span class="mb badge-m">M</span>.get_target_len()</a>+</div></div>+<div class="l2"><div class="l2-title" onclick="toggleL2(this)"><span>jsonl2scp</span><span class="arr">▸</span></div><div class="l2-children">+<a class="l3-item" data-target="e534" onclick="showEntry('e534')"><span class="mb badge-f">F</span>gen_scp_from_jsonl</a>+<a class="l3-item" data-target="e535" onclick="showEntry('e535')"><span class="mb badge-f">F</span>main_hydra</a>+</div></div>+<div class="l2"><div class="l2-title" onclick="toggleL2(this)"><span>preprocessor</span><span class="arr">▸</span></div><div class="l2-children">+<a class="l3-item" data-target="e536" onclick="showEntry('e536')"><span class="mb badge-c">C</span>SpeechPreprocessSpeedPerturb</a>+<a class="l3-item l3m" data-target="e537" onclick="showEntry('e537')"><span class="mb badge-m">M</span>.forward()</a>+<a class="l3-item" data-target="e538" onclick="showEntry('e538')"><span class="mb badge-c">C</span>TextPreprocessSegDict</a>+<a class="l3-item l3m" data-target="e539" onclick="showEntry('e539')"><span class="mb badge-m">M</span>.forward()</a>+</div></div>+<div class="l2"><div class="l2-title" onclick="toggleL2(this)"><span>samplers</span><span class="arr">▸</span></div><div class="l2-children">+<a class="l3-item" data-target="e540" onclick="showEntry('e540')"><span class="mb badge-f">F</span>CustomDistributedBatchSampler_fn</a>+<a class="l3-item" data-target="e541" onclick="showEntry('e541')"><span class="mb badge-c">C</span>CustomDistributedBatchSampler</a>+<a class="l3-item l3m" data-target="e542" onclick="showEntry('e542')"><span class="mb badge-m">M</span>.set_epoch()</a>+<a class="l3-item" data-target="e543" onclick="showEntry('e543')"><span class="mb badge-c">C</span>CustomDistributedBufferBatchSampler</a>+<a class="l3-item l3m" data-target="e544" onclick="showEntry('e544')"><span class="mb badge-m">M</span>.set_epoch()</a>+<a class="l3-item" data-target="e545" onclick="showEntry('e545')"><span class="mb badge-c">C</span>CustomDistributedDynamicBatchSampler</a>+<a class="l3-item l3m" data-target="e546" onclick="showEntry('e546')"><span class="mb badge-m">M</span>.set_epoch()</a>+<a class="l3-item" data-target="e547" onclick="showEntry('e547')"><span class="mb badge-c">C</span>CustomDistributedBufferDynamicBatchSampler</a>+<a class="l3-item l3m" data-target="e548" onclick="showEntry('e548')"><span class="mb badge-m">M</span>.set_epoch()</a>+<a class="l3-item" data-target="e549" onclick="showEntry('e549')"><span class="mb badge-c">C</span>DistributedSamplerWarp</a>+<a class="l3-item l3m" data-target="e550" onclick="showEntry('e550')"><span class="mb badge-m">M</span>.set_epoch()</a>+</div></div>+<div class="l2"><div class="l2-title" onclick="toggleL2(this)"><span>scp2jsonl</span><span class="arr">▸</span></div><div class="l2-children">+<a class="l3-item" data-target="e551" onclick="showEntry('e551')"><span class="mb badge-f">F</span>gen_jsonl_from_wav_text_list</a>+<a class="l3-item" data-target="e552" onclick="showEntry('e552')"><span class="mb badge-f">F</span>parse_context_length</a>+<a class="l3-item" data-target="e553" onclick="showEntry('e553')"><span class="mb badge-f">F</span>main_hydra</a>+</div></div>+<div class="l2"><div class="l2-title" onclick="toggleL2(this)"><span>scp2len</span><span class="arr">▸</span></div><div class="l2-children">+<a class="l3-item" data-target="e554" onclick="showEntry('e554')"><span class="mb badge-f">F</span>gen_jsonl_from_wav_text_list</a>+<a class="l3-item" data-target="e555" onclick="showEntry('e555')"><span class="mb badge-f">F</span>parse_context_length</a>+<a class="l3-item" data-target="e556" onclick="showEntry('e556')"><span class="mb badge-f">F</span>main_hydra</a>+</div></div>+<div class="l2"><div class="l2-title" onclick="toggleL2(this)"><span>sensevoice2jsonl</span><span class="arr">▸</span></div><div class="l2-children">+<a class="l3-item" data-target="e557" onclick="showEntry('e557')"><span class="mb badge-f">F</span>gen_jsonl_from_wav_text_list</a>+<a class="l3-item" data-target="e558" onclick="showEntry('e558')"><span class="mb badge-f">F</span>contains_punctuation</a>+<a class="l3-item" data-target="e559" onclick="showEntry('e559')"><span class="mb badge-f">F</span>parse_context_length</a>+<a class="l3-item" data-target="e560" onclick="showEntry('e560')"><span class="mb badge-f">F</span>main_hydra</a>+</div></div>+<div class="l2"><div class="l2-title" onclick="toggleL2(this)"><span>update_jsonl</span><span class="arr">▸</span></div><div class="l2-children">+<a class="l3-item" data-target="e561" onclick="showEntry('e561')"><span class="mb badge-f">F</span>gen_scp_from_jsonl</a>+<a class="l3-item" data-target="e562" onclick="showEntry('e562')"><span class="mb badge-f">F</span>update_data</a>+<a class="l3-item" data-target="e563" onclick="showEntry('e563')"><span class="mb badge-f">F</span>update_wav_len</a>+<a class="l3-item" data-target="e564" onclick="showEntry('e564')"><span class="mb badge-f">F</span>main_hydra</a>+</div></div>+<div class="l2"><div class="l2-title" onclick="toggleL2(this)"><span>dataloader_entry</span><span class="arr">▸</span></div><div class="l2-children">+<a class="l3-item" data-target="e565" onclick="showEntry('e565')"><span class="mb badge-f">F</span>DataloaderMapStyle</a>+<a class="l3-item" data-target="e566" onclick="showEntry('e566')"><span class="mb badge-c">C</span>DataloaderMapStyle</a>+<a class="l3-item l3m" data-target="e567" onclick="showEntry('e567')"><span class="mb badge-m">M</span>.build_iter()</a>+<a class="l3-item" data-target="e568" onclick="showEntry('e568')"><span class="mb badge-f">F</span>DataloaderIterable</a>+</div></div>+</div></div>+<div class="l1"><div class="l1-title" onclick="toggleL1(this)"><span>funasr.losses</span><span class="cnt">1</span><span class="arr">▸</span></div><div class="l1-children">+<div class="l2"><div class="l2-title" onclick="toggleL2(this)"><span>label_smoothing_loss</span><span class="arr">▸</span></div><div class="l2-children">+<a class="l3-item" data-target="e569" onclick="showEntry('e569')"><span class="mb badge-c">C</span>LabelSmoothingLoss</a>+<a class="l3-item l3m" data-target="e570" onclick="showEntry('e570')"><span class="mb badge-m">M</span>.forward()</a>+<a class="l3-item" data-target="e571" onclick="showEntry('e571')"><span class="mb badge-c">C</span>SequenceBinaryCrossEntropy</a>+<a class="l3-item l3m" data-target="e572" onclick="showEntry('e572')"><span class="mb badge-m">M</span>.forward()</a>+<a class="l3-item" data-target="e573" onclick="showEntry('e573')"><span class="mb badge-c">C</span>NllLoss</a>+<a class="l3-item l3m" data-target="e574" onclick="showEntry('e574')"><span class="mb badge-m">M</span>.forward()</a>+</div></div>+</div></div>+<div class="l1"><div class="l1-title" onclick="toggleL1(this)"><span>funasr.schedulers</span><span class="cnt">4</span><span class="arr">▸</span></div><div class="l1-children">+<div class="l2"><div class="l2-title" onclick="toggleL2(this)"><span>lambdalr_cus</span><span class="arr">▸</span></div><div class="l2-children">+<a class="l3-item" data-target="e575" onclick="showEntry('e575')"><span class="mb badge-c">C</span>CustomLambdaLR</a>+<a class="l3-item l3m" data-target="e576" onclick="showEntry('e576')"><span class="mb badge-m">M</span>.get_lr()</a>+</div></div>+<div class="l2"><div class="l2-title" onclick="toggleL2(this)"><span>noam_lr</span><span class="arr">▸</span></div><div class="l2-children">+<a class="l3-item" data-target="e577" onclick="showEntry('e577')"><span class="mb badge-c">C</span>NoamLR</a>+<a class="l3-item l3m" data-target="e578" onclick="showEntry('e578')"><span class="mb badge-m">M</span>.lr_for_WarmupLR()</a>+<a class="l3-item l3m" data-target="e579" onclick="showEntry('e579')"><span class="mb badge-m">M</span>.get_lr()</a>+</div></div>+<div class="l2"><div class="l2-title" onclick="toggleL2(this)"><span>tri_stage_scheduler</span><span class="arr">▸</span></div><div class="l2-children">+<a class="l3-item" data-target="e580" onclick="showEntry('e580')"><span class="mb badge-c">C</span>TriStageLR</a>+<a class="l3-item l3m" data-target="e581" onclick="showEntry('e581')"><span class="mb badge-m">M</span>.init_tri_stage_scheudler()</a>+<a class="l3-item l3m" data-target="e582" onclick="showEntry('e582')"><span class="mb badge-m">M</span>.step_update()</a>+<a class="l3-item l3m" data-target="e583" onclick="showEntry('e583')"><span class="mb badge-m">M</span>.set_optimizer_lr()</a>+<a class="l3-item l3m" data-target="e584" onclick="showEntry('e584')"><span class="mb badge-m">M</span>.get_lr()</a>+</div></div>+<div class="l2"><div class="l2-title" onclick="toggleL2(this)"><span>warmup_lr</span><span class="arr">▸</span></div><div class="l2-children">+<a class="l3-item" data-target="e585" onclick="showEntry('e585')"><span class="mb badge-c">C</span>WarmupLR</a>+<a class="l3-item l3m" data-target="e586" onclick="showEntry('e586')"><span class="mb badge-m">M</span>.get_lr()</a>+</div></div>+</div></div>++</div>+<div class="api-content">+<div class="api-welcome" id="api-welcome"><h2>API Reference</h2><p>250 entries · auto-generated from source<br>Click source code to expand. Links point to latest GitHub code.</p></div>+<div class="api-detail" id="e1"><span class="dtag badge-f">function</span><h2><code>is_npu_available()</code></h2><div class="dmod">funasr.auto.auto_model · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/auto/auto_model.py#L34" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>检查NPU是否可用。</p>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code>def is_npu_available():+ """检查NPU是否可用。"""+ try:+ import torch_npu++ return torch_npu.npu.is_available()+ except ImportError:+ return False+</code></pre></details></div>+<div class="api-detail" id="e2"><span class="dtag badge-f">function</span><h2><code>prepare_data_iterator(data_in, input_len, data_type, key)</code></h2><div class="dmod">funasr.auto.auto_model · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/auto/auto_model.py#L111" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p class="muted">No documentation yet.</p></div><details class="src-block"><summary>📄 Source code</summary><pre><code>def prepare_data_iterator(data_in, input_len=None, data_type=None, key=None):+ """ """+ data_list = []+ key_list = []+ filelist = [".scp", ".txt", ".json", ".jsonl", ".text"]++ chars = string.ascii_letters + string.digits+ if isinstance(data_in, str):+ if data_in.startswith("http://") or data_in.startswith("https://"): # url+ data_in = download_from_url(data_in)++ if isinstance(data_in, str) and os.path.exists(+ data_in+ ): # wav_path; filelist: wav.scp, file.jsonl;text.txt;+ _, file_extension = os.path.splitext(data_in)+ file_extension = file_extension.lower()+ if file_extension in filelist: # filelist: wav.scp, file.jsonl;text.txt;+ with open(data_in, encoding="utf-8") as fin:+ for line in fin:+ key = "rand_key_" + "".join(random.choice(chars) for _ in range(13))+ if data_in.endswith(".jsonl"): # file.jsonl: json.dumps({"source": data})+ lines = json.loads(line.strip())+ data = lines["source"]+ key = lines.get("key", key)+ else: # filelist, wav.scp, text.txt: id \t data or data+ lines = line.strip().split(maxsplit=1)+ data = lines[1] if len(lines) > 1 else lines[0]+ key = lines[0] if len(lines) > 1 else key++ data_list.append(data)+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/auto/auto_model.py#L111" target="_blank" class="src-more">View full source on GitHub →</a></details></div>+<div class="api-detail" id="e3"><span class="dtag badge-c">class</span><h2>AutoModel</h2><div class="dmod">funasr.auto.auto_model · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/auto/auto_model.py#L182" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p class="muted">No documentation yet.</p></div><details class="src-block"><summary>📄 Source code</summary><pre><code>class AutoModel:++ def __init__(self, **kwargs):+ """Initialize AutoModel with ASR model and optional sub-models.++ Args:+ model (str): Model name (hub alias or full ID) or local path.+ device (str): Device for inference. "cuda:0", "cpu", "mps", "npu:0".+ Falls back to CPU if specified device is unavailable.+ vad_model (str, optional): VAD model for long audio segmentation.+ Enables processing of any-length audio.+ vad_kwargs (dict, optional): VAD config, e.g. {"max_single_segment_time": 60000}.+ punc_model (str, optional): Punctuation restoration model.+ Not needed for Fun-ASR-Nano/SenseVoice/Qwen3-ASR (they output punctuation natively).+ spk_model (str, optional): Speaker model for diarization ("cam++" or full model ID).+ Requires vad_model. For Qwen3-ASR, also requires forced_aligner.+ spk_mode (str, optional): Speaker diarization mode. "punc_segment" (default) or "vad_segment".+ hub (str): Model hub. "ms" (ModelScope, default) or "hf" (HuggingFace).+ ncpu (int): CPU threads (default: 4).+ disable_update (bool): Skip version check on startup.+ disable_pbar (bool): Disable tqdm progress bars.+ **kwargs: Additional model-specific parameters (passed to config.yaml overrides).++ Examples:+ >>> model = AutoModel(model="paraformer-zh", vad_model="fsmn-vad", punc_model="ct-punc")+ >>> model = AutoModel(model="FunAudioLLM/Fun-ASR-Nano-2512", trust_remote_code=True,+ ... remote_code="./model.py", vad_model="fsmn-vad", spk_model="cam++", hub="hf")+ """+ try:+ from funasr.utils.version_checker import check_for_update+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/auto/auto_model.py#L182" target="_blank" class="src-more">View full source on GitHub →</a></details><h3 class="mt">Methods</h3><div class="mblk"><code>.build_model(**kwargs)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/auto/auto_model.py#L275" target="_blank" class="gh-link">L275</a><div class="mdoc"><p>Download model from hub, build all components, and load pretrained weights.</p>+<p>This method handles the full model construction pipeline:</p>+<p>1. Download model files from ModelScope/HuggingFace (if not local)</p>+<p>2. Parse config.yaml to determine model class, tokenizer, frontend</p>+<p>3. Instantiate tokenizer, frontend, and model via the registry</p>+<p>4. Load pretrained weights from model.pt</p>+<h4>Args:</h4>+<ul>+<li><code>**kwargs</code> — Must include 'model' (str). All other config.yaml fields can be overridden.</li>+</ul>+<h4>Returns:</h4>+<ul>+<li><code>tuple</code> — (model, kwargs) where model is the instantiated nn.Module and</li>+<li class="sub">kwargs contains the resolved configuration.</li>+</ul>+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def build_model(**kwargs):+ """Download model from hub, build all components, and load pretrained weights.++ This method handles the full model construction pipeline:+ 1. Download model files from ModelScope/HuggingFace (if not local)+ 2. Parse config.yaml to determine model class, tokenizer, frontend+ 3. Instantiate tokenizer, frontend, and model via the registry+ 4. Load pretrained weights from model.pt++ Args:+ **kwargs: Must include 'model' (str). All other config.yaml fields can be overridden.++ Returns:+ tuple: (model, kwargs) where model is the instantiated nn.Module and+ kwargs contains the resolved configuration.+ """+ assert "model" in kwargs+ if "model_conf" not in kwargs:+ logging.info("download models from model hub: {}".format(kwargs.get("hub", "ms")))+ kwargs = download_model(**kwargs)++ set_all_random_seed(kwargs.get("seed", 0))++ device = kwargs.get("device", "cuda")+ if (+ (device.startswith("cuda") and not torch.cuda.is_available())+ or (device.startswith("xpu") and not torch.xpu.is_available())+ or (device.startswith("mps") and not torch.backends.mps.is_available())+ or (device.startswith("npu") and not is_npu_available())+ or kwargs.get("ngpu", 1) == 0+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/auto/auto_model.py#L275" target="_blank" class="src-more">View full source →</a></details></div><div class="mblk"><code>.generate(input, input_len, progress_callback, **cfg)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/auto/auto_model.py#L436" target="_blank" class="gh-link">L436</a><div class="mdoc"><p>Run speech recognition on input audio.</p>+<p>This is the primary user-facing method. It automatically routes to:</p>+<ul>+<li>inference() if no vad_model is configured (single utterance)</li>+<li>inference_with_vad() if vad_model is configured (long audio with segmentation)</li>+</ul>+<h4>Args:</h4>+<ul>+<li><code>input</code> — Audio input. Accepts:</li>+<li>File path (str): "audio.wav", "audio.mp3"</li>+<li>URL (str): "https://..."</li>+<li>numpy array: raw audio samples (float32, 16kHz)</li>+<li>list: batch of file paths or arrays</li>+<li>bytes: raw audio bytes</li>+<li><code>input_len (tensor, optional)</code> — Length of each input sample.</li>+<li><code>progress_callback (callable, optional)</code> — fn(current, total) called during processing.</li>+<li><code>**cfg</code> — Runtime parameters:</li>+<li>cache (dict): State cache for streaming mode. Pass {} for first call.</li>+<li>hotword (str/list): Keywords to boost recognition accuracy.</li>+<li>language (str): Language hint ("auto", "zh", "en", "Chinese", etc.)</li>+<li>batch_size_s (int): Dynamic batch total duration in seconds.</li>+<li>is_final (bool): Last chunk flag for streaming mode.</li>+<li>return_spk_res (bool): Return speaker diarization results.</li>+<li>sentence_timestamp (bool): Return sentence-level timestamps.</li>+<li>use_itn (bool): Apply inverse text normalization (SenseVoice).</li>+</ul>+<h4>Returns:</h4>+<p>list[dict]: Results for each input sample. Common fields:</p>+<ul>+<li>"key" (str): Sample identifier</li>+<li>"text" (str): Recognized text</li>+<li>"timestamp" (list): [[start_ms, end_ms], ...] per character/word</li>+<li>"sentence_info" (list): [{text, start, end, spk, timestamp}, ...] when spk enabled</li>+</ul>+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def generate(self, input, input_len=None, progress_callback=None, **cfg):+ """Run speech recognition on input audio.++ This is the primary user-facing method. It automatically routes to:+ - inference() if no vad_model is configured (single utterance)+ - inference_with_vad() if vad_model is configured (long audio with segmentation)++ Args:+ input: Audio input. Accepts:+ - File path (str): "audio.wav", "audio.mp3"+ - URL (str): "https://..."+ - numpy array: raw audio samples (float32, 16kHz)+ - list: batch of file paths or arrays+ - bytes: raw audio bytes+ input_len (tensor, optional): Length of each input sample.+ progress_callback (callable, optional): fn(current, total) called during processing.+ **cfg: Runtime parameters:+ - cache (dict): State cache for streaming mode. Pass {} for first call.+ - hotword (str/list): Keywords to boost recognition accuracy.+ - language (str): Language hint ("auto", "zh", "en", "Chinese", etc.)+ - batch_size_s (int): Dynamic batch total duration in seconds.+ - is_final (bool): Last chunk flag for streaming mode.+ - return_spk_res (bool): Return speaker diarization results.+ - sentence_timestamp (bool): Return sentence-level timestamps.+ - use_itn (bool): Apply inverse text normalization (SenseVoice).++ Returns:+ list[dict]: Results for each input sample. Common fields:+ - "key" (str): Sample identifier+ - "text" (str): Recognized text+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/auto/auto_model.py#L436" target="_blank" class="src-more">View full source →</a></details></div><div class="mblk"><code>.inference(input, input_len, model, kwargs, key, progress_callback, **cfg)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/auto/auto_model.py#L490" target="_blank" class="gh-link">L490</a><div class="mdoc"><p>Run model inference on input data (internal method).</p>+<p>Handles batching, timing, and progress reporting. Called by generate()</p>+<p>and inference_with_vad(). Typically not called directly by users.</p>+<h4>Args:</h4>+<ul>+<li><code>input</code> — Audio data, file path, or text (for punc model).</li>+<li><code>input_len (tensor, optional)</code> — Input lengths for batch.</li>+<li><code>model (nn.Module, optional)</code> — Override model (used for VAD/PUNC/SPK sub-models).</li>+<li><code>kwargs (dict, optional)</code> — Override kwargs (used for sub-model configs).</li>+<li><code>key (list, optional)</code> — Sample identifiers.</li>+<li><code>progress_callback (callable, optional)</code> — Progress reporting function.</li>+<li><code>**cfg</code> — Additional config merged into kwargs.</li>+</ul>+<h4>Returns:</h4>+<p>list[dict]: Model inference results.</p>+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def inference(+ self,+ input,+ input_len=None,+ model=None,+ kwargs=None,+ key=None,+ progress_callback=None,+ **cfg,+ ):+ """Run model inference on input data (internal method).++ Handles batching, timing, and progress reporting. Called by generate()+ and inference_with_vad(). Typically not called directly by users.++ Args:+ input: Audio data, file path, or text (for punc model).+ input_len (tensor, optional): Input lengths for batch.+ model (nn.Module, optional): Override model (used for VAD/PUNC/SPK sub-models).+ kwargs (dict, optional): Override kwargs (used for sub-model configs).+ key (list, optional): Sample identifiers.+ progress_callback (callable, optional): Progress reporting function.+ **cfg: Additional config merged into kwargs.++ Returns:+ list[dict]: Model inference results.+ """+ if kwargs is None:+ self._reset_runtime_configs()+ kwargs = self.kwargs if kwargs is None else kwargs+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/auto/auto_model.py#L490" target="_blank" class="src-more">View full source →</a></details></div><div class="mblk"><code>.inference_with_vad(input, input_len, **cfg)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/auto/auto_model.py#L592" target="_blank" class="gh-link">L592</a><div class="mdoc"><p>Run ASR with VAD segmentation, punctuation, and optional speaker diarization.</p>+<p>Pipeline:</p>+<p>1. VAD: Segment audio into speech regions</p>+<p>2. ASR: Recognize each segment (sorted by length for efficient batching)</p>+<p>3. Timestamp merge: Combine per-segment timestamps with VAD offsets</p>+<p>4. Punctuation: Add punctuation to combined text (if punc_model configured)</p>+<p>5. Speaker diarization: Cluster speaker embeddings and assign labels (if spk_model configured)</p>+<h4>Args:</h4>+<ul>+<li><code>input</code> — Audio file path, URL, or numpy array.</li>+<li><code>input_len</code> — Not used (kept for interface consistency).</li>+<li><code>**cfg</code> — Runtime parameters (same as generate()).</li>+</ul>+<h4>Returns:</h4>+<p>list[dict]: Results with fields: key, text, timestamp, sentence_info, raw_text.</p>+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def inference_with_vad(self, input, input_len=None, **cfg):+ """Run ASR with VAD segmentation, punctuation, and optional speaker diarization.++ Pipeline:+ 1. VAD: Segment audio into speech regions+ 2. ASR: Recognize each segment (sorted by length for efficient batching)+ 3. Timestamp merge: Combine per-segment timestamps with VAD offsets+ 4. Punctuation: Add punctuation to combined text (if punc_model configured)+ 5. Speaker diarization: Cluster speaker embeddings and assign labels (if spk_model configured)++ Args:+ input: Audio file path, URL, or numpy array.+ input_len: Not used (kept for interface consistency).+ **cfg: Runtime parameters (same as generate()).++ Returns:+ list[dict]: Results with fields: key, text, timestamp, sentence_info, raw_text.+ """+ self._reset_runtime_configs()+ if self.spk_model is not None and "output_timestamp" not in cfg:+ cfg["output_timestamp"] = True+ cfg["return_time_stamps"] = True+ kwargs = self.kwargs+ # step.1: compute the vad model+ deep_update(self.vad_kwargs, cfg)+ beg_vad = time.time()+ res = self.inference(+ input, input_len=input_len, model=self.vad_model, kwargs=self.vad_kwargs, **cfg+ )+ end_vad = time.time()+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/auto/auto_model.py#L592" target="_blank" class="src-more">View full source →</a></details></div><div class="mblk"><code>.export(input, **cfg)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/auto/auto_model.py#L911" target="_blank" class="gh-link">L911</a><div class="mdoc"><p>Export model to ONNX format.</p>+<p>Creates a deep copy of the model to isolate ONNX operator monkey-patching,</p>+<p>then runs torch.onnx.export. The original model remains usable after export.</p>+<h4>Args:</h4>+<ul>+<li><code>input</code> — Sample input for tracing (auto-generated if None).</li>+<li><code>**cfg</code> — Export parameters:</li>+<li>type (str): Export format, "onnx" (default).</li>+<li>quantize (bool): Whether to quantize the model.</li>+<li>device (str): Device for export.</li>+</ul>+<h4>Returns:</h4>+<ul>+<li><code>str</code> — Path to the exported model directory.</li>+</ul>+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def export(self, input=None, **cfg):+ """Export model to ONNX format.++ Creates a deep copy of the model to isolate ONNX operator monkey-patching,+ then runs torch.onnx.export. The original model remains usable after export.++ Args:+ input: Sample input for tracing (auto-generated if None).+ **cfg: Export parameters:+ - type (str): Export format, "onnx" (default).+ - quantize (bool): Whether to quantize the model.+ - device (str): Device for export.++ Returns:+ str: Path to the exported model directory.+ """+ """++ :param input:+ :param type:+ :param quantize:+ :param fallback_num:+ :param calib_num:+ :param opset_version:+ :param cfg:+ :return:+ """++ device = cfg.get("device", "cpu")++</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/auto/auto_model.py#L911" target="_blank" class="src-more">View full source →</a></details></div></div>+<div class="api-detail" id="e4"><span class="dtag badge-m">method</span><h2><code>AutoModel.build_model(**kwargs)</code></h2><div class="dmod">funasr.auto.auto_model.AutoModel · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/auto/auto_model.py#L275" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Download model from hub, build all components, and load pretrained weights.</p>+<p>This method handles the full model construction pipeline:</p>+<p>1. Download model files from ModelScope/HuggingFace (if not local)</p>+<p>2. Parse config.yaml to determine model class, tokenizer, frontend</p>+<p>3. Instantiate tokenizer, frontend, and model via the registry</p>+<p>4. Load pretrained weights from model.pt</p>+<h4>Args:</h4>+<ul>+<li><code>**kwargs</code> — Must include 'model' (str). All other config.yaml fields can be overridden.</li>+</ul>+<h4>Returns:</h4>+<ul>+<li><code>tuple</code> — (model, kwargs) where model is the instantiated nn.Module and</li>+<li class="sub">kwargs contains the resolved configuration.</li>+</ul>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def build_model(**kwargs):+ """Download model from hub, build all components, and load pretrained weights.++ This method handles the full model construction pipeline:+ 1. Download model files from ModelScope/HuggingFace (if not local)+ 2. Parse config.yaml to determine model class, tokenizer, frontend+ 3. Instantiate tokenizer, frontend, and model via the registry+ 4. Load pretrained weights from model.pt++ Args:+ **kwargs: Must include 'model' (str). All other config.yaml fields can be overridden.++ Returns:+ tuple: (model, kwargs) where model is the instantiated nn.Module and+ kwargs contains the resolved configuration.+ """+ assert "model" in kwargs+ if "model_conf" not in kwargs:+ logging.info("download models from model hub: {}".format(kwargs.get("hub", "ms")))+ kwargs = download_model(**kwargs)++ set_all_random_seed(kwargs.get("seed", 0))++ device = kwargs.get("device", "cuda")+ if (+ (device.startswith("cuda") and not torch.cuda.is_available())+ or (device.startswith("xpu") and not torch.xpu.is_available())+ or (device.startswith("mps") and not torch.backends.mps.is_available())+ or (device.startswith("npu") and not is_npu_available())+ or kwargs.get("ngpu", 1) == 0+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/auto/auto_model.py#L275" target="_blank" class="src-more">View full source on GitHub →</a></details></div>+<div class="api-detail" id="e5"><span class="dtag badge-m">method</span><h2><code>AutoModel.generate(input, input_len, progress_callback, **cfg)</code></h2><div class="dmod">funasr.auto.auto_model.AutoModel · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/auto/auto_model.py#L436" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Run speech recognition on input audio.</p>+<p>This is the primary user-facing method. It automatically routes to:</p>+<ul>+<li>inference() if no vad_model is configured (single utterance)</li>+<li>inference_with_vad() if vad_model is configured (long audio with segmentation)</li>+</ul>+<h4>Args:</h4>+<ul>+<li><code>input</code> — Audio input. Accepts:</li>+<li>File path (str): "audio.wav", "audio.mp3"</li>+<li>URL (str): "https://..."</li>+<li>numpy array: raw audio samples (float32, 16kHz)</li>+<li>list: batch of file paths or arrays</li>+<li>bytes: raw audio bytes</li>+<li><code>input_len (tensor, optional)</code> — Length of each input sample.</li>+<li><code>progress_callback (callable, optional)</code> — fn(current, total) called during processing.</li>+<li><code>**cfg</code> — Runtime parameters:</li>+<li>cache (dict): State cache for streaming mode. Pass {} for first call.</li>+<li>hotword (str/list): Keywords to boost recognition accuracy.</li>+<li>language (str): Language hint ("auto", "zh", "en", "Chinese", etc.)</li>+<li>batch_size_s (int): Dynamic batch total duration in seconds.</li>+<li>is_final (bool): Last chunk flag for streaming mode.</li>+<li>return_spk_res (bool): Return speaker diarization results.</li>+<li>sentence_timestamp (bool): Return sentence-level timestamps.</li>+<li>use_itn (bool): Apply inverse text normalization (SenseVoice).</li>+</ul>+<h4>Returns:</h4>+<p>list[dict]: Results for each input sample. Common fields:</p>+<ul>+<li>"key" (str): Sample identifier</li>+<li>"text" (str): Recognized text</li>+<li>"timestamp" (list): [[start_ms, end_ms], ...] per character/word</li>+<li>"sentence_info" (list): [{text, start, end, spk, timestamp}, ...] when spk enabled</li>+</ul>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def generate(self, input, input_len=None, progress_callback=None, **cfg):+ """Run speech recognition on input audio.++ This is the primary user-facing method. It automatically routes to:+ - inference() if no vad_model is configured (single utterance)+ - inference_with_vad() if vad_model is configured (long audio with segmentation)++ Args:+ input: Audio input. Accepts:+ - File path (str): "audio.wav", "audio.mp3"+ - URL (str): "https://..."+ - numpy array: raw audio samples (float32, 16kHz)+ - list: batch of file paths or arrays+ - bytes: raw audio bytes+ input_len (tensor, optional): Length of each input sample.+ progress_callback (callable, optional): fn(current, total) called during processing.+ **cfg: Runtime parameters:+ - cache (dict): State cache for streaming mode. Pass {} for first call.+ - hotword (str/list): Keywords to boost recognition accuracy.+ - language (str): Language hint ("auto", "zh", "en", "Chinese", etc.)+ - batch_size_s (int): Dynamic batch total duration in seconds.+ - is_final (bool): Last chunk flag for streaming mode.+ - return_spk_res (bool): Return speaker diarization results.+ - sentence_timestamp (bool): Return sentence-level timestamps.+ - use_itn (bool): Apply inverse text normalization (SenseVoice).++ Returns:+ list[dict]: Results for each input sample. Common fields:+ - "key" (str): Sample identifier+ - "text" (str): Recognized text+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/auto/auto_model.py#L436" target="_blank" class="src-more">View full source on GitHub →</a></details></div>+<div class="api-detail" id="e6"><span class="dtag badge-m">method</span><h2><code>AutoModel.inference(input, input_len, model, kwargs, key, progress_callback, **cfg)</code></h2><div class="dmod">funasr.auto.auto_model.AutoModel · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/auto/auto_model.py#L490" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Run model inference on input data (internal method).</p>+<p>Handles batching, timing, and progress reporting. Called by generate()</p>+<p>and inference_with_vad(). Typically not called directly by users.</p>+<h4>Args:</h4>+<ul>+<li><code>input</code> — Audio data, file path, or text (for punc model).</li>+<li><code>input_len (tensor, optional)</code> — Input lengths for batch.</li>+<li><code>model (nn.Module, optional)</code> — Override model (used for VAD/PUNC/SPK sub-models).</li>+<li><code>kwargs (dict, optional)</code> — Override kwargs (used for sub-model configs).</li>+<li><code>key (list, optional)</code> — Sample identifiers.</li>+<li><code>progress_callback (callable, optional)</code> — Progress reporting function.</li>+<li><code>**cfg</code> — Additional config merged into kwargs.</li>+</ul>+<h4>Returns:</h4>+<p>list[dict]: Model inference results.</p>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def inference(+ self,+ input,+ input_len=None,+ model=None,+ kwargs=None,+ key=None,+ progress_callback=None,+ **cfg,+ ):+ """Run model inference on input data (internal method).++ Handles batching, timing, and progress reporting. Called by generate()+ and inference_with_vad(). Typically not called directly by users.++ Args:+ input: Audio data, file path, or text (for punc model).+ input_len (tensor, optional): Input lengths for batch.+ model (nn.Module, optional): Override model (used for VAD/PUNC/SPK sub-models).+ kwargs (dict, optional): Override kwargs (used for sub-model configs).+ key (list, optional): Sample identifiers.+ progress_callback (callable, optional): Progress reporting function.+ **cfg: Additional config merged into kwargs.++ Returns:+ list[dict]: Model inference results.+ """+ if kwargs is None:+ self._reset_runtime_configs()+ kwargs = self.kwargs if kwargs is None else kwargs+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/auto/auto_model.py#L490" target="_blank" class="src-more">View full source on GitHub →</a></details></div>+<div class="api-detail" id="e7"><span class="dtag badge-m">method</span><h2><code>AutoModel.inference_with_vad(input, input_len, **cfg)</code></h2><div class="dmod">funasr.auto.auto_model.AutoModel · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/auto/auto_model.py#L592" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Run ASR with VAD segmentation, punctuation, and optional speaker diarization.</p>+<p>Pipeline:</p>+<p>1. VAD: Segment audio into speech regions</p>+<p>2. ASR: Recognize each segment (sorted by length for efficient batching)</p>+<p>3. Timestamp merge: Combine per-segment timestamps with VAD offsets</p>+<p>4. Punctuation: Add punctuation to combined text (if punc_model configured)</p>+<p>5. Speaker diarization: Cluster speaker embeddings and assign labels (if spk_model configured)</p>+<h4>Args:</h4>+<ul>+<li><code>input</code> — Audio file path, URL, or numpy array.</li>+<li><code>input_len</code> — Not used (kept for interface consistency).</li>+<li><code>**cfg</code> — Runtime parameters (same as generate()).</li>+</ul>+<h4>Returns:</h4>+<p>list[dict]: Results with fields: key, text, timestamp, sentence_info, raw_text.</p>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def inference_with_vad(self, input, input_len=None, **cfg):+ """Run ASR with VAD segmentation, punctuation, and optional speaker diarization.++ Pipeline:+ 1. VAD: Segment audio into speech regions+ 2. ASR: Recognize each segment (sorted by length for efficient batching)+ 3. Timestamp merge: Combine per-segment timestamps with VAD offsets+ 4. Punctuation: Add punctuation to combined text (if punc_model configured)+ 5. Speaker diarization: Cluster speaker embeddings and assign labels (if spk_model configured)++ Args:+ input: Audio file path, URL, or numpy array.+ input_len: Not used (kept for interface consistency).+ **cfg: Runtime parameters (same as generate()).++ Returns:+ list[dict]: Results with fields: key, text, timestamp, sentence_info, raw_text.+ """+ self._reset_runtime_configs()+ if self.spk_model is not None and "output_timestamp" not in cfg:+ cfg["output_timestamp"] = True+ cfg["return_time_stamps"] = True+ kwargs = self.kwargs+ # step.1: compute the vad model+ deep_update(self.vad_kwargs, cfg)+ beg_vad = time.time()+ res = self.inference(+ input, input_len=input_len, model=self.vad_model, kwargs=self.vad_kwargs, **cfg+ )+ end_vad = time.time()+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/auto/auto_model.py#L592" target="_blank" class="src-more">View full source on GitHub →</a></details></div>+<div class="api-detail" id="e8"><span class="dtag badge-m">method</span><h2><code>AutoModel.export(input, **cfg)</code></h2><div class="dmod">funasr.auto.auto_model.AutoModel · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/auto/auto_model.py#L911" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Export model to ONNX format.</p>+<p>Creates a deep copy of the model to isolate ONNX operator monkey-patching,</p>+<p>then runs torch.onnx.export. The original model remains usable after export.</p>+<h4>Args:</h4>+<ul>+<li><code>input</code> — Sample input for tracing (auto-generated if None).</li>+<li><code>**cfg</code> — Export parameters:</li>+<li>type (str): Export format, "onnx" (default).</li>+<li>quantize (bool): Whether to quantize the model.</li>+<li>device (str): Device for export.</li>+</ul>+<h4>Returns:</h4>+<ul>+<li><code>str</code> — Path to the exported model directory.</li>+</ul>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def export(self, input=None, **cfg):+ """Export model to ONNX format.++ Creates a deep copy of the model to isolate ONNX operator monkey-patching,+ then runs torch.onnx.export. The original model remains usable after export.++ Args:+ input: Sample input for tracing (auto-generated if None).+ **cfg: Export parameters:+ - type (str): Export format, "onnx" (default).+ - quantize (bool): Whether to quantize the model.+ - device (str): Device for export.++ Returns:+ str: Path to the exported model directory.+ """+ """++ :param input:+ :param type:+ :param quantize:+ :param fallback_num:+ :param calib_num:+ :param opset_version:+ :param cfg:+ :return:+ """++ device = cfg.get("device", "cpu")++</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/auto/auto_model.py#L911" target="_blank" class="src-more">View full source on GitHub →</a></details></div>+<div class="api-detail" id="e9"><span class="dtag badge-c">class</span><h2>RegisterTables</h2><div class="dmod">funasr.register.register · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/register.py#L8" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Registry system for classes.</p>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code>class RegisterTables:+ """Registry system for classes."""++ model_classes = {}+ frontend_classes = {}+ specaug_classes = {}+ normalize_classes = {}+ encoder_classes = {}+ decoder_classes = {}+ joint_network_classes = {}+ predictor_classes = {}+ stride_conv_classes = {}+ tokenizer_classes = {}+ dataloader_classes = {}+ batch_sampler_classes = {}+ dataset_classes = {}+ index_ds_classes = {}++ def print(self, key: str = None) -> None:+ """Print registered classes."""+ print("\ntables: \n")+ fields = vars(self)+ headers = ["register name", "class name", "class location"]+ for classes_key, classes_dict in fields.items():+ if classes_key.endswith("_meta") and (key is None or key in classes_key):+ print(f"----------- ** {classes_key.replace('_meta', '')} ** --------------")+ metas = []+ for register_key, meta in classes_dict.items():+ metas.append(meta)+ metas.sort(key=lambda x: x[0])+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/register.py#L8" target="_blank" class="src-more">View full source on GitHub →</a></details><h3 class="mt">Methods</h3><div class="mblk"><code>.print(key)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/register.py#L26" target="_blank" class="gh-link">L26</a><div class="mdoc"><p>Print registered classes.</p>+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def print(self, key: str = None) -> None:+ """Print registered classes."""+ print("\ntables: \n")+ fields = vars(self)+ headers = ["register name", "class name", "class location"]+ for classes_key, classes_dict in fields.items():+ if classes_key.endswith("_meta") and (key is None or key in classes_key):+ print(f"----------- ** {classes_key.replace('_meta', '')} ** --------------")+ metas = []+ for register_key, meta in classes_dict.items():+ metas.append(meta)+ metas.sort(key=lambda x: x[0])+ data = [headers] + metas+ col_widths = [max(len(str(item)) for item in col) for col in zip(*data)]++ for row in data:+ print(+ "| "+ + " | ".join(str(item).ljust(width) for item, width in zip(row, col_widths))+ + " |"+ )+ print("\n")+</code></pre></details></div><div class="mblk"><code>.register(register_tables_key, key)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/register.py#L49" target="_blank" class="gh-link">L49</a><div class="mdoc"><p>Decorator to register a class.</p>+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def register(self, register_tables_key: str, key: str = None) -> callable:+ """Decorator to register a class."""++ def decorator(target_class):+ """Decorator.++ Args:+ target_class: TODO.+ """+ if not hasattr(self, register_tables_key):+ setattr(self, register_tables_key, {})+ logging.debug(f"New registry table added: {register_tables_key}")++ registry = getattr(self, register_tables_key)+ registry_key = key if key is not None else target_class.__name__++ if registry_key in registry:+ logging.debug(+ f"Key {registry_key} already exists in {register_tables_key}, re-register"+ )++ registry[registry_key] = target_class++ register_tables_key_meta = register_tables_key + "_meta"+ if not hasattr(self, register_tables_key_meta):+ setattr(self, register_tables_key_meta, {})+ registry_meta = getattr(self, register_tables_key_meta)++ class_file = inspect.getfile(target_class)+ class_line = inspect.getsourcelines(target_class)[1]+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/register.py#L49" target="_blank" class="src-more">View full source →</a></details></div></div>+<div class="api-detail" id="e10"><span class="dtag badge-m">method</span><h2><code>RegisterTables.print(key)</code></h2><div class="dmod">funasr.register.register.RegisterTables · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/register.py#L26" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Print registered classes.</p>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def print(self, key: str = None) -> None:+ """Print registered classes."""+ print("\ntables: \n")+ fields = vars(self)+ headers = ["register name", "class name", "class location"]+ for classes_key, classes_dict in fields.items():+ if classes_key.endswith("_meta") and (key is None or key in classes_key):+ print(f"----------- ** {classes_key.replace('_meta', '')} ** --------------")+ metas = []+ for register_key, meta in classes_dict.items():+ metas.append(meta)+ metas.sort(key=lambda x: x[0])+ data = [headers] + metas+ col_widths = [max(len(str(item)) for item in col) for col in zip(*data)]++ for row in data:+ print(+ "| "+ + " | ".join(str(item).ljust(width) for item, width in zip(row, col_widths))+ + " |"+ )+ print("\n")+</code></pre></details></div>+<div class="api-detail" id="e11"><span class="dtag badge-m">method</span><h2><code>RegisterTables.register(register_tables_key, key)</code></h2><div class="dmod">funasr.register.register.RegisterTables · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/register.py#L49" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Decorator to register a class.</p>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def register(self, register_tables_key: str, key: str = None) -> callable:+ """Decorator to register a class."""++ def decorator(target_class):+ """Decorator.++ Args:+ target_class: TODO.+ """+ if not hasattr(self, register_tables_key):+ setattr(self, register_tables_key, {})+ logging.debug(f"New registry table added: {register_tables_key}")++ registry = getattr(self, register_tables_key)+ registry_key = key if key is not None else target_class.__name__++ if registry_key in registry:+ logging.debug(+ f"Key {registry_key} already exists in {register_tables_key}, re-register"+ )++ registry[registry_key] = target_class++ register_tables_key_meta = register_tables_key + "_meta"+ if not hasattr(self, register_tables_key_meta):+ setattr(self, register_tables_key_meta, {})+ registry_meta = getattr(self, register_tables_key_meta)++ class_file = inspect.getfile(target_class)+ class_line = inspect.getsourcelines(target_class)[1]+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/register.py#L49" target="_blank" class="src-more">View full source on GitHub →</a></details></div>+<div class="api-detail" id="e12"><span class="dtag badge-c">class</span><h2>BAT</h2><div class="dmod">funasr.models.bat · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/bat/model.py#L41" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><ul>+<li><code>BAT (Boundary-Aware Transducer)</code> — Low-latency RNN-T model with boundary detection.</li>+</ul>+<p>Inherits from Transducer. Designed for streaming ASR with reduced latency</p>+<p>by predicting token boundaries explicitly.</p>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code>class BAT(Transducer):+ """BAT (Boundary-Aware Transducer): Low-latency RNN-T model with boundary detection.++ Inherits from Transducer. Designed for streaming ASR with reduced latency+ by predicting token boundaries explicitly.+ """++ pass+</code></pre></details></div>+<div class="api-detail" id="e13"><span class="dtag badge-c">class</span><h2>BiCifParaformer</h2><div class="dmod">funasr.models.bicif_paraformer · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/bicif_paraformer/model.py#L43" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><ul>+<li><code>BiCifParaformer</code> — Paraformer with Bidirectional CIF for Timestamp Prediction.</li>+</ul>+<p>Extends Paraformer with a second CIF predictor that provides accurate</p>+<ul>+<li><code>character</code> — level timestamp prediction alongside ASR. Uses bidirectional</li>+</ul>+<p>information flow for better alignment between audio frames and text tokens.</p>+<p>Reference:</p>+<ul>+<li>FunASR: A Fundamental End-to-End Speech Recognition Toolkit (https://arxiv.org/abs/2305.11013)</li>+<li>Achieving timestamp prediction while recognizing with non-autoregressive end-to-end ASR model</li>+</ul>+<p>(https://arxiv.org/abs/2301.12343)</p>+<h4>Output:</h4>+<p>{"key": str, "text": str, "timestamp": [[start_ms, end_ms], ...]}</p>+<ul>+<li><code>Author</code> — Speech Lab of DAMO Academy, Alibaba Group</li>+</ul>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code>class BiCifParaformer(Paraformer):+ """BiCifParaformer: Paraformer with Bidirectional CIF for Timestamp Prediction.++ Extends Paraformer with a second CIF predictor that provides accurate+ character-level timestamp prediction alongside ASR. Uses bidirectional+ information flow for better alignment between audio frames and text tokens.++ Reference:+ - FunASR: A Fundamental End-to-End Speech Recognition Toolkit (https://arxiv.org/abs/2305.11013)+ - Achieving timestamp prediction while recognizing with non-autoregressive end-to-end ASR model+ (https://arxiv.org/abs/2301.12343)++ Output:+ {"key": str, "text": str, "timestamp": [[start_ms, end_ms], ...]}++ Author: Speech Lab of DAMO Academy, Alibaba Group+ """++ def __init__(+ self,+ *args,+ **kwargs,+ ):+ """Initialize BiCifParaformer.++ Args:+ *args: Variable positional arguments.+ **kwargs: Additional keyword arguments.+ """+ super().__init__(*args, **kwargs)+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/bicif_paraformer/model.py#L43" target="_blank" class="src-more">View full source on GitHub →</a></details><h3 class="mt">Methods</h3><div class="mblk"><code>.calc_predictor(encoder_out, encoder_out_lens)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/bicif_paraformer/model.py#L162" target="_blank" class="gh-link">L162</a><div class="mdoc"><p>Calc predictor.</p>+<h4>Args:</h4>+<ul>+<li><code>encoder_out</code> — Encoder output tensor.</li>+<li><code>encoder_out_lens</code> — Encoder output lengths.</li>+</ul>+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def calc_predictor(self, encoder_out, encoder_out_lens):+ """Calc predictor.++ Args:+ encoder_out: Encoder output tensor.+ encoder_out_lens: Encoder output lengths.+ """+ encoder_out_mask = (+ ~make_pad_mask(encoder_out_lens, maxlen=encoder_out.size(1))[:, None, :]+ ).to(encoder_out.device)+ pre_acoustic_embeds, pre_token_length, alphas, pre_peak_index, pre_token_length2 = (+ self.predictor(encoder_out, None, encoder_out_mask, ignore_id=self.ignore_id)+ )+ return pre_acoustic_embeds, pre_token_length, alphas, pre_peak_index+</code></pre></details></div><div class="mblk"><code>.calc_predictor_timestamp(encoder_out, encoder_out_lens, token_num)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/bicif_paraformer/model.py#L177" target="_blank" class="gh-link">L177</a><div class="mdoc"><p>Calc predictor timestamp.</p>+<h4>Args:</h4>+<ul>+<li><code>encoder_out</code> — Encoder output tensor.</li>+<li><code>encoder_out_lens</code> — Encoder output lengths.</li>+<li><code>token_num</code> — TODO.</li>+</ul>+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def calc_predictor_timestamp(self, encoder_out, encoder_out_lens, token_num):+ """Calc predictor timestamp.++ Args:+ encoder_out: Encoder output tensor.+ encoder_out_lens: Encoder output lengths.+ token_num: TODO.+ """+ encoder_out_mask = (+ ~make_pad_mask(encoder_out_lens, maxlen=encoder_out.size(1))[:, None, :]+ ).to(encoder_out.device)+ ds_alphas, ds_cif_peak, us_alphas, us_peaks = self.predictor.get_upsample_timestamp(+ encoder_out, encoder_out_mask, token_num+ )+ return ds_alphas, ds_cif_peak, us_alphas, us_peaks+</code></pre></details></div><div class="mblk"><code>.forward(speech, speech_lengths, text, text_lengths, **kwargs)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/bicif_paraformer/model.py#L193" target="_blank" class="gh-link">L193</a><div class="mdoc"><p>Frontend + Encoder + Decoder + Calc loss</p>+<h4>Args:</h4>+<ul>+<li><code>speech</code> — (Batch, Length, ...)</li>+<li><code>speech_lengths</code> — (Batch, )</li>+<li><code>text</code> — (Batch, Length)</li>+<li><code>text_lengths</code> — (Batch,)</li>+</ul>+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def forward(+ self,+ speech: torch.Tensor,+ speech_lengths: torch.Tensor,+ text: torch.Tensor,+ text_lengths: torch.Tensor,+ **kwargs,+ ) -> Tuple[torch.Tensor, Dict[str, torch.Tensor], torch.Tensor]:+ """Frontend + Encoder + Decoder + Calc loss+ Args:+ speech: (Batch, Length, ...)+ speech_lengths: (Batch, )+ text: (Batch, Length)+ text_lengths: (Batch,)+ """+ if len(text_lengths.size()) > 1:+ text_lengths = text_lengths[:, 0]+ if len(speech_lengths.size()) > 1:+ speech_lengths = speech_lengths[:, 0]++ batch_size = speech.shape[0]++ # Encoder+ encoder_out, encoder_out_lens = self.encode(speech, speech_lengths)++ loss_ctc, cer_ctc = None, None+ loss_pre = None+ stats = dict()++ # decoder: CTC branch+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/bicif_paraformer/model.py#L193" target="_blank" class="src-more">View full source →</a></details></div><div class="mblk"><code>.inference(data_in, data_lengths, key, tokenizer, frontend, **kwargs)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/bicif_paraformer/model.py#L271" target="_blank" class="gh-link">L271</a><div class="mdoc"><p>Run inference on input data.</p>+<h4>Args:</h4>+<ul>+<li><code>data_in</code> — Input data (audio samples, file paths, or text).</li>+<li><code>data_lengths</code> — Lengths of each input sample in the batch.</li>+<li><code>key</code> — Sample identifiers.</li>+<li><code>tokenizer</code> — Tokenizer instance for text encoding/decoding.</li>+<li><code>frontend</code> — Audio frontend for feature extraction.</li>+<li><code>**kwargs</code> — Additional keyword arguments.</li>+</ul>+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def inference(+ self,+ data_in,+ data_lengths=None,+ key: list = None,+ tokenizer=None,+ frontend=None,+ **kwargs,+ ):++ # init beamsearch+ """Run inference on input data.++ Args:+ data_in: Input data (audio samples, file paths, or text).+ data_lengths: Lengths of each input sample in the batch.+ key: Sample identifiers.+ tokenizer: Tokenizer instance for text encoding/decoding.+ frontend: Audio frontend for feature extraction.+ **kwargs: Additional keyword arguments.+ """+ is_use_ctc = kwargs.get("decoding_ctc_weight", 0.0) > 0.00001 and self.ctc != None+ is_use_lm = (+ kwargs.get("lm_weight", 0.0) > 0.00001 and kwargs.get("lm_file", None) is not None+ )+ if self.beam_search is None and (is_use_lm or is_use_ctc):+ logging.info("enable beam_search")+ self.init_beam_search(**kwargs)+ self.nbest = kwargs.get("nbest", 1)++</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/bicif_paraformer/model.py#L271" target="_blank" class="src-more">View full source →</a></details></div><div class="mblk"><code>.export(**kwargs)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/bicif_paraformer/model.py#L430" target="_blank" class="gh-link">L430</a><div class="mdoc"><p>Export.</p>+<h4>Args:</h4>+<ul>+<li><code>**kwargs</code> — Additional keyword arguments.</li>+</ul>+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def export(self, **kwargs):+ """Export.++ Args:+ **kwargs: Additional keyword arguments.+ """+ from .export_meta import export_rebuild_model++ if "max_seq_len" not in kwargs:+ kwargs["max_seq_len"] = 512+ models = export_rebuild_model(model=self, **kwargs)+ return models+</code></pre></details></div></div>+<div class="api-detail" id="e14"><span class="dtag badge-m">method</span><h2><code>BiCifParaformer.calc_predictor(encoder_out, encoder_out_lens)</code></h2><div class="dmod">funasr.models.bicif_paraformer.BiCifParaformer · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/bicif_paraformer/model.py#L162" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Calc predictor.</p>+<h4>Args:</h4>+<ul>+<li><code>encoder_out</code> — Encoder output tensor.</li>+<li><code>encoder_out_lens</code> — Encoder output lengths.</li>+</ul>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def calc_predictor(self, encoder_out, encoder_out_lens):+ """Calc predictor.++ Args:+ encoder_out: Encoder output tensor.+ encoder_out_lens: Encoder output lengths.+ """+ encoder_out_mask = (+ ~make_pad_mask(encoder_out_lens, maxlen=encoder_out.size(1))[:, None, :]+ ).to(encoder_out.device)+ pre_acoustic_embeds, pre_token_length, alphas, pre_peak_index, pre_token_length2 = (+ self.predictor(encoder_out, None, encoder_out_mask, ignore_id=self.ignore_id)+ )+ return pre_acoustic_embeds, pre_token_length, alphas, pre_peak_index+</code></pre></details></div>+<div class="api-detail" id="e15"><span class="dtag badge-m">method</span><h2><code>BiCifParaformer.calc_predictor_timestamp(encoder_out, encoder_out_lens, token_num)</code></h2><div class="dmod">funasr.models.bicif_paraformer.BiCifParaformer · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/bicif_paraformer/model.py#L177" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Calc predictor timestamp.</p>+<h4>Args:</h4>+<ul>+<li><code>encoder_out</code> — Encoder output tensor.</li>+<li><code>encoder_out_lens</code> — Encoder output lengths.</li>+<li><code>token_num</code> — TODO.</li>+</ul>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def calc_predictor_timestamp(self, encoder_out, encoder_out_lens, token_num):+ """Calc predictor timestamp.++ Args:+ encoder_out: Encoder output tensor.+ encoder_out_lens: Encoder output lengths.+ token_num: TODO.+ """+ encoder_out_mask = (+ ~make_pad_mask(encoder_out_lens, maxlen=encoder_out.size(1))[:, None, :]+ ).to(encoder_out.device)+ ds_alphas, ds_cif_peak, us_alphas, us_peaks = self.predictor.get_upsample_timestamp(+ encoder_out, encoder_out_mask, token_num+ )+ return ds_alphas, ds_cif_peak, us_alphas, us_peaks+</code></pre></details></div>+<div class="api-detail" id="e16"><span class="dtag badge-m">method</span><h2><code>BiCifParaformer.forward(speech, speech_lengths, text, text_lengths, **kwargs)</code></h2><div class="dmod">funasr.models.bicif_paraformer.BiCifParaformer · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/bicif_paraformer/model.py#L193" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Frontend + Encoder + Decoder + Calc loss</p>+<h4>Args:</h4>+<ul>+<li><code>speech</code> — (Batch, Length, ...)</li>+<li><code>speech_lengths</code> — (Batch, )</li>+<li><code>text</code> — (Batch, Length)</li>+<li><code>text_lengths</code> — (Batch,)</li>+</ul>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def forward(+ self,+ speech: torch.Tensor,+ speech_lengths: torch.Tensor,+ text: torch.Tensor,+ text_lengths: torch.Tensor,+ **kwargs,+ ) -> Tuple[torch.Tensor, Dict[str, torch.Tensor], torch.Tensor]:+ """Frontend + Encoder + Decoder + Calc loss+ Args:+ speech: (Batch, Length, ...)+ speech_lengths: (Batch, )+ text: (Batch, Length)+ text_lengths: (Batch,)+ """+ if len(text_lengths.size()) > 1:+ text_lengths = text_lengths[:, 0]+ if len(speech_lengths.size()) > 1:+ speech_lengths = speech_lengths[:, 0]++ batch_size = speech.shape[0]++ # Encoder+ encoder_out, encoder_out_lens = self.encode(speech, speech_lengths)++ loss_ctc, cer_ctc = None, None+ loss_pre = None+ stats = dict()++ # decoder: CTC branch+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/bicif_paraformer/model.py#L193" target="_blank" class="src-more">View full source on GitHub →</a></details></div>+<div class="api-detail" id="e17"><span class="dtag badge-m">method</span><h2><code>BiCifParaformer.inference(data_in, data_lengths, key, tokenizer, frontend, **kwargs)</code></h2><div class="dmod">funasr.models.bicif_paraformer.BiCifParaformer · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/bicif_paraformer/model.py#L271" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Run inference on input data.</p>+<h4>Args:</h4>+<ul>+<li><code>data_in</code> — Input data (audio samples, file paths, or text).</li>+<li><code>data_lengths</code> — Lengths of each input sample in the batch.</li>+<li><code>key</code> — Sample identifiers.</li>+<li><code>tokenizer</code> — Tokenizer instance for text encoding/decoding.</li>+<li><code>frontend</code> — Audio frontend for feature extraction.</li>+<li><code>**kwargs</code> — Additional keyword arguments.</li>+</ul>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def inference(+ self,+ data_in,+ data_lengths=None,+ key: list = None,+ tokenizer=None,+ frontend=None,+ **kwargs,+ ):++ # init beamsearch+ """Run inference on input data.++ Args:+ data_in: Input data (audio samples, file paths, or text).+ data_lengths: Lengths of each input sample in the batch.+ key: Sample identifiers.+ tokenizer: Tokenizer instance for text encoding/decoding.+ frontend: Audio frontend for feature extraction.+ **kwargs: Additional keyword arguments.+ """+ is_use_ctc = kwargs.get("decoding_ctc_weight", 0.0) > 0.00001 and self.ctc != None+ is_use_lm = (+ kwargs.get("lm_weight", 0.0) > 0.00001 and kwargs.get("lm_file", None) is not None+ )+ if self.beam_search is None and (is_use_lm or is_use_ctc):+ logging.info("enable beam_search")+ self.init_beam_search(**kwargs)+ self.nbest = kwargs.get("nbest", 1)++</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/bicif_paraformer/model.py#L271" target="_blank" class="src-more">View full source on GitHub →</a></details></div>+<div class="api-detail" id="e18"><span class="dtag badge-m">method</span><h2><code>BiCifParaformer.export(**kwargs)</code></h2><div class="dmod">funasr.models.bicif_paraformer.BiCifParaformer · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/bicif_paraformer/model.py#L430" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Export.</p>+<h4>Args:</h4>+<ul>+<li><code>**kwargs</code> — Additional keyword arguments.</li>+</ul>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def export(self, **kwargs):+ """Export.++ Args:+ **kwargs: Additional keyword arguments.+ """+ from .export_meta import export_rebuild_model++ if "max_seq_len" not in kwargs:+ kwargs["max_seq_len"] = 512+ models = export_rebuild_model(model=self, **kwargs)+ return models+</code></pre></details></div>+<div class="api-detail" id="e19"><span class="dtag badge-c">class</span><h2>Branchformer</h2><div class="dmod">funasr.models.branchformer · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/branchformer/model.py#L8" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><ul>+<li><code>Branchformer</code> — Parallel branch encoder architecture.</li>+</ul>+<p>Uses parallel branches of self-attention and convolution that are</p>+<p>merged via concatenation. Alternative to Conformer with similar accuracy.</p>+<p>Inherits Transformer pipeline for training and inference.</p>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code>class Branchformer(Transformer):+ """Branchformer: Parallel branch encoder architecture.++ Uses parallel branches of self-attention and convolution that are+ merged via concatenation. Alternative to Conformer with similar accuracy.++ Inherits Transformer pipeline for training and inference.+ """++ def __init__(+ self,+ *args,+ **kwargs,+ ):++ """Initialize Branchformer.++ Args:+ *args: Variable positional arguments.+ **kwargs: Additional keyword arguments.+ """+ super().__init__(*args, **kwargs)+</code></pre></details></div>+<div class="api-detail" id="e20"><span class="dtag badge-c">class</span><h2>CAMPPlus</h2><div class="dmod">funasr.models.campplus · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/campplus/model.py#L43" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>CAM++ Speaker Verification Model.</p>+<p>Extracts fixed-dimensional speaker embeddings from variable-length audio.</p>+<p>Used for speaker verification and speaker diarization pipelines.</p>+<ul>+<li><code>Output</code> — 192-dimensional speaker embedding per utterance.</li>+</ul>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code>class CAMPPlus(torch.nn.Module):+ """CAM++ Speaker Verification Model.++ Extracts fixed-dimensional speaker embeddings from variable-length audio.+ Used for speaker verification and speaker diarization pipelines.++ Output: 192-dimensional speaker embedding per utterance.+ """++ def __init__(+ self,+ feat_dim=80,+ embedding_size=192,+ growth_rate=32,+ bn_size=4,+ init_channels=128,+ config_str="batchnorm-relu",+ memory_efficient=True,+ output_level="segment",+ **kwargs,+ ):+ """Initialize CAMPPlus.++ Args:+ feat_dim: Size/dimension parameter.+ embedding_size: Size/dimension parameter.+ growth_rate: TODO.+ bn_size: Size/dimension parameter.+ init_channels: TODO.+ config_str: TODO.+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/campplus/model.py#L43" target="_blank" class="src-more">View full source on GitHub →</a></details><h3 class="mt">Methods</h3><div class="mblk"><code>.forward(x)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/campplus/model.py#L141" target="_blank" class="gh-link">L141</a><div class="mdoc"><p>Extract speaker embedding from fbank features.</p>+<h4>Args:</h4>+<ul>+<li><code>x (Tensor)</code> — Input fbank features, shape (batch, time, feat_dim).</li>+</ul>+<h4>Returns:</h4>+<ul>+<li><code>Tensor</code> — Speaker embedding, shape (batch, embedding_size) for segment level,</li>+<li class="sub">or (batch, time, channels) for frame level.</li>+</ul>+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def forward(self, x):+ """Extract speaker embedding from fbank features.++ Args:+ x (Tensor): Input fbank features, shape (batch, time, feat_dim).++ Returns:+ Tensor: Speaker embedding, shape (batch, embedding_size) for segment level,+ or (batch, time, channels) for frame level.+ """+ x = x.permute(0, 2, 1) # (B,T,F) => (B,F,T)+ x = self.head(x)+ x = self.xvector(x)+ if self.output_level == "frame":+ x = x.transpose(1, 2)+ return x+</code></pre></details></div><div class="mblk"><code>.inference(data_in, data_lengths, key, tokenizer, frontend, **kwargs)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/campplus/model.py#L158" target="_blank" class="gh-link">L158</a><div class="mdoc"><p>Run speaker embedding extraction on audio input.</p>+<h4>Args:</h4>+<ul>+<li><code>data_in</code> — Audio input (file path, numpy array, or list).</li>+<li><code>data_lengths</code> — Not used.</li>+<li><code>key (list)</code> — Sample identifiers.</li>+<li><code>tokenizer</code> — Not used.</li>+<li><code>frontend</code> — Not used.</li>+<li><code>**kwargs</code> — Must include 'device' (str) and optional 'fs' (int, default 16000).</li>+</ul>+<h4>Returns:</h4>+<ul>+<li><code>tuple</code> — (results, meta_data) where results is</li>+<li class="sub">[{"spk_embedding": Tensor of shape (1, 192)}]</li>+</ul>+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def inference(+ self,+ data_in,+ data_lengths=None,+ key: list = None,+ tokenizer=None,+ frontend=None,+ **kwargs,+ ):+ """Run speaker embedding extraction on audio input.++ Args:+ data_in: Audio input (file path, numpy array, or list).+ data_lengths: Not used.+ key (list): Sample identifiers.+ tokenizer: Not used.+ frontend: Not used.+ **kwargs: Must include 'device' (str) and optional 'fs' (int, default 16000).++ Returns:+ tuple: (results, meta_data) where results is+ [{"spk_embedding": Tensor of shape (1, 192)}]+ """+ # extract fbank feats+ meta_data = {}+ time1 = time.perf_counter()+ audio_sample_list = load_audio_text_image_video(+ data_in, fs=16000, audio_fs=kwargs.get("fs", 16000), data_type="sound"+ )+ time2 = time.perf_counter()+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/campplus/model.py#L158" target="_blank" class="src-more">View full source →</a></details></div></div>+<div class="api-detail" id="e21"><span class="dtag badge-m">method</span><h2><code>CAMPPlus.forward(x)</code></h2><div class="dmod">funasr.models.campplus.CAMPPlus · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/campplus/model.py#L141" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Extract speaker embedding from fbank features.</p>+<h4>Args:</h4>+<ul>+<li><code>x (Tensor)</code> — Input fbank features, shape (batch, time, feat_dim).</li>+</ul>+<h4>Returns:</h4>+<ul>+<li><code>Tensor</code> — Speaker embedding, shape (batch, embedding_size) for segment level,</li>+<li class="sub">or (batch, time, channels) for frame level.</li>+</ul>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def forward(self, x):+ """Extract speaker embedding from fbank features.++ Args:+ x (Tensor): Input fbank features, shape (batch, time, feat_dim).++ Returns:+ Tensor: Speaker embedding, shape (batch, embedding_size) for segment level,+ or (batch, time, channels) for frame level.+ """+ x = x.permute(0, 2, 1) # (B,T,F) => (B,F,T)+ x = self.head(x)+ x = self.xvector(x)+ if self.output_level == "frame":+ x = x.transpose(1, 2)+ return x+</code></pre></details></div>+<div class="api-detail" id="e22"><span class="dtag badge-m">method</span><h2><code>CAMPPlus.inference(data_in, data_lengths, key, tokenizer, frontend, **kwargs)</code></h2><div class="dmod">funasr.models.campplus.CAMPPlus · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/campplus/model.py#L158" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Run speaker embedding extraction on audio input.</p>+<h4>Args:</h4>+<ul>+<li><code>data_in</code> — Audio input (file path, numpy array, or list).</li>+<li><code>data_lengths</code> — Not used.</li>+<li><code>key (list)</code> — Sample identifiers.</li>+<li><code>tokenizer</code> — Not used.</li>+<li><code>frontend</code> — Not used.</li>+<li><code>**kwargs</code> — Must include 'device' (str) and optional 'fs' (int, default 16000).</li>+</ul>+<h4>Returns:</h4>+<ul>+<li><code>tuple</code> — (results, meta_data) where results is</li>+<li class="sub">[{"spk_embedding": Tensor of shape (1, 192)}]</li>+</ul>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def inference(+ self,+ data_in,+ data_lengths=None,+ key: list = None,+ tokenizer=None,+ frontend=None,+ **kwargs,+ ):+ """Run speaker embedding extraction on audio input.++ Args:+ data_in: Audio input (file path, numpy array, or list).+ data_lengths: Not used.+ key (list): Sample identifiers.+ tokenizer: Not used.+ frontend: Not used.+ **kwargs: Must include 'device' (str) and optional 'fs' (int, default 16000).++ Returns:+ tuple: (results, meta_data) where results is+ [{"spk_embedding": Tensor of shape (1, 192)}]+ """+ # extract fbank feats+ meta_data = {}+ time1 = time.perf_counter()+ audio_sample_list = load_audio_text_image_video(+ data_in, fs=16000, audio_fs=kwargs.get("fs", 16000), data_type="sound"+ )+ time2 = time.perf_counter()+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/campplus/model.py#L158" target="_blank" class="src-more">View full source on GitHub →</a></details></div>+<div class="api-detail" id="e23"><span class="dtag badge-c">class</span><h2>Conformer</h2><div class="dmod">funasr.models.conformer · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/conformer/model.py#L10" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><ul>+<li><code>Conformer</code> — CTC-attention hybrid encoder-decoder model.</li>+</ul>+<p>Combines convolution and self-attention in the encoder for better</p>+<p>local and global context modeling. Inherits full Transformer pipeline</p>+<p>(CTC + attention decoder + beam search).</p>+<ul>+<li><code>Output</code> — {"key": str, "text": str}</li>+</ul>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code>class Conformer(Transformer):+ """Conformer: CTC-attention hybrid encoder-decoder model.++ Combines convolution and self-attention in the encoder for better+ local and global context modeling. Inherits full Transformer pipeline+ (CTC + attention decoder + beam search).++ Output: {"key": str, "text": str}+ """++ def __init__(+ self,+ *args,+ **kwargs,+ ):++ """Initialize Conformer.++ Args:+ *args: Variable positional arguments.+ **kwargs: Additional keyword arguments.+ """+ super().__init__(*args, **kwargs)+</code></pre></details></div>+<div class="api-detail" id="e24"><span class="dtag badge-c">class</span><h2>Conformer</h2><div class="dmod">funasr.models.conformer_rwkv · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/conformer_rwkv/model.py#L10" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><ul>+<li><code>CTC</code> — attention hybrid Encoder-Decoder model</li>+</ul>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code>class Conformer(Transformer):+ """CTC-attention hybrid Encoder-Decoder model"""++ def __init__(+ self,+ *args,+ **kwargs,+ ):++ """Initialize Conformer.++ Args:+ *args: Variable positional arguments.+ **kwargs: Additional keyword arguments.+ """+ super().__init__(*args, **kwargs)+</code></pre></details></div>+<div class="api-detail" id="e25"><span class="dtag badge-c">class</span><h2>ContextualParaformer</h2><div class="dmod">funasr.models.contextual_paraformer · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/contextual_paraformer/model.py#L46" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><ul>+<li><code>ContextualParaformer</code> — Paraformer with hotword/context biasing.</li>+</ul>+<p>Extends Paraformer with a context encoder that incorporates user-defined</p>+<p>hotwords/keywords to boost recognition of domain-specific terms.</p>+<ul>+<li><code>Usage</code> — Pass hotwords via generate(hotword='term1 term2').</li>+</ul>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code>class ContextualParaformer(Paraformer):+ """ContextualParaformer: Paraformer with hotword/context biasing.++ Extends Paraformer with a context encoder that incorporates user-defined+ hotwords/keywords to boost recognition of domain-specific terms.++ Usage: Pass hotwords via generate(hotword='term1 term2').+ """++ def __init__(+ self,+ *args,+ **kwargs,+ ):+ """Initialize ContextualParaformer.++ Args:+ *args: Variable positional arguments.+ **kwargs: Additional keyword arguments.+ """+ super().__init__(*args, **kwargs)++ self.target_buffer_length = kwargs.get("target_buffer_length", -1)+ inner_dim = kwargs.get("inner_dim", 256)+ bias_encoder_type = kwargs.get("bias_encoder_type", "lstm")+ use_decoder_embedding = kwargs.get("use_decoder_embedding", False)+ crit_attn_weight = kwargs.get("crit_attn_weight", 0.0)+ crit_attn_smooth = kwargs.get("crit_attn_smooth", 0.0)+ bias_encoder_dropout_rate = kwargs.get("bias_encoder_dropout_rate", 0.0)++</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/contextual_paraformer/model.py#L46" target="_blank" class="src-more">View full source on GitHub →</a></details><h3 class="mt">Methods</h3><div class="mblk"><code>.forward(speech, speech_lengths, text, text_lengths, **kwargs)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/contextual_paraformer/model.py#L96" target="_blank" class="gh-link">L96</a><div class="mdoc"><p>Frontend + Encoder + Decoder + Calc loss</p>+<h4>Args:</h4>+<ul>+<li><code>speech</code> — (Batch, Length, ...)</li>+<li><code>speech_lengths</code> — (Batch, )</li>+<li><code>text</code> — (Batch, Length)</li>+<li><code>text_lengths</code> — (Batch,)</li>+</ul>+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def forward(+ self,+ speech: torch.Tensor,+ speech_lengths: torch.Tensor,+ text: torch.Tensor,+ text_lengths: torch.Tensor,+ **kwargs,+ ) -> Tuple[torch.Tensor, Dict[str, torch.Tensor], torch.Tensor]:+ """Frontend + Encoder + Decoder + Calc loss++ Args:+ speech: (Batch, Length, ...)+ speech_lengths: (Batch, )+ text: (Batch, Length)+ text_lengths: (Batch,)+ """+ text_lengths = text_lengths.squeeze()+ speech_lengths = speech_lengths.squeeze()++ batch_size = speech.shape[0]++ hotword_pad = kwargs.get("hotword_pad")+ hotword_lengths = kwargs.get("hotword_lengths")+ # dha_pad = kwargs.get("dha_pad")++ # 1. Encoder+ encoder_out, encoder_out_lens = self.encode(speech, speech_lengths)++ loss_ctc, cer_ctc = None, None++</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/contextual_paraformer/model.py#L96" target="_blank" class="src-more">View full source →</a></details></div><div class="mblk"><code>.sampler(encoder_out, encoder_out_lens, ys_pad, ys_pad_lens, pre_acoustic_embeds, contextual_info)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/contextual_paraformer/model.py#L268" target="_blank" class="gh-link">L268</a><div class="mdoc"><p>Sampler.</p>+<h4>Args:</h4>+<ul>+<li><code>encoder_out</code> — Encoder output tensor.</li>+<li><code>encoder_out_lens</code> — Encoder output lengths.</li>+<li><code>ys_pad</code> — TODO.</li>+<li><code>ys_pad_lens</code> — Lengths of ys_pad.</li>+<li><code>pre_acoustic_embeds</code> — TODO.</li>+<li><code>contextual_info</code> — TODO.</li>+</ul>+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def sampler(+ self,+ encoder_out,+ encoder_out_lens,+ ys_pad,+ ys_pad_lens,+ pre_acoustic_embeds,+ contextual_info,+ ):+ """Sampler.++ Args:+ encoder_out: Encoder output tensor.+ encoder_out_lens: Encoder output lengths.+ ys_pad: TODO.+ ys_pad_lens: Lengths of ys_pad.+ pre_acoustic_embeds: TODO.+ contextual_info: TODO.+ """+ tgt_mask = (~make_pad_mask(ys_pad_lens, maxlen=ys_pad_lens.max())[:, :, None]).to(+ ys_pad.device+ )+ ys_pad = ys_pad * tgt_mask[:, :, 0]+ if self.share_embedding:+ ys_pad_embed = self.decoder.output_layer.weight[ys_pad]+ else:+ ys_pad_embed = self.decoder.embed(ys_pad)+ with torch.no_grad():+ decoder_outs = self.decoder(+ encoder_out,+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/contextual_paraformer/model.py#L268" target="_blank" class="src-more">View full source →</a></details></div><div class="mblk"><code>.cal_decoder_with_predictor(encoder_out, encoder_out_lens, sematic_embeds, ys_pad_lens, hw_list, clas_scale)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/contextual_paraformer/model.py#L331" target="_blank" class="gh-link">L331</a><div class="mdoc"><p>Cal decoder with predictor.</p>+<h4>Args:</h4>+<ul>+<li><code>encoder_out</code> — Encoder output tensor.</li>+<li><code>encoder_out_lens</code> — Encoder output lengths.</li>+<li><code>sematic_embeds</code> — TODO.</li>+<li><code>ys_pad_lens</code> — Lengths of ys_pad.</li>+<li><code>hw_list</code> — TODO.</li>+<li><code>clas_scale</code> — TODO.</li>+</ul>+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def cal_decoder_with_predictor(+ self,+ encoder_out,+ encoder_out_lens,+ sematic_embeds,+ ys_pad_lens,+ hw_list=None,+ clas_scale=1.0,+ ):+ """Cal decoder with predictor.++ Args:+ encoder_out: Encoder output tensor.+ encoder_out_lens: Encoder output lengths.+ sematic_embeds: TODO.+ ys_pad_lens: Lengths of ys_pad.+ hw_list: TODO.+ clas_scale: TODO.+ """+ if hw_list is None:+ hw_list = [torch.Tensor([1]).long().to(encoder_out.device)] # empty hotword list+ hw_list_pad = pad_list(hw_list, 0)+ if self.use_decoder_embedding:+ hw_embed = self.decoder.embed(hw_list_pad)+ else:+ hw_embed = self.bias_embed(hw_list_pad)+ hw_embed, (h_n, _) = self.bias_encoder(hw_embed)+ hw_embed = h_n.repeat(encoder_out.shape[0], 1, 1)+ else:+ hw_lengths = [len(i) for i in hw_list]+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/contextual_paraformer/model.py#L331" target="_blank" class="src-more">View full source →</a></details></div><div class="mblk"><code>.inference(data_in, data_lengths, key, tokenizer, frontend, **kwargs)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/contextual_paraformer/model.py#L387" target="_blank" class="gh-link">L387</a><div class="mdoc"><p>Run inference on input data.</p>+<h4>Args:</h4>+<ul>+<li><code>data_in</code> — Input data (audio samples, file paths, or text).</li>+<li><code>data_lengths</code> — Lengths of each input sample in the batch.</li>+<li><code>key</code> — Sample identifiers.</li>+<li><code>tokenizer</code> — Tokenizer instance for text encoding/decoding.</li>+<li><code>frontend</code> — Audio frontend for feature extraction.</li>+<li><code>**kwargs</code> — Additional keyword arguments.</li>+</ul>+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def inference(+ self,+ data_in,+ data_lengths=None,+ key: list = None,+ tokenizer=None,+ frontend=None,+ **kwargs,+ ):+ # init beamsearch++ """Run inference on input data.++ Args:+ data_in: Input data (audio samples, file paths, or text).+ data_lengths: Lengths of each input sample in the batch.+ key: Sample identifiers.+ tokenizer: Tokenizer instance for text encoding/decoding.+ frontend: Audio frontend for feature extraction.+ **kwargs: Additional keyword arguments.+ """+ is_use_ctc = kwargs.get("decoding_ctc_weight", 0.0) > 0.00001 and self.ctc != None+ is_use_lm = (+ kwargs.get("lm_weight", 0.0) > 0.00001 and kwargs.get("lm_file", None) is not None+ )+ if self.beam_search is None and (is_use_lm or is_use_ctc):+ logging.info("enable beam_search")+ self.init_beam_search(**kwargs)+ self.nbest = kwargs.get("nbest", 1)++</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/contextual_paraformer/model.py#L387" target="_blank" class="src-more">View full source →</a></details></div><div class="mblk"><code>.generate_hotwords_list(hotword_list_or_file, tokenizer, frontend)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/contextual_paraformer/model.py#L534" target="_blank" class="gh-link">L534</a><div class="mdoc"><p>Generate hotwords list.</p>+<h4>Args:</h4>+<ul>+<li><code>hotword_list_or_file</code> — TODO.</li>+<li><code>tokenizer</code> — Tokenizer instance for text encoding/decoding.</li>+<li><code>frontend</code> — Audio frontend for feature extraction.</li>+</ul>+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def generate_hotwords_list(self, hotword_list_or_file, tokenizer=None, frontend=None):+ """Generate hotwords list.++ Args:+ hotword_list_or_file: TODO.+ tokenizer: Tokenizer instance for text encoding/decoding.+ frontend: Audio frontend for feature extraction.+ """+ def load_seg_dict(seg_dict_file):+ """Load seg dict.++ Args:+ seg_dict_file: TODO.+ """+ seg_dict = {}+ assert isinstance(seg_dict_file, str)+ with open(seg_dict_file, "r", encoding="utf8") as f:+ lines = f.readlines()+ for line in lines:+ s = line.strip().split()+ key = s[0]+ value = s[1:]+ seg_dict[key] = " ".join(value)+ return seg_dict++ def seg_tokenize(txt, seg_dict):+ """Seg tokenize.++ Args:+ txt: TODO.+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/contextual_paraformer/model.py#L534" target="_blank" class="src-more">View full source →</a></details></div><div class="mblk"><code>.export(**kwargs)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/contextual_paraformer/model.py#L659" target="_blank" class="gh-link">L659</a><div class="mdoc"><p>Export.</p>+<h4>Args:</h4>+<ul>+<li><code>**kwargs</code> — Additional keyword arguments.</li>+</ul>+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def export(+ self,+ **kwargs,+ ):+ """Export.++ Args:+ **kwargs: Additional keyword arguments.+ """+ if "max_seq_len" not in kwargs:+ kwargs["max_seq_len"] = 512+ from .export_meta import export_rebuild_model++ models = export_rebuild_model(model=self, **kwargs)+ return models+</code></pre></details></div></div>+<div class="api-detail" id="e26"><span class="dtag badge-m">method</span><h2><code>ContextualParaformer.forward(speech, speech_lengths, text, text_lengths, **kwargs)</code></h2><div class="dmod">funasr.models.contextual_paraformer.ContextualParaformer · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/contextual_paraformer/model.py#L96" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Frontend + Encoder + Decoder + Calc loss</p>+<h4>Args:</h4>+<ul>+<li><code>speech</code> — (Batch, Length, ...)</li>+<li><code>speech_lengths</code> — (Batch, )</li>+<li><code>text</code> — (Batch, Length)</li>+<li><code>text_lengths</code> — (Batch,)</li>+</ul>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def forward(+ self,+ speech: torch.Tensor,+ speech_lengths: torch.Tensor,+ text: torch.Tensor,+ text_lengths: torch.Tensor,+ **kwargs,+ ) -> Tuple[torch.Tensor, Dict[str, torch.Tensor], torch.Tensor]:+ """Frontend + Encoder + Decoder + Calc loss++ Args:+ speech: (Batch, Length, ...)+ speech_lengths: (Batch, )+ text: (Batch, Length)+ text_lengths: (Batch,)+ """+ text_lengths = text_lengths.squeeze()+ speech_lengths = speech_lengths.squeeze()++ batch_size = speech.shape[0]++ hotword_pad = kwargs.get("hotword_pad")+ hotword_lengths = kwargs.get("hotword_lengths")+ # dha_pad = kwargs.get("dha_pad")++ # 1. Encoder+ encoder_out, encoder_out_lens = self.encode(speech, speech_lengths)++ loss_ctc, cer_ctc = None, None++</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/contextual_paraformer/model.py#L96" target="_blank" class="src-more">View full source on GitHub →</a></details></div>+<div class="api-detail" id="e27"><span class="dtag badge-m">method</span><h2><code>ContextualParaformer.sampler(encoder_out, encoder_out_lens, ys_pad, ys_pad_lens, pre_acoustic_embeds, contextual_info)</code></h2><div class="dmod">funasr.models.contextual_paraformer.ContextualParaformer · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/contextual_paraformer/model.py#L268" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Sampler.</p>+<h4>Args:</h4>+<ul>+<li><code>encoder_out</code> — Encoder output tensor.</li>+<li><code>encoder_out_lens</code> — Encoder output lengths.</li>+<li><code>ys_pad</code> — TODO.</li>+<li><code>ys_pad_lens</code> — Lengths of ys_pad.</li>+<li><code>pre_acoustic_embeds</code> — TODO.</li>+<li><code>contextual_info</code> — TODO.</li>+</ul>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def sampler(+ self,+ encoder_out,+ encoder_out_lens,+ ys_pad,+ ys_pad_lens,+ pre_acoustic_embeds,+ contextual_info,+ ):+ """Sampler.++ Args:+ encoder_out: Encoder output tensor.+ encoder_out_lens: Encoder output lengths.+ ys_pad: TODO.+ ys_pad_lens: Lengths of ys_pad.+ pre_acoustic_embeds: TODO.+ contextual_info: TODO.+ """+ tgt_mask = (~make_pad_mask(ys_pad_lens, maxlen=ys_pad_lens.max())[:, :, None]).to(+ ys_pad.device+ )+ ys_pad = ys_pad * tgt_mask[:, :, 0]+ if self.share_embedding:+ ys_pad_embed = self.decoder.output_layer.weight[ys_pad]+ else:+ ys_pad_embed = self.decoder.embed(ys_pad)+ with torch.no_grad():+ decoder_outs = self.decoder(+ encoder_out,+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/contextual_paraformer/model.py#L268" target="_blank" class="src-more">View full source on GitHub →</a></details></div>+<div class="api-detail" id="e28"><span class="dtag badge-m">method</span><h2><code>ContextualParaformer.cal_decoder_with_predictor(encoder_out, encoder_out_lens, sematic_embeds, ys_pad_lens, hw_list, clas_scale)</code></h2><div class="dmod">funasr.models.contextual_paraformer.ContextualParaformer · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/contextual_paraformer/model.py#L331" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Cal decoder with predictor.</p>+<h4>Args:</h4>+<ul>+<li><code>encoder_out</code> — Encoder output tensor.</li>+<li><code>encoder_out_lens</code> — Encoder output lengths.</li>+<li><code>sematic_embeds</code> — TODO.</li>+<li><code>ys_pad_lens</code> — Lengths of ys_pad.</li>+<li><code>hw_list</code> — TODO.</li>+<li><code>clas_scale</code> — TODO.</li>+</ul>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def cal_decoder_with_predictor(+ self,+ encoder_out,+ encoder_out_lens,+ sematic_embeds,+ ys_pad_lens,+ hw_list=None,+ clas_scale=1.0,+ ):+ """Cal decoder with predictor.++ Args:+ encoder_out: Encoder output tensor.+ encoder_out_lens: Encoder output lengths.+ sematic_embeds: TODO.+ ys_pad_lens: Lengths of ys_pad.+ hw_list: TODO.+ clas_scale: TODO.+ """+ if hw_list is None:+ hw_list = [torch.Tensor([1]).long().to(encoder_out.device)] # empty hotword list+ hw_list_pad = pad_list(hw_list, 0)+ if self.use_decoder_embedding:+ hw_embed = self.decoder.embed(hw_list_pad)+ else:+ hw_embed = self.bias_embed(hw_list_pad)+ hw_embed, (h_n, _) = self.bias_encoder(hw_embed)+ hw_embed = h_n.repeat(encoder_out.shape[0], 1, 1)+ else:+ hw_lengths = [len(i) for i in hw_list]+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/contextual_paraformer/model.py#L331" target="_blank" class="src-more">View full source on GitHub →</a></details></div>+<div class="api-detail" id="e29"><span class="dtag badge-m">method</span><h2><code>ContextualParaformer.inference(data_in, data_lengths, key, tokenizer, frontend, **kwargs)</code></h2><div class="dmod">funasr.models.contextual_paraformer.ContextualParaformer · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/contextual_paraformer/model.py#L387" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Run inference on input data.</p>+<h4>Args:</h4>+<ul>+<li><code>data_in</code> — Input data (audio samples, file paths, or text).</li>+<li><code>data_lengths</code> — Lengths of each input sample in the batch.</li>+<li><code>key</code> — Sample identifiers.</li>+<li><code>tokenizer</code> — Tokenizer instance for text encoding/decoding.</li>+<li><code>frontend</code> — Audio frontend for feature extraction.</li>+<li><code>**kwargs</code> — Additional keyword arguments.</li>+</ul>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def inference(+ self,+ data_in,+ data_lengths=None,+ key: list = None,+ tokenizer=None,+ frontend=None,+ **kwargs,+ ):+ # init beamsearch++ """Run inference on input data.++ Args:+ data_in: Input data (audio samples, file paths, or text).+ data_lengths: Lengths of each input sample in the batch.+ key: Sample identifiers.+ tokenizer: Tokenizer instance for text encoding/decoding.+ frontend: Audio frontend for feature extraction.+ **kwargs: Additional keyword arguments.+ """+ is_use_ctc = kwargs.get("decoding_ctc_weight", 0.0) > 0.00001 and self.ctc != None+ is_use_lm = (+ kwargs.get("lm_weight", 0.0) > 0.00001 and kwargs.get("lm_file", None) is not None+ )+ if self.beam_search is None and (is_use_lm or is_use_ctc):+ logging.info("enable beam_search")+ self.init_beam_search(**kwargs)+ self.nbest = kwargs.get("nbest", 1)++</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/contextual_paraformer/model.py#L387" target="_blank" class="src-more">View full source on GitHub →</a></details></div>+<div class="api-detail" id="e30"><span class="dtag badge-m">method</span><h2><code>ContextualParaformer.generate_hotwords_list(hotword_list_or_file, tokenizer, frontend)</code></h2><div class="dmod">funasr.models.contextual_paraformer.ContextualParaformer · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/contextual_paraformer/model.py#L534" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Generate hotwords list.</p>+<h4>Args:</h4>+<ul>+<li><code>hotword_list_or_file</code> — TODO.</li>+<li><code>tokenizer</code> — Tokenizer instance for text encoding/decoding.</li>+<li><code>frontend</code> — Audio frontend for feature extraction.</li>+</ul>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def generate_hotwords_list(self, hotword_list_or_file, tokenizer=None, frontend=None):+ """Generate hotwords list.++ Args:+ hotword_list_or_file: TODO.+ tokenizer: Tokenizer instance for text encoding/decoding.+ frontend: Audio frontend for feature extraction.+ """+ def load_seg_dict(seg_dict_file):+ """Load seg dict.++ Args:+ seg_dict_file: TODO.+ """+ seg_dict = {}+ assert isinstance(seg_dict_file, str)+ with open(seg_dict_file, "r", encoding="utf8") as f:+ lines = f.readlines()+ for line in lines:+ s = line.strip().split()+ key = s[0]+ value = s[1:]+ seg_dict[key] = " ".join(value)+ return seg_dict++ def seg_tokenize(txt, seg_dict):+ """Seg tokenize.++ Args:+ txt: TODO.+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/contextual_paraformer/model.py#L534" target="_blank" class="src-more">View full source on GitHub →</a></details></div>+<div class="api-detail" id="e31"><span class="dtag badge-m">method</span><h2><code>ContextualParaformer.export(**kwargs)</code></h2><div class="dmod">funasr.models.contextual_paraformer.ContextualParaformer · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/contextual_paraformer/model.py#L659" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Export.</p>+<h4>Args:</h4>+<ul>+<li><code>**kwargs</code> — Additional keyword arguments.</li>+</ul>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def export(+ self,+ **kwargs,+ ):+ """Export.++ Args:+ **kwargs: Additional keyword arguments.+ """+ if "max_seq_len" not in kwargs:+ kwargs["max_seq_len"] = 512+ from .export_meta import export_rebuild_model++ models = export_rebuild_model(model=self, **kwargs)+ return models+</code></pre></details></div>+<div class="api-detail" id="e32"><span class="dtag badge-c">class</span><h2>CTTransformer</h2><div class="dmod">funasr.models.ct_transformer · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/ct_transformer/model.py#L40" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><ul>+<li><code>CT</code> — Transformer: Punctuation Restoration Model.</li>+</ul>+<p>Adds punctuation (comma, period, question mark) to unpunctuated text.</p>+<p>Supports Chinese and English. Used as punc_model in the ASR pipeline.</p>+<ul>+<li><code>Output</code> — {"key": "...", "text": "punctuated text", "punc_array": Tensor}</li>+</ul>+<p>punc_array encoding: 1=none, 2=comma(,), 3=period(。), 4=question(?)</p>+<ul>+<li><code>Note</code> — Not needed for Fun-ASR-Nano/SenseVoice/Qwen3-ASR (they output punctuation natively).</li>+</ul>+<p>Only required for Paraformer models.</p>+<ul>+<li><code>Author</code> — Speech Lab of DAMO Academy, Alibaba Group</li>+<li><code>CT</code> — Transformer: Controllable time-delay transformer for real-time punctuation prediction and disfluency detection</li>+</ul>+<p>https://arxiv.org/pdf/2003.01309.pdf</p>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code>class CTTransformer(torch.nn.Module):+ """CT-Transformer: Punctuation Restoration Model.++ Adds punctuation (comma, period, question mark) to unpunctuated text.+ Supports Chinese and English. Used as punc_model in the ASR pipeline.++ Output: {"key": "...", "text": "punctuated text", "punc_array": Tensor}+ punc_array encoding: 1=none, 2=comma(,), 3=period(。), 4=question(?)++ Note: Not needed for Fun-ASR-Nano/SenseVoice/Qwen3-ASR (they output punctuation natively).+ Only required for Paraformer models.++ Author: Speech Lab of DAMO Academy, Alibaba Group+ CT-Transformer: Controllable time-delay transformer for real-time punctuation prediction and disfluency detection+ https://arxiv.org/pdf/2003.01309.pdf+ """++ def __init__(+ self,+ encoder: str = None,+ encoder_conf: dict = None,+ vocab_size: int = -1,+ punc_list: list = None,+ punc_weight: list = None,+ embed_unit: int = 128,+ att_unit: int = 256,+ dropout_rate: float = 0.5,+ ignore_id: int = -1,+ sos: int = 1,+ eos: int = 2,+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/ct_transformer/model.py#L40" target="_blank" class="src-more">View full source on GitHub →</a></details><h3 class="mt">Methods</h3><div class="mblk"><code>.punc_forward(text, text_lengths, **kwargs)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/ct_transformer/model.py#L113" target="_blank" class="gh-link">L113</a><div class="mdoc"><p>Compute loss value from buffer sequences.</p>+<h4>Args:</h4>+<ul>+<li><code>input (torch.Tensor)</code> — Input ids. (batch, len)</li>+<li><code>hidden (torch.Tensor)</code> — Target ids. (batch, len)</li>+</ul>+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def punc_forward(self, text: torch.Tensor, text_lengths: torch.Tensor, **kwargs):+ """Compute loss value from buffer sequences.++ Args:+ input (torch.Tensor): Input ids. (batch, len)+ hidden (torch.Tensor): Target ids. (batch, len)++ """+ x = self.embed(text)+ # mask = self._target_mask(input)+ h, _, _ = self.encoder(x, text_lengths)+ y = self.decoder(h)+ return y, None+</code></pre></details></div><div class="mblk"><code>.with_vad()</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/ct_transformer/model.py#L127" target="_blank" class="gh-link">L127</a><div class="mdoc"><p>With vad.</p>+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def with_vad(self):+ """With vad."""+ return False+</code></pre></details></div><div class="mblk"><code>.score(y, state, x)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/ct_transformer/model.py#L131" target="_blank" class="gh-link">L131</a><div class="mdoc"><p>Score new token.</p>+<h4>Args:</h4>+<ul>+<li><code>y (torch.Tensor)</code> — 1D torch.int64 prefix tokens.</li>+<li><code>state</code> — Scorer state for prefix tokens</li>+<li><code>x (torch.Tensor)</code> — encoder feature that generates ys.</li>+</ul>+<h4>Returns:</h4>+<p>tuple[torch.Tensor, Any]: Tuple of</p>+<p>torch.float32 scores for next token (vocab_size)</p>+<p>and next state for ys</p>+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def score(self, y: torch.Tensor, state: Any, x: torch.Tensor) -> Tuple[torch.Tensor, Any]:+ """Score new token.++ Args:+ y (torch.Tensor): 1D torch.int64 prefix tokens.+ state: Scorer state for prefix tokens+ x (torch.Tensor): encoder feature that generates ys.++ Returns:+ tuple[torch.Tensor, Any]: Tuple of+ torch.float32 scores for next token (vocab_size)+ and next state for ys++ """+ y = y.unsqueeze(0)+ h, _, cache = self.encoder.forward_one_step(+ self.embed(y), self._target_mask(y), cache=state+ )+ h = self.decoder(h[:, -1])+ logp = h.log_softmax(dim=-1).squeeze(0)+ return logp, cache+</code></pre></details></div><div class="mblk"><code>.batch_score(ys, states, xs)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/ct_transformer/model.py#L153" target="_blank" class="gh-link">L153</a><div class="mdoc"><p>Score new token batch.</p>+<h4>Args:</h4>+<ul>+<li><code>ys (torch.Tensor)</code> — torch.int64 prefix tokens (n_batch, ylen).</li>+<li><code>states (List[Any])</code> — Scorer states for prefix tokens.</li>+</ul>+<p>xs (torch.Tensor):</p>+<p>The encoder feature that generates ys (n_batch, xlen, n_feat).</p>+<h4>Returns:</h4>+<p>tuple[torch.Tensor, List[Any]]: Tuple of</p>+<p>batchfied scores for next token with shape of `(n_batch, vocab_size)`</p>+<p>and next state list for ys.</p>+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def batch_score(+ self, ys: torch.Tensor, states: List[Any], xs: torch.Tensor+ ) -> Tuple[torch.Tensor, List[Any]]:+ """Score new token batch.++ Args:+ ys (torch.Tensor): torch.int64 prefix tokens (n_batch, ylen).+ states (List[Any]): Scorer states for prefix tokens.+ xs (torch.Tensor):+ The encoder feature that generates ys (n_batch, xlen, n_feat).++ Returns:+ tuple[torch.Tensor, List[Any]]: Tuple of+ batchfied scores for next token with shape of `(n_batch, vocab_size)`+ and next state list for ys.++ """+ # merge states+ n_batch = len(ys)+ n_layers = len(self.encoder.encoders)+ if states[0] is None:+ batch_state = None+ else:+ # transpose state of [batch, layer] into [layer, batch]+ batch_state = [+ torch.stack([states[b][i] for b in range(n_batch)]) for i in range(n_layers)+ ]++ # batch decoding+ h, _, states = self.encoder.forward_one_step(+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/ct_transformer/model.py#L153" target="_blank" class="src-more">View full source →</a></details></div><div class="mblk"><code>.nll(text, punc, text_lengths, punc_lengths, max_length, vad_indexes, vad_indexes_lengths)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/ct_transformer/model.py#L192" target="_blank" class="gh-link">L192</a><div class="mdoc"><p>Compute negative log likelihood(nll)</p>+<p>Normally, this function is called in batchify_nll.</p>+<h4>Args:</h4>+<ul>+<li><code>text</code> — (Batch, Length)</li>+<li><code>punc</code> — (Batch, Length)</li>+<li><code>text_lengths</code> — (Batch,)</li>+<li><code>max_lengths</code> — int</li>+</ul>+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def nll(+ self,+ text: torch.Tensor,+ punc: torch.Tensor,+ text_lengths: torch.Tensor,+ punc_lengths: torch.Tensor,+ max_length: Optional[int] = None,+ vad_indexes: Optional[torch.Tensor] = None,+ vad_indexes_lengths: Optional[torch.Tensor] = None,+ ) -> Tuple[torch.Tensor, torch.Tensor]:+ """Compute negative log likelihood(nll)++ Normally, this function is called in batchify_nll.+ Args:+ text: (Batch, Length)+ punc: (Batch, Length)+ text_lengths: (Batch,)+ max_lengths: int+ """+ batch_size = text.size(0)+ # For data parallel+ if max_length is None:+ text = text[:, : text_lengths.max()]+ punc = punc[:, : text_lengths.max()]+ else:+ text = text[:, :max_length]+ punc = punc[:, :max_length]++ if self.with_vad():+ # Should be VadRealtimeTransformer+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/ct_transformer/model.py#L192" target="_blank" class="src-more">View full source →</a></details></div><div class="mblk"><code>.forward(text, punc, text_lengths, punc_lengths, vad_indexes, vad_indexes_lengths)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/ct_transformer/model.py#L262" target="_blank" class="gh-link">L262</a><div class="mdoc"><p>Forward pass for training.</p>+<h4>Args:</h4>+<ul>+<li><code>text</code> — Text tensor or string input.</li>+<li><code>punc</code> — TODO.</li>+<li><code>text_lengths</code> — Length of each text sample.</li>+<li><code>punc_lengths</code> — Lengths of punc.</li>+<li><code>vad_indexes</code> — TODO.</li>+<li><code>vad_indexes_lengths</code> — Lengths of vad_indexes.</li>+</ul>+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def forward(+ self,+ text: torch.Tensor,+ punc: torch.Tensor,+ text_lengths: torch.Tensor,+ punc_lengths: torch.Tensor,+ vad_indexes: Optional[torch.Tensor] = None,+ vad_indexes_lengths: Optional[torch.Tensor] = None,+ ):+ """Forward pass for training.++ Args:+ text: Text tensor or string input.+ punc: TODO.+ text_lengths: Length of each text sample.+ punc_lengths: Lengths of punc.+ vad_indexes: TODO.+ vad_indexes_lengths: Lengths of vad_indexes.+ """+ nll, y_lengths = self.nll(text, punc, text_lengths, punc_lengths, vad_indexes=vad_indexes)+ ntokens = y_lengths.sum()+ loss = nll.sum() / ntokens+ stats = dict(loss=loss.detach())++ # force_gatherable: to-device and to-tensor if scalar for DataParallel+ loss, stats, weight = force_gatherable((loss, stats, ntokens), loss.device)+ return loss, stats, weight+</code></pre></details></div><div class="mblk"><code>.inference(data_in, data_lengths, key, tokenizer, frontend, **kwargs)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/ct_transformer/model.py#L290" target="_blank" class="gh-link">L290</a><div class="mdoc"><p>Run inference on input data.</p>+<h4>Args:</h4>+<ul>+<li><code>data_in</code> — Input data (audio samples, file paths, or text).</li>+<li><code>data_lengths</code> — Lengths of each input sample in the batch.</li>+<li><code>key</code> — Sample identifiers.</li>+<li><code>tokenizer</code> — Tokenizer instance for text encoding/decoding.</li>+<li><code>frontend</code> — Audio frontend for feature extraction.</li>+<li><code>**kwargs</code> — Additional keyword arguments.</li>+</ul>+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def inference(+ self,+ data_in,+ data_lengths=None,+ key: list = None,+ tokenizer=None,+ frontend=None,+ **kwargs,+ ):+ """Run inference on input data.++ Args:+ data_in: Input data (audio samples, file paths, or text).+ data_lengths: Lengths of each input sample in the batch.+ key: Sample identifiers.+ tokenizer: Tokenizer instance for text encoding/decoding.+ frontend: Audio frontend for feature extraction.+ **kwargs: Additional keyword arguments.+ """+ assert len(data_in) == 1+ if not data_in[0] or (isinstance(data_in[0], str) and not data_in[0].strip()):+ meta_data = {"batch_data_time": -1}+ return [{"key": key[0] if key else "", "text": "", "punc_array": None}], meta_data+ text = load_audio_text_image_video(data_in, data_type=kwargs.get("kwargs", "text"))[0]+ vad_indexes = kwargs.get("vad_indexes", None)+ # text = data_in[0]+ # text_lengths = data_lengths[0] if data_lengths is not None else None+ split_size = kwargs.get("split_size", 20)++ tokens = split_words(text, jieba_usr_dict=self.jieba_usr_dict)+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/ct_transformer/model.py#L290" target="_blank" class="src-more">View full source →</a></details></div><div class="mblk"><code>.export(**kwargs)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/ct_transformer/model.py#L471" target="_blank" class="gh-link">L471</a><div class="mdoc"><p>Export.</p>+<h4>Args:</h4>+<ul>+<li><code>**kwargs</code> — Additional keyword arguments.</li>+</ul>+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def export(self, **kwargs):++ """Export.++ Args:+ **kwargs: Additional keyword arguments.+ """+ from .export_meta import export_rebuild_model++ models = export_rebuild_model(model=self, **kwargs)+ return models+</code></pre></details></div></div>+<div class="api-detail" id="e33"><span class="dtag badge-m">method</span><h2><code>CTTransformer.punc_forward(text, text_lengths, **kwargs)</code></h2><div class="dmod">funasr.models.ct_transformer.CTTransformer · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/ct_transformer/model.py#L113" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Compute loss value from buffer sequences.</p>+<h4>Args:</h4>+<ul>+<li><code>input (torch.Tensor)</code> — Input ids. (batch, len)</li>+<li><code>hidden (torch.Tensor)</code> — Target ids. (batch, len)</li>+</ul>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def punc_forward(self, text: torch.Tensor, text_lengths: torch.Tensor, **kwargs):+ """Compute loss value from buffer sequences.++ Args:+ input (torch.Tensor): Input ids. (batch, len)+ hidden (torch.Tensor): Target ids. (batch, len)++ """+ x = self.embed(text)+ # mask = self._target_mask(input)+ h, _, _ = self.encoder(x, text_lengths)+ y = self.decoder(h)+ return y, None+</code></pre></details></div>+<div class="api-detail" id="e34"><span class="dtag badge-m">method</span><h2><code>CTTransformer.with_vad()</code></h2><div class="dmod">funasr.models.ct_transformer.CTTransformer · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/ct_transformer/model.py#L127" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>With vad.</p>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def with_vad(self):+ """With vad."""+ return False+</code></pre></details></div>+<div class="api-detail" id="e35"><span class="dtag badge-m">method</span><h2><code>CTTransformer.score(y, state, x)</code></h2><div class="dmod">funasr.models.ct_transformer.CTTransformer · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/ct_transformer/model.py#L131" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Score new token.</p>+<h4>Args:</h4>+<ul>+<li><code>y (torch.Tensor)</code> — 1D torch.int64 prefix tokens.</li>+<li><code>state</code> — Scorer state for prefix tokens</li>+<li><code>x (torch.Tensor)</code> — encoder feature that generates ys.</li>+</ul>+<h4>Returns:</h4>+<p>tuple[torch.Tensor, Any]: Tuple of</p>+<p>torch.float32 scores for next token (vocab_size)</p>+<p>and next state for ys</p>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def score(self, y: torch.Tensor, state: Any, x: torch.Tensor) -> Tuple[torch.Tensor, Any]:+ """Score new token.++ Args:+ y (torch.Tensor): 1D torch.int64 prefix tokens.+ state: Scorer state for prefix tokens+ x (torch.Tensor): encoder feature that generates ys.++ Returns:+ tuple[torch.Tensor, Any]: Tuple of+ torch.float32 scores for next token (vocab_size)+ and next state for ys++ """+ y = y.unsqueeze(0)+ h, _, cache = self.encoder.forward_one_step(+ self.embed(y), self._target_mask(y), cache=state+ )+ h = self.decoder(h[:, -1])+ logp = h.log_softmax(dim=-1).squeeze(0)+ return logp, cache+</code></pre></details></div>+<div class="api-detail" id="e36"><span class="dtag badge-m">method</span><h2><code>CTTransformer.batch_score(ys, states, xs)</code></h2><div class="dmod">funasr.models.ct_transformer.CTTransformer · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/ct_transformer/model.py#L153" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Score new token batch.</p>+<h4>Args:</h4>+<ul>+<li><code>ys (torch.Tensor)</code> — torch.int64 prefix tokens (n_batch, ylen).</li>+<li><code>states (List[Any])</code> — Scorer states for prefix tokens.</li>+</ul>+<p>xs (torch.Tensor):</p>+<p>The encoder feature that generates ys (n_batch, xlen, n_feat).</p>+<h4>Returns:</h4>+<p>tuple[torch.Tensor, List[Any]]: Tuple of</p>+<p>batchfied scores for next token with shape of `(n_batch, vocab_size)`</p>+<p>and next state list for ys.</p>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def batch_score(+ self, ys: torch.Tensor, states: List[Any], xs: torch.Tensor+ ) -> Tuple[torch.Tensor, List[Any]]:+ """Score new token batch.++ Args:+ ys (torch.Tensor): torch.int64 prefix tokens (n_batch, ylen).+ states (List[Any]): Scorer states for prefix tokens.+ xs (torch.Tensor):+ The encoder feature that generates ys (n_batch, xlen, n_feat).++ Returns:+ tuple[torch.Tensor, List[Any]]: Tuple of+ batchfied scores for next token with shape of `(n_batch, vocab_size)`+ and next state list for ys.++ """+ # merge states+ n_batch = len(ys)+ n_layers = len(self.encoder.encoders)+ if states[0] is None:+ batch_state = None+ else:+ # transpose state of [batch, layer] into [layer, batch]+ batch_state = [+ torch.stack([states[b][i] for b in range(n_batch)]) for i in range(n_layers)+ ]++ # batch decoding+ h, _, states = self.encoder.forward_one_step(+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/ct_transformer/model.py#L153" target="_blank" class="src-more">View full source on GitHub →</a></details></div>+<div class="api-detail" id="e37"><span class="dtag badge-m">method</span><h2><code>CTTransformer.nll(text, punc, text_lengths, punc_lengths, max_length, vad_indexes, vad_indexes_lengths)</code></h2><div class="dmod">funasr.models.ct_transformer.CTTransformer · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/ct_transformer/model.py#L192" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Compute negative log likelihood(nll)</p>+<p>Normally, this function is called in batchify_nll.</p>+<h4>Args:</h4>+<ul>+<li><code>text</code> — (Batch, Length)</li>+<li><code>punc</code> — (Batch, Length)</li>+<li><code>text_lengths</code> — (Batch,)</li>+<li><code>max_lengths</code> — int</li>+</ul>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def nll(+ self,+ text: torch.Tensor,+ punc: torch.Tensor,+ text_lengths: torch.Tensor,+ punc_lengths: torch.Tensor,+ max_length: Optional[int] = None,+ vad_indexes: Optional[torch.Tensor] = None,+ vad_indexes_lengths: Optional[torch.Tensor] = None,+ ) -> Tuple[torch.Tensor, torch.Tensor]:+ """Compute negative log likelihood(nll)++ Normally, this function is called in batchify_nll.+ Args:+ text: (Batch, Length)+ punc: (Batch, Length)+ text_lengths: (Batch,)+ max_lengths: int+ """+ batch_size = text.size(0)+ # For data parallel+ if max_length is None:+ text = text[:, : text_lengths.max()]+ punc = punc[:, : text_lengths.max()]+ else:+ text = text[:, :max_length]+ punc = punc[:, :max_length]++ if self.with_vad():+ # Should be VadRealtimeTransformer+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/ct_transformer/model.py#L192" target="_blank" class="src-more">View full source on GitHub →</a></details></div>+<div class="api-detail" id="e38"><span class="dtag badge-m">method</span><h2><code>CTTransformer.forward(text, punc, text_lengths, punc_lengths, vad_indexes, vad_indexes_lengths)</code></h2><div class="dmod">funasr.models.ct_transformer.CTTransformer · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/ct_transformer/model.py#L262" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Forward pass for training.</p>+<h4>Args:</h4>+<ul>+<li><code>text</code> — Text tensor or string input.</li>+<li><code>punc</code> — TODO.</li>+<li><code>text_lengths</code> — Length of each text sample.</li>+<li><code>punc_lengths</code> — Lengths of punc.</li>+<li><code>vad_indexes</code> — TODO.</li>+<li><code>vad_indexes_lengths</code> — Lengths of vad_indexes.</li>+</ul>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def forward(+ self,+ text: torch.Tensor,+ punc: torch.Tensor,+ text_lengths: torch.Tensor,+ punc_lengths: torch.Tensor,+ vad_indexes: Optional[torch.Tensor] = None,+ vad_indexes_lengths: Optional[torch.Tensor] = None,+ ):+ """Forward pass for training.++ Args:+ text: Text tensor or string input.+ punc: TODO.+ text_lengths: Length of each text sample.+ punc_lengths: Lengths of punc.+ vad_indexes: TODO.+ vad_indexes_lengths: Lengths of vad_indexes.+ """+ nll, y_lengths = self.nll(text, punc, text_lengths, punc_lengths, vad_indexes=vad_indexes)+ ntokens = y_lengths.sum()+ loss = nll.sum() / ntokens+ stats = dict(loss=loss.detach())++ # force_gatherable: to-device and to-tensor if scalar for DataParallel+ loss, stats, weight = force_gatherable((loss, stats, ntokens), loss.device)+ return loss, stats, weight+</code></pre></details></div>+<div class="api-detail" id="e39"><span class="dtag badge-m">method</span><h2><code>CTTransformer.inference(data_in, data_lengths, key, tokenizer, frontend, **kwargs)</code></h2><div class="dmod">funasr.models.ct_transformer.CTTransformer · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/ct_transformer/model.py#L290" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Run inference on input data.</p>+<h4>Args:</h4>+<ul>+<li><code>data_in</code> — Input data (audio samples, file paths, or text).</li>+<li><code>data_lengths</code> — Lengths of each input sample in the batch.</li>+<li><code>key</code> — Sample identifiers.</li>+<li><code>tokenizer</code> — Tokenizer instance for text encoding/decoding.</li>+<li><code>frontend</code> — Audio frontend for feature extraction.</li>+<li><code>**kwargs</code> — Additional keyword arguments.</li>+</ul>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def inference(+ self,+ data_in,+ data_lengths=None,+ key: list = None,+ tokenizer=None,+ frontend=None,+ **kwargs,+ ):+ """Run inference on input data.++ Args:+ data_in: Input data (audio samples, file paths, or text).+ data_lengths: Lengths of each input sample in the batch.+ key: Sample identifiers.+ tokenizer: Tokenizer instance for text encoding/decoding.+ frontend: Audio frontend for feature extraction.+ **kwargs: Additional keyword arguments.+ """+ assert len(data_in) == 1+ if not data_in[0] or (isinstance(data_in[0], str) and not data_in[0].strip()):+ meta_data = {"batch_data_time": -1}+ return [{"key": key[0] if key else "", "text": "", "punc_array": None}], meta_data+ text = load_audio_text_image_video(data_in, data_type=kwargs.get("kwargs", "text"))[0]+ vad_indexes = kwargs.get("vad_indexes", None)+ # text = data_in[0]+ # text_lengths = data_lengths[0] if data_lengths is not None else None+ split_size = kwargs.get("split_size", 20)++ tokens = split_words(text, jieba_usr_dict=self.jieba_usr_dict)+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/ct_transformer/model.py#L290" target="_blank" class="src-more">View full source on GitHub →</a></details></div>+<div class="api-detail" id="e40"><span class="dtag badge-m">method</span><h2><code>CTTransformer.export(**kwargs)</code></h2><div class="dmod">funasr.models.ct_transformer.CTTransformer · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/ct_transformer/model.py#L471" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Export.</p>+<h4>Args:</h4>+<ul>+<li><code>**kwargs</code> — Additional keyword arguments.</li>+</ul>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def export(self, **kwargs):++ """Export.++ Args:+ **kwargs: Additional keyword arguments.+ """+ from .export_meta import export_rebuild_model++ models = export_rebuild_model(model=self, **kwargs)+ return models+</code></pre></details></div>+<div class="api-detail" id="e41"><span class="dtag badge-c">class</span><h2>CTTransformerStreaming</h2><div class="dmod">funasr.models.ct_transformer_streaming · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/ct_transformer_streaming/model.py#L33" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><ul>+<li><code>CT</code> — Transformer Streaming: Online punctuation restoration.</li>+</ul>+<p>Processes text incrementally with a sliding window, maintaining cache</p>+<p>of previous context for consistent punctuation decisions across chunks.</p>+<p>Used as punc_model in streaming ASR pipelines.</p>+<p>Supports VAD-aware punctuation: uses VAD boundaries to improve sentence segmentation.</p>+<ul>+<li><code>Reference</code> — https://arxiv.org/pdf/2003.01309.pdf</li>+<li><code>Output</code> — {"key": str, "text": str, "punc_array": Tensor}</li>+<li><code>Author</code> — Speech Lab of DAMO Academy, Alibaba Group</li>+</ul>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code>class CTTransformerStreaming(CTTransformer):+ """CT-Transformer Streaming: Online punctuation restoration.++ Processes text incrementally with a sliding window, maintaining cache+ of previous context for consistent punctuation decisions across chunks.+ Used as punc_model in streaming ASR pipelines.++ Supports VAD-aware punctuation: uses VAD boundaries to improve sentence segmentation.++ Reference: https://arxiv.org/pdf/2003.01309.pdf+ Output: {"key": str, "text": str, "punc_array": Tensor}++ Author: Speech Lab of DAMO Academy, Alibaba Group+ """++ def __init__(+ self,+ *args,+ **kwargs,+ ):+ """Initialize CTTransformerStreaming.++ Args:+ *args: Variable positional arguments.+ **kwargs: Additional keyword arguments.+ """+ super().__init__(*args, **kwargs)++ def punc_forward(+ self, text: torch.Tensor, text_lengths: torch.Tensor, vad_indexes: torch.Tensor, **kwargs+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/ct_transformer_streaming/model.py#L33" target="_blank" class="src-more">View full source on GitHub →</a></details><h3 class="mt">Methods</h3><div class="mblk"><code>.punc_forward(text, text_lengths, vad_indexes, **kwargs)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/ct_transformer_streaming/model.py#L61" target="_blank" class="gh-link">L61</a><div class="mdoc"><p>Compute loss value from buffer sequences.</p>+<h4>Args:</h4>+<ul>+<li><code>input (torch.Tensor)</code> — Input ids. (batch, len)</li>+<li><code>hidden (torch.Tensor)</code> — Target ids. (batch, len)</li>+</ul>+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def punc_forward(+ self, text: torch.Tensor, text_lengths: torch.Tensor, vad_indexes: torch.Tensor, **kwargs+ ):+ """Compute loss value from buffer sequences.++ Args:+ input (torch.Tensor): Input ids. (batch, len)+ hidden (torch.Tensor): Target ids. (batch, len)++ """+ x = self.embed(text)+ # mask = self._target_mask(input)+ h, _, _ = self.encoder(x, text_lengths, vad_indexes=vad_indexes)+ y = self.decoder(h)+ return y, None+</code></pre></details></div><div class="mblk"><code>.with_vad()</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/ct_transformer_streaming/model.py#L77" target="_blank" class="gh-link">L77</a><div class="mdoc"><p>With vad.</p>+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def with_vad(self):+ """With vad."""+ return True+</code></pre></details></div><div class="mblk"><code>.inference(data_in, data_lengths, key, tokenizer, frontend, cache, **kwargs)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/ct_transformer_streaming/model.py#L81" target="_blank" class="gh-link">L81</a><div class="mdoc"><p>Run inference on input data.</p>+<h4>Args:</h4>+<ul>+<li><code>data_in</code> — Input data (audio samples, file paths, or text).</li>+<li><code>data_lengths</code> — Lengths of each input sample in the batch.</li>+<li><code>key</code> — Sample identifiers.</li>+<li><code>tokenizer</code> — Tokenizer instance for text encoding/decoding.</li>+<li><code>frontend</code> — Audio frontend for feature extraction.</li>+<li><code>cache</code> — State cache dict for streaming inference.</li>+<li><code>**kwargs</code> — Additional keyword arguments.</li>+</ul>+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def inference(+ self,+ data_in,+ data_lengths=None,+ key: list = None,+ tokenizer=None,+ frontend=None,+ cache: dict = None,+ **kwargs,+ ):+ """Run inference on input data.++ Args:+ data_in: Input data (audio samples, file paths, or text).+ data_lengths: Lengths of each input sample in the batch.+ key: Sample identifiers.+ tokenizer: Tokenizer instance for text encoding/decoding.+ frontend: Audio frontend for feature extraction.+ cache: State cache dict for streaming inference.+ **kwargs: Additional keyword arguments.+ """+ if cache is None:+ cache = {}+ assert len(data_in) == 1++ if len(cache) == 0:+ cache["pre_text"] = []+ text = load_audio_text_image_video(data_in, data_type=kwargs.get("kwargs", "text"))[0]+ text = "".join(cache["pre_text"]) + " " + text++</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/ct_transformer_streaming/model.py#L81" target="_blank" class="src-more">View full source →</a></details></div><div class="mblk"><code>.export(**kwargs)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/ct_transformer_streaming/model.py#L221" target="_blank" class="gh-link">L221</a><div class="mdoc"><p>Export.</p>+<h4>Args:</h4>+<ul>+<li><code>**kwargs</code> — Additional keyword arguments.</li>+</ul>+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def export(self, **kwargs):++ """Export.++ Args:+ **kwargs: Additional keyword arguments.+ """+ from .export_meta import export_rebuild_model++ models = export_rebuild_model(model=self, **kwargs)+ return models+</code></pre></details></div></div>+<div class="api-detail" id="e42"><span class="dtag badge-m">method</span><h2><code>CTTransformerStreaming.punc_forward(text, text_lengths, vad_indexes, **kwargs)</code></h2><div class="dmod">funasr.models.ct_transformer_streaming.CTTransformerStreaming · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/ct_transformer_streaming/model.py#L61" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Compute loss value from buffer sequences.</p>+<h4>Args:</h4>+<ul>+<li><code>input (torch.Tensor)</code> — Input ids. (batch, len)</li>+<li><code>hidden (torch.Tensor)</code> — Target ids. (batch, len)</li>+</ul>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def punc_forward(+ self, text: torch.Tensor, text_lengths: torch.Tensor, vad_indexes: torch.Tensor, **kwargs+ ):+ """Compute loss value from buffer sequences.++ Args:+ input (torch.Tensor): Input ids. (batch, len)+ hidden (torch.Tensor): Target ids. (batch, len)++ """+ x = self.embed(text)+ # mask = self._target_mask(input)+ h, _, _ = self.encoder(x, text_lengths, vad_indexes=vad_indexes)+ y = self.decoder(h)+ return y, None+</code></pre></details></div>+<div class="api-detail" id="e43"><span class="dtag badge-m">method</span><h2><code>CTTransformerStreaming.with_vad()</code></h2><div class="dmod">funasr.models.ct_transformer_streaming.CTTransformerStreaming · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/ct_transformer_streaming/model.py#L77" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>With vad.</p>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def with_vad(self):+ """With vad."""+ return True+</code></pre></details></div>+<div class="api-detail" id="e44"><span class="dtag badge-m">method</span><h2><code>CTTransformerStreaming.inference(data_in, data_lengths, key, tokenizer, frontend, cache, **kwargs)</code></h2><div class="dmod">funasr.models.ct_transformer_streaming.CTTransformerStreaming · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/ct_transformer_streaming/model.py#L81" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Run inference on input data.</p>+<h4>Args:</h4>+<ul>+<li><code>data_in</code> — Input data (audio samples, file paths, or text).</li>+<li><code>data_lengths</code> — Lengths of each input sample in the batch.</li>+<li><code>key</code> — Sample identifiers.</li>+<li><code>tokenizer</code> — Tokenizer instance for text encoding/decoding.</li>+<li><code>frontend</code> — Audio frontend for feature extraction.</li>+<li><code>cache</code> — State cache dict for streaming inference.</li>+<li><code>**kwargs</code> — Additional keyword arguments.</li>+</ul>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def inference(+ self,+ data_in,+ data_lengths=None,+ key: list = None,+ tokenizer=None,+ frontend=None,+ cache: dict = None,+ **kwargs,+ ):+ """Run inference on input data.++ Args:+ data_in: Input data (audio samples, file paths, or text).+ data_lengths: Lengths of each input sample in the batch.+ key: Sample identifiers.+ tokenizer: Tokenizer instance for text encoding/decoding.+ frontend: Audio frontend for feature extraction.+ cache: State cache dict for streaming inference.+ **kwargs: Additional keyword arguments.+ """+ if cache is None:+ cache = {}+ assert len(data_in) == 1++ if len(cache) == 0:+ cache["pre_text"] = []+ text = load_audio_text_image_video(data_in, data_type=kwargs.get("kwargs", "text"))[0]+ text = "".join(cache["pre_text"]) + " " + text++</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/ct_transformer_streaming/model.py#L81" target="_blank" class="src-more">View full source on GitHub →</a></details></div>+<div class="api-detail" id="e45"><span class="dtag badge-m">method</span><h2><code>CTTransformerStreaming.export(**kwargs)</code></h2><div class="dmod">funasr.models.ct_transformer_streaming.CTTransformerStreaming · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/ct_transformer_streaming/model.py#L221" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Export.</p>+<h4>Args:</h4>+<ul>+<li><code>**kwargs</code> — Additional keyword arguments.</li>+</ul>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def export(self, **kwargs):++ """Export.++ Args:+ **kwargs: Additional keyword arguments.+ """+ from .export_meta import export_rebuild_model++ models = export_rebuild_model(model=self, **kwargs)+ return models+</code></pre></details></div>+<div class="api-detail" id="e46"><span class="dtag badge-c">class</span><h2>Transformer</h2><div class="dmod">funasr.models.ctc · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/ctc/model.py#L18" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><ul>+<li><code>CTC</code> — attention hybrid Encoder-Decoder model</li>+</ul>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code>class Transformer(nn.Module):+ """CTC-attention hybrid Encoder-Decoder model"""++ def __init__(+ self,+ specaug: str = None,+ specaug_conf: dict = None,+ normalize: str = None,+ normalize_conf: dict = None,+ encoder: str = None,+ encoder_conf: dict = None,+ ctc_conf: dict = None,+ input_size: int = 80,+ vocab_size: int = -1,+ ignore_id: int = -1,+ blank_id: int = 0,+ sos: int = 1,+ eos: int = 2,+ length_normalized_loss: bool = False,+ **kwargs,+ ):++ """Initialize Transformer.++ Args:+ specaug: TODO.+ specaug_conf: Configuration dict for specaug.+ normalize: TODO.+ normalize_conf: Configuration dict for normalize.+ encoder: TODO.+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/ctc/model.py#L18" target="_blank" class="src-more">View full source on GitHub →</a></details><h3 class="mt">Methods</h3><div class="mblk"><code>.forward(speech, speech_lengths, text, text_lengths, **kwargs)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/ctc/model.py#L89" target="_blank" class="gh-link">L89</a><div class="mdoc"><p>Encoder + Decoder + Calc loss</p>+<h4>Args:</h4>+<ul>+<li><code>speech</code> — (Batch, Length, ...)</li>+<li><code>speech_lengths</code> — (Batch, )</li>+<li><code>text</code> — (Batch, Length)</li>+<li><code>text_lengths</code> — (Batch,)</li>+</ul>+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def forward(+ self,+ speech: torch.Tensor,+ speech_lengths: torch.Tensor,+ text: torch.Tensor,+ text_lengths: torch.Tensor,+ **kwargs,+ ) -> Tuple[torch.Tensor, Dict[str, torch.Tensor], torch.Tensor]:+ """Encoder + Decoder + Calc loss+ Args:+ speech: (Batch, Length, ...)+ speech_lengths: (Batch, )+ text: (Batch, Length)+ text_lengths: (Batch,)+ """+ # import pdb;+ # pdb.set_trace()+ if len(text_lengths.size()) > 1:+ text_lengths = text_lengths[:, 0]+ if len(speech_lengths.size()) > 1:+ speech_lengths = speech_lengths[:, 0]++ batch_size = speech.shape[0]++ # 1. Encoder+ encoder_out, encoder_out_lens = self.encode(speech, speech_lengths)++ loss_ctc, cer_ctc = None, None+ stats = dict()++</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/ctc/model.py#L89" target="_blank" class="src-more">View full source →</a></details></div><div class="mblk"><code>.encode(speech, speech_lengths, **kwargs)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/ctc/model.py#L134" target="_blank" class="gh-link">L134</a><div class="mdoc"><p>Frontend + Encoder. Note that this method is used by asr_inference.py</p>+<h4>Args:</h4>+<ul>+<li><code>speech</code> — (Batch, Length, ...)</li>+<li><code>speech_lengths</code> — (Batch, )</li>+<li><code>ind</code> — int</li>+</ul>+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def encode(+ self,+ speech: torch.Tensor,+ speech_lengths: torch.Tensor,+ **kwargs,+ ) -> Tuple[torch.Tensor, torch.Tensor]:+ """Frontend + Encoder. Note that this method is used by asr_inference.py+ Args:+ speech: (Batch, Length, ...)+ speech_lengths: (Batch, )+ ind: int+ """++ # Data augmentation+ if self.specaug is not None and self.training:+ speech, speech_lengths = self.specaug(speech, speech_lengths)++ # Normalization for feature: e.g. Global-CMVN, Utterance-CMVN+ if self.normalize is not None:+ speech, speech_lengths = self.normalize(speech, speech_lengths)++ # Forward encoder+ # feats: (Batch, Length, Dim)+ # -> encoder_out: (Batch, Length2, Dim2)+ encoder_out, encoder_out_lens = self.encoder(speech, speech_lengths)++ return encoder_out, encoder_out_lens+</code></pre></details></div><div class="mblk"><code>.inference(data_in, data_lengths, key, tokenizer, frontend, **kwargs)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/ctc/model.py#L189" target="_blank" class="gh-link">L189</a><div class="mdoc"><p>Run inference on input data.</p>+<h4>Args:</h4>+<ul>+<li><code>data_in</code> — Input data (audio samples, file paths, or text).</li>+<li><code>data_lengths</code> — Lengths of each input sample in the batch.</li>+<li><code>key</code> — Sample identifiers.</li>+<li><code>tokenizer</code> — Tokenizer instance for text encoding/decoding.</li>+<li><code>frontend</code> — Audio frontend for feature extraction.</li>+<li><code>**kwargs</code> — Additional keyword arguments.</li>+</ul>+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def inference(+ self,+ data_in,+ data_lengths=None,+ key: list = None,+ tokenizer=None,+ frontend=None,+ **kwargs,+ ):++ """Run inference on input data.++ Args:+ data_in: Input data (audio samples, file paths, or text).+ data_lengths: Lengths of each input sample in the batch.+ key: Sample identifiers.+ tokenizer: Tokenizer instance for text encoding/decoding.+ frontend: Audio frontend for feature extraction.+ **kwargs: Additional keyword arguments.+ """+ if kwargs.get("batch_size", 1) > 1:+ raise NotImplementedError("batch decoding is not implemented")++ meta_data = {}+ if (+ isinstance(data_in, torch.Tensor) and kwargs.get("data_type", "sound") == "fbank"+ ): # fbank+ speech, speech_lengths = data_in, data_lengths+ if len(speech.shape) < 3:+ speech = speech[None, :, :]+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/ctc/model.py#L189" target="_blank" class="src-more">View full source →</a></details></div></div>+<div class="api-detail" id="e47"><span class="dtag badge-m">method</span><h2><code>Transformer.forward(speech, speech_lengths, text, text_lengths, **kwargs)</code></h2><div class="dmod">funasr.models.ctc.Transformer · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/ctc/model.py#L89" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Encoder + Decoder + Calc loss</p>+<h4>Args:</h4>+<ul>+<li><code>speech</code> — (Batch, Length, ...)</li>+<li><code>speech_lengths</code> — (Batch, )</li>+<li><code>text</code> — (Batch, Length)</li>+<li><code>text_lengths</code> — (Batch,)</li>+</ul>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def forward(+ self,+ speech: torch.Tensor,+ speech_lengths: torch.Tensor,+ text: torch.Tensor,+ text_lengths: torch.Tensor,+ **kwargs,+ ) -> Tuple[torch.Tensor, Dict[str, torch.Tensor], torch.Tensor]:+ """Encoder + Decoder + Calc loss+ Args:+ speech: (Batch, Length, ...)+ speech_lengths: (Batch, )+ text: (Batch, Length)+ text_lengths: (Batch,)+ """+ # import pdb;+ # pdb.set_trace()+ if len(text_lengths.size()) > 1:+ text_lengths = text_lengths[:, 0]+ if len(speech_lengths.size()) > 1:+ speech_lengths = speech_lengths[:, 0]++ batch_size = speech.shape[0]++ # 1. Encoder+ encoder_out, encoder_out_lens = self.encode(speech, speech_lengths)++ loss_ctc, cer_ctc = None, None+ stats = dict()++</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/ctc/model.py#L89" target="_blank" class="src-more">View full source on GitHub →</a></details></div>+<div class="api-detail" id="e48"><span class="dtag badge-m">method</span><h2><code>Transformer.encode(speech, speech_lengths, **kwargs)</code></h2><div class="dmod">funasr.models.ctc.Transformer · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/ctc/model.py#L134" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Frontend + Encoder. Note that this method is used by asr_inference.py</p>+<h4>Args:</h4>+<ul>+<li><code>speech</code> — (Batch, Length, ...)</li>+<li><code>speech_lengths</code> — (Batch, )</li>+<li><code>ind</code> — int</li>+</ul>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def encode(+ self,+ speech: torch.Tensor,+ speech_lengths: torch.Tensor,+ **kwargs,+ ) -> Tuple[torch.Tensor, torch.Tensor]:+ """Frontend + Encoder. Note that this method is used by asr_inference.py+ Args:+ speech: (Batch, Length, ...)+ speech_lengths: (Batch, )+ ind: int+ """++ # Data augmentation+ if self.specaug is not None and self.training:+ speech, speech_lengths = self.specaug(speech, speech_lengths)++ # Normalization for feature: e.g. Global-CMVN, Utterance-CMVN+ if self.normalize is not None:+ speech, speech_lengths = self.normalize(speech, speech_lengths)++ # Forward encoder+ # feats: (Batch, Length, Dim)+ # -> encoder_out: (Batch, Length2, Dim2)+ encoder_out, encoder_out_lens = self.encoder(speech, speech_lengths)++ return encoder_out, encoder_out_lens+</code></pre></details></div>+<div class="api-detail" id="e49"><span class="dtag badge-m">method</span><h2><code>Transformer.inference(data_in, data_lengths, key, tokenizer, frontend, **kwargs)</code></h2><div class="dmod">funasr.models.ctc.Transformer · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/ctc/model.py#L189" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Run inference on input data.</p>+<h4>Args:</h4>+<ul>+<li><code>data_in</code> — Input data (audio samples, file paths, or text).</li>+<li><code>data_lengths</code> — Lengths of each input sample in the batch.</li>+<li><code>key</code> — Sample identifiers.</li>+<li><code>tokenizer</code> — Tokenizer instance for text encoding/decoding.</li>+<li><code>frontend</code> — Audio frontend for feature extraction.</li>+<li><code>**kwargs</code> — Additional keyword arguments.</li>+</ul>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def inference(+ self,+ data_in,+ data_lengths=None,+ key: list = None,+ tokenizer=None,+ frontend=None,+ **kwargs,+ ):++ """Run inference on input data.++ Args:+ data_in: Input data (audio samples, file paths, or text).+ data_lengths: Lengths of each input sample in the batch.+ key: Sample identifiers.+ tokenizer: Tokenizer instance for text encoding/decoding.+ frontend: Audio frontend for feature extraction.+ **kwargs: Additional keyword arguments.+ """+ if kwargs.get("batch_size", 1) > 1:+ raise NotImplementedError("batch decoding is not implemented")++ meta_data = {}+ if (+ isinstance(data_in, torch.Tensor) and kwargs.get("data_type", "sound") == "fbank"+ ): # fbank+ speech, speech_lengths = data_in, data_lengths+ if len(speech.shape) < 3:+ speech = speech[None, :, :]+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/ctc/model.py#L189" target="_blank" class="src-more">View full source on GitHub →</a></details></div>+<div class="api-detail" id="e50"><span class="dtag badge-c">class</span><h2>EBranchformer</h2><div class="dmod">funasr.models.e_branchformer · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/e_branchformer/model.py#L8" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><ul>+<li><code>E</code> — Branchformer: Enhanced Branchformer with improved merging.</li>+</ul>+<p>Uses element-wise merging instead of concatenation for parallel branches,</p>+<p>resulting in better parameter efficiency.</p>+<p>Inherits Transformer pipeline.</p>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code>class EBranchformer(Transformer):+ """E-Branchformer: Enhanced Branchformer with improved merging.++ Uses element-wise merging instead of concatenation for parallel branches,+ resulting in better parameter efficiency.++ Inherits Transformer pipeline.+ """++ def __init__(+ self,+ *args,+ **kwargs,+ ):++ """Initialize EBranchformer.++ Args:+ *args: Variable positional arguments.+ **kwargs: Additional keyword arguments.+ """+ super().__init__(*args, **kwargs)+</code></pre></details></div>+<div class="api-detail" id="e51"><span class="dtag badge-c">class</span><h2>EParaformer</h2><div class="dmod">funasr.models.e_paraformer · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/e_paraformer/model.py#L31" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><ul>+<li><code>E</code> — Paraformer: Enhanced Paraformer with streaming support.</li>+</ul>+<p>Extended Paraformer supporting both offline and streaming modes</p>+<p>through dynamic masking in the encoder. Used for 2-pass decoding</p>+<p>where first pass provides streaming results and second pass refines.</p>+<ul>+<li><code>Output</code> — {"key": str, "text": str, "timestamp": [[start_ms, end_ms], ...]}</li>+<li><code>Author</code> — Speech Lab of DAMO Academy, Alibaba Group</li>+<li><code>Paraformer</code> — Fast and Accurate Parallel Transformer for Non-autoregressive End-to-End Speech Recognition</li>+</ul>+<p>https://arxiv.org/abs/2206.08317</p>+<ul>+<li><code>Author</code> — Kun Zou, [email protected]</li>+<li><code>E</code> — Paraformer: A Faster and Better Parallel Transformer for Non-autoregressive End-to-End Mandarin Speech Recognition</li>+</ul>+<p>https://www.isca-archive.org/interspeech_2024/zou24_interspeech.pdf</p>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code>class EParaformer(torch.nn.Module):+ """E-Paraformer: Enhanced Paraformer with streaming support.++ Extended Paraformer supporting both offline and streaming modes+ through dynamic masking in the encoder. Used for 2-pass decoding+ where first pass provides streaming results and second pass refines.++ Output: {"key": str, "text": str, "timestamp": [[start_ms, end_ms], ...]}++ Author: Speech Lab of DAMO Academy, Alibaba Group+ Paraformer: Fast and Accurate Parallel Transformer for Non-autoregressive End-to-End Speech Recognition+ https://arxiv.org/abs/2206.08317+ Author: Kun Zou, [email protected]+ E-Paraformer: A Faster and Better Parallel Transformer for Non-autoregressive End-to-End Mandarin Speech Recognition+ https://www.isca-archive.org/interspeech_2024/zou24_interspeech.pdf+ """++ def __init__(+ self,+ specaug: Optional[str] = None,+ specaug_conf: Optional[Dict] = None,+ normalize: str = None,+ normalize_conf: Optional[Dict] = None,+ encoder: str = None,+ encoder_conf: Optional[Dict] = None,+ decoder: str = None,+ decoder_conf: Optional[Dict] = None,+ ctc: str = None,+ ctc_conf: Optional[Dict] = None,+ predictor: str = None,+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/e_paraformer/model.py#L31" target="_blank" class="src-more">View full source on GitHub →</a></details><h3 class="mt">Methods</h3><div class="mblk"><code>.forward(speech, speech_lengths, text, text_lengths, **kwargs)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/e_paraformer/model.py#L220" target="_blank" class="gh-link">L220</a><div class="mdoc"><p>Encoder + Decoder + Calc loss</p>+<h4>Args:</h4>+<ul>+<li><code>speech</code> — (Batch, Length, ...)</li>+<li><code>speech_lengths</code> — (Batch, )</li>+<li><code>text</code> — (Batch, Length)</li>+<li><code>text_lengths</code> — (Batch,)</li>+</ul>+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def forward(+ self,+ speech: torch.Tensor,+ speech_lengths: torch.Tensor,+ text: torch.Tensor,+ text_lengths: torch.Tensor,+ **kwargs,+ ) -> Tuple[torch.Tensor, Dict[str, torch.Tensor], torch.Tensor]:+ """Encoder + Decoder + Calc loss+ Args:+ speech: (Batch, Length, ...)+ speech_lengths: (Batch, )+ text: (Batch, Length)+ text_lengths: (Batch,)+ """+ if len(text_lengths.size()) > 1:+ text_lengths = text_lengths[:, 0]+ if len(speech_lengths.size()) > 1:+ speech_lengths = speech_lengths[:, 0]++ batch_size = speech.shape[0]++ # Encoder+ encoder_out, encoder_out_lens = self.encode(speech, speech_lengths)++ loss_ctc, cer_ctc = None, None+ loss_pre = None+ stats = dict()++ # decoder: CTC branch+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/e_paraformer/model.py#L220" target="_blank" class="src-more">View full source →</a></details></div><div class="mblk"><code>.encode(speech, speech_lengths, **kwargs)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/e_paraformer/model.py#L292" target="_blank" class="gh-link">L292</a><div class="mdoc"><p>Encoder. Note that this method is used by asr_inference.py</p>+<h4>Args:</h4>+<ul>+<li><code>speech</code> — (Batch, Length, ...)</li>+<li><code>speech_lengths</code> — (Batch, )</li>+<li><code>ind</code> — int</li>+</ul>+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def encode(+ self,+ speech: torch.Tensor,+ speech_lengths: torch.Tensor,+ **kwargs,+ ) -> Tuple[torch.Tensor, torch.Tensor]:+ """Encoder. Note that this method is used by asr_inference.py+ Args:+ speech: (Batch, Length, ...)+ speech_lengths: (Batch, )+ ind: int+ """+ with autocast(False):++ # Data augmentation+ if self.specaug is not None and self.training:+ speech, speech_lengths = self.specaug(speech, speech_lengths)++ # Normalization for feature: e.g. Global-CMVN, Utterance-CMVN+ if self.normalize is not None:+ speech, speech_lengths = self.normalize(speech, speech_lengths)++ # Forward encoder+ encoder_out, encoder_out_lens, _ = self.encoder(speech, speech_lengths)+ if isinstance(encoder_out, tuple):+ encoder_out = encoder_out[0]++ return encoder_out, encoder_out_lens+</code></pre></details></div><div class="mblk"><code>.calc_predictor(encoder_out, encoder_out_lens)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/e_paraformer/model.py#L321" target="_blank" class="gh-link">L321</a><div class="mdoc"><p>Calc predictor.</p>+<h4>Args:</h4>+<ul>+<li><code>encoder_out</code> — Encoder output tensor.</li>+<li><code>encoder_out_lens</code> — Encoder output lengths.</li>+</ul>+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def calc_predictor(self, encoder_out, encoder_out_lens):++ """Calc predictor.++ Args:+ encoder_out: Encoder output tensor.+ encoder_out_lens: Encoder output lengths.+ """+ encoder_out_mask = (+ ~make_pad_mask(encoder_out_lens, maxlen=encoder_out.size(1))[:, None, :]+ ).to(encoder_out.device)+ pre_acoustic_embeds, pre_token_length, alphas, pre_peak_index = self.predictor(+ encoder_out, None, encoder_out_mask, ignore_id=self.ignore_id+ )+ return pre_acoustic_embeds, pre_token_length, alphas, pre_peak_index+</code></pre></details></div><div class="mblk"><code>.cal_decoder_with_predictor(encoder_out, encoder_out_lens, sematic_embeds, ys_pad_lens)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/e_paraformer/model.py#L337" target="_blank" class="gh-link">L337</a><div class="mdoc"><p>Cal decoder with predictor.</p>+<h4>Args:</h4>+<ul>+<li><code>encoder_out</code> — Encoder output tensor.</li>+<li><code>encoder_out_lens</code> — Encoder output lengths.</li>+<li><code>sematic_embeds</code> — TODO.</li>+<li><code>ys_pad_lens</code> — Lengths of ys_pad.</li>+</ul>+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def cal_decoder_with_predictor(+ self, encoder_out, encoder_out_lens, sematic_embeds, ys_pad_lens+ ):++ """Cal decoder with predictor.++ Args:+ encoder_out: Encoder output tensor.+ encoder_out_lens: Encoder output lengths.+ sematic_embeds: TODO.+ ys_pad_lens: Lengths of ys_pad.+ """+ decoder_outs = self.decoder(encoder_out, encoder_out_lens, sematic_embeds, ys_pad_lens)+ decoder_out = decoder_outs[0]+ decoder_out = torch.log_softmax(decoder_out, dim=-1)+ return decoder_out, ys_pad_lens+</code></pre></details></div><div class="mblk"><code>.sampler(encoder_out, encoder_out_lens, ys_pad, ys_pad_lens, pre_acoustic_embeds)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/e_paraformer/model.py#L425" target="_blank" class="gh-link">L425</a><div class="mdoc"><p>Sampler.</p>+<h4>Args:</h4>+<ul>+<li><code>encoder_out</code> — Encoder output tensor.</li>+<li><code>encoder_out_lens</code> — Encoder output lengths.</li>+<li><code>ys_pad</code> — TODO.</li>+<li><code>ys_pad_lens</code> — Lengths of ys_pad.</li>+<li><code>pre_acoustic_embeds</code> — TODO.</li>+</ul>+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def sampler(self, encoder_out, encoder_out_lens, ys_pad, ys_pad_lens, pre_acoustic_embeds):++ """Sampler.++ Args:+ encoder_out: Encoder output tensor.+ encoder_out_lens: Encoder output lengths.+ ys_pad: TODO.+ ys_pad_lens: Lengths of ys_pad.+ pre_acoustic_embeds: TODO.+ """+ tgt_mask = (~make_pad_mask(ys_pad_lens, maxlen=ys_pad_lens.max())[:, :, None]).to(+ ys_pad.device+ )+ ys_pad_masked = ys_pad * tgt_mask[:, :, 0]+ if self.share_embedding:+ ys_pad_embed = self.decoder.output_layer.weight[ys_pad_masked]+ else:+ ys_pad_embed = self.decoder.embed(ys_pad_masked)+ with torch.no_grad():+ decoder_outs = self.decoder(+ encoder_out, encoder_out_lens, pre_acoustic_embeds, ys_pad_lens+ )+ decoder_out, _ = decoder_outs[0], decoder_outs[1]+ pred_tokens = decoder_out.argmax(-1)+ nonpad_positions = ys_pad.ne(self.ignore_id)+ seq_lens = (nonpad_positions).sum(1)+ same_num = ((pred_tokens == ys_pad) & nonpad_positions).sum(1)+ input_mask = torch.ones_like(nonpad_positions)+ bsz, seq_len = ys_pad.size()+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/e_paraformer/model.py#L425" target="_blank" class="src-more">View full source →</a></details></div><div class="mblk"><code>.sampler_with_grad(encoder_out, encoder_out_lens, ys_pad, ys_pad_lens, pre_acoustic_embeds)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/e_paraformer/model.py#L474" target="_blank" class="gh-link">L474</a><div class="mdoc"><p>Sampler with grad.</p>+<h4>Args:</h4>+<ul>+<li><code>encoder_out</code> — Encoder output tensor.</li>+<li><code>encoder_out_lens</code> — Encoder output lengths.</li>+<li><code>ys_pad</code> — TODO.</li>+<li><code>ys_pad_lens</code> — Lengths of ys_pad.</li>+<li><code>pre_acoustic_embeds</code> — TODO.</li>+</ul>+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def sampler_with_grad(self, encoder_out, encoder_out_lens, ys_pad, ys_pad_lens, pre_acoustic_embeds):++ """Sampler with grad.++ Args:+ encoder_out: Encoder output tensor.+ encoder_out_lens: Encoder output lengths.+ ys_pad: TODO.+ ys_pad_lens: Lengths of ys_pad.+ pre_acoustic_embeds: TODO.+ """+ tgt_mask = (~make_pad_mask(ys_pad_lens, maxlen=ys_pad_lens.max())[:, :, None]).to(+ ys_pad.device+ )+ ys_pad_masked = ys_pad * tgt_mask[:, :, 0]+ if self.share_embedding:+ ys_pad_embed = self.decoder.output_layer.weight[ys_pad_masked]+ else:+ ys_pad_embed = self.decoder.embed(ys_pad_masked)+ decoder_outs = self.decoder(+ encoder_out, encoder_out_lens, pre_acoustic_embeds, ys_pad_lens+ )+ decoder_out, _ = decoder_outs[0], decoder_outs[1]+ pred_tokens = decoder_out.argmax(-1)+ nonpad_positions = ys_pad.ne(self.ignore_id)+ seq_lens = (nonpad_positions).sum(1)+ same_num = ((pred_tokens == ys_pad) & nonpad_positions).sum(1)+ input_mask = torch.ones_like(nonpad_positions)+ bsz, seq_len = ys_pad.size()+ for li in range(bsz):+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/e_paraformer/model.py#L474" target="_blank" class="src-more">View full source →</a></details></div><div class="mblk"><code>.init_beam_search(**kwargs)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/e_paraformer/model.py#L548" target="_blank" class="gh-link">L548</a><div class="mdoc"><p>Init beam search.</p>+<h4>Args:</h4>+<ul>+<li><code>**kwargs</code> — Additional keyword arguments.</li>+</ul>+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def init_beam_search(+ self,+ **kwargs,+ ):+ """Init beam search.++ Args:+ **kwargs: Additional keyword arguments.+ """+ from funasr.models.paraformer.search import BeamSearchPara+ from funasr.models.transformer.scorers.ctc import CTCPrefixScorer+ from funasr.models.transformer.scorers.length_bonus import LengthBonus++ # 1. Build ASR model+ scorers = {}++ if self.ctc != None:+ ctc = CTCPrefixScorer(ctc=self.ctc, eos=self.eos)+ scorers.update(ctc=ctc)+ token_list = kwargs.get("token_list")+ scorers.update(+ length_bonus=LengthBonus(len(token_list)),+ )++ # 3. Build ngram model+ # ngram is not supported now+ ngram = None+ scorers["ngram"] = ngram++ weights = dict(+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/e_paraformer/model.py#L548" target="_blank" class="src-more">View full source →</a></details></div><div class="mblk"><code>.inference(data_in, data_lengths, key, tokenizer, frontend, **kwargs)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/e_paraformer/model.py#L600" target="_blank" class="gh-link">L600</a><div class="mdoc"><p>Run inference on input data.</p>+<h4>Args:</h4>+<ul>+<li><code>data_in</code> — Input data (audio samples, file paths, or text).</li>+<li><code>data_lengths</code> — Lengths of each input sample in the batch.</li>+<li><code>key</code> — Sample identifiers.</li>+<li><code>tokenizer</code> — Tokenizer instance for text encoding/decoding.</li>+<li><code>frontend</code> — Audio frontend for feature extraction.</li>+<li><code>**kwargs</code> — Additional keyword arguments.</li>+</ul>+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def inference(+ self,+ data_in,+ data_lengths=None,+ key: list = None,+ tokenizer=None,+ frontend=None,+ **kwargs,+ ):+ # init beamsearch+ """Run inference on input data.++ Args:+ data_in: Input data (audio samples, file paths, or text).+ data_lengths: Lengths of each input sample in the batch.+ key: Sample identifiers.+ tokenizer: Tokenizer instance for text encoding/decoding.+ frontend: Audio frontend for feature extraction.+ **kwargs: Additional keyword arguments.+ """+ is_use_ctc = kwargs.get("decoding_ctc_weight", 0.0) > 0.00001 and self.ctc != None+ is_use_lm = (+ kwargs.get("lm_weight", 0.0) > 0.00001 and kwargs.get("lm_file", None) is not None+ )+ pred_timestamp = kwargs.get("pred_timestamp", False)+ if self.beam_search is None and (is_use_lm or is_use_ctc):+ logging.info("enable beam_search")+ self.init_beam_search(**kwargs)+ self.nbest = kwargs.get("nbest", 1)++</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/e_paraformer/model.py#L600" target="_blank" class="src-more">View full source →</a></details></div><div class="mblk"><code>.export(**kwargs)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/e_paraformer/model.py#L765" target="_blank" class="gh-link">L765</a><div class="mdoc"><p>Export.</p>+<h4>Args:</h4>+<ul>+<li><code>**kwargs</code> — Additional keyword arguments.</li>+</ul>+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def export(self, **kwargs):+ """Export.++ Args:+ **kwargs: Additional keyword arguments.+ """+ from .export_meta import export_rebuild_model++ if "max_seq_len" not in kwargs:+ kwargs["max_seq_len"] = 512+ models = export_rebuild_model(model=self, **kwargs)+ return models+</code></pre></details></div></div>+<div class="api-detail" id="e52"><span class="dtag badge-m">method</span><h2><code>EParaformer.forward(speech, speech_lengths, text, text_lengths, **kwargs)</code></h2><div class="dmod">funasr.models.e_paraformer.EParaformer · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/e_paraformer/model.py#L220" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Encoder + Decoder + Calc loss</p>+<h4>Args:</h4>+<ul>+<li><code>speech</code> — (Batch, Length, ...)</li>+<li><code>speech_lengths</code> — (Batch, )</li>+<li><code>text</code> — (Batch, Length)</li>+<li><code>text_lengths</code> — (Batch,)</li>+</ul>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def forward(+ self,+ speech: torch.Tensor,+ speech_lengths: torch.Tensor,+ text: torch.Tensor,+ text_lengths: torch.Tensor,+ **kwargs,+ ) -> Tuple[torch.Tensor, Dict[str, torch.Tensor], torch.Tensor]:+ """Encoder + Decoder + Calc loss+ Args:+ speech: (Batch, Length, ...)+ speech_lengths: (Batch, )+ text: (Batch, Length)+ text_lengths: (Batch,)+ """+ if len(text_lengths.size()) > 1:+ text_lengths = text_lengths[:, 0]+ if len(speech_lengths.size()) > 1:+ speech_lengths = speech_lengths[:, 0]++ batch_size = speech.shape[0]++ # Encoder+ encoder_out, encoder_out_lens = self.encode(speech, speech_lengths)++ loss_ctc, cer_ctc = None, None+ loss_pre = None+ stats = dict()++ # decoder: CTC branch+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/e_paraformer/model.py#L220" target="_blank" class="src-more">View full source on GitHub →</a></details></div>+<div class="api-detail" id="e53"><span class="dtag badge-m">method</span><h2><code>EParaformer.encode(speech, speech_lengths, **kwargs)</code></h2><div class="dmod">funasr.models.e_paraformer.EParaformer · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/e_paraformer/model.py#L292" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Encoder. Note that this method is used by asr_inference.py</p>+<h4>Args:</h4>+<ul>+<li><code>speech</code> — (Batch, Length, ...)</li>+<li><code>speech_lengths</code> — (Batch, )</li>+<li><code>ind</code> — int</li>+</ul>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def encode(+ self,+ speech: torch.Tensor,+ speech_lengths: torch.Tensor,+ **kwargs,+ ) -> Tuple[torch.Tensor, torch.Tensor]:+ """Encoder. Note that this method is used by asr_inference.py+ Args:+ speech: (Batch, Length, ...)+ speech_lengths: (Batch, )+ ind: int+ """+ with autocast(False):++ # Data augmentation+ if self.specaug is not None and self.training:+ speech, speech_lengths = self.specaug(speech, speech_lengths)++ # Normalization for feature: e.g. Global-CMVN, Utterance-CMVN+ if self.normalize is not None:+ speech, speech_lengths = self.normalize(speech, speech_lengths)++ # Forward encoder+ encoder_out, encoder_out_lens, _ = self.encoder(speech, speech_lengths)+ if isinstance(encoder_out, tuple):+ encoder_out = encoder_out[0]++ return encoder_out, encoder_out_lens+</code></pre></details></div>+<div class="api-detail" id="e54"><span class="dtag badge-m">method</span><h2><code>EParaformer.calc_predictor(encoder_out, encoder_out_lens)</code></h2><div class="dmod">funasr.models.e_paraformer.EParaformer · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/e_paraformer/model.py#L321" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Calc predictor.</p>+<h4>Args:</h4>+<ul>+<li><code>encoder_out</code> — Encoder output tensor.</li>+<li><code>encoder_out_lens</code> — Encoder output lengths.</li>+</ul>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def calc_predictor(self, encoder_out, encoder_out_lens):++ """Calc predictor.++ Args:+ encoder_out: Encoder output tensor.+ encoder_out_lens: Encoder output lengths.+ """+ encoder_out_mask = (+ ~make_pad_mask(encoder_out_lens, maxlen=encoder_out.size(1))[:, None, :]+ ).to(encoder_out.device)+ pre_acoustic_embeds, pre_token_length, alphas, pre_peak_index = self.predictor(+ encoder_out, None, encoder_out_mask, ignore_id=self.ignore_id+ )+ return pre_acoustic_embeds, pre_token_length, alphas, pre_peak_index+</code></pre></details></div>+<div class="api-detail" id="e55"><span class="dtag badge-m">method</span><h2><code>EParaformer.cal_decoder_with_predictor(encoder_out, encoder_out_lens, sematic_embeds, ys_pad_lens)</code></h2><div class="dmod">funasr.models.e_paraformer.EParaformer · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/e_paraformer/model.py#L337" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Cal decoder with predictor.</p>+<h4>Args:</h4>+<ul>+<li><code>encoder_out</code> — Encoder output tensor.</li>+<li><code>encoder_out_lens</code> — Encoder output lengths.</li>+<li><code>sematic_embeds</code> — TODO.</li>+<li><code>ys_pad_lens</code> — Lengths of ys_pad.</li>+</ul>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def cal_decoder_with_predictor(+ self, encoder_out, encoder_out_lens, sematic_embeds, ys_pad_lens+ ):++ """Cal decoder with predictor.++ Args:+ encoder_out: Encoder output tensor.+ encoder_out_lens: Encoder output lengths.+ sematic_embeds: TODO.+ ys_pad_lens: Lengths of ys_pad.+ """+ decoder_outs = self.decoder(encoder_out, encoder_out_lens, sematic_embeds, ys_pad_lens)+ decoder_out = decoder_outs[0]+ decoder_out = torch.log_softmax(decoder_out, dim=-1)+ return decoder_out, ys_pad_lens+</code></pre></details></div>+<div class="api-detail" id="e56"><span class="dtag badge-m">method</span><h2><code>EParaformer.sampler(encoder_out, encoder_out_lens, ys_pad, ys_pad_lens, pre_acoustic_embeds)</code></h2><div class="dmod">funasr.models.e_paraformer.EParaformer · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/e_paraformer/model.py#L425" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Sampler.</p>+<h4>Args:</h4>+<ul>+<li><code>encoder_out</code> — Encoder output tensor.</li>+<li><code>encoder_out_lens</code> — Encoder output lengths.</li>+<li><code>ys_pad</code> — TODO.</li>+<li><code>ys_pad_lens</code> — Lengths of ys_pad.</li>+<li><code>pre_acoustic_embeds</code> — TODO.</li>+</ul>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def sampler(self, encoder_out, encoder_out_lens, ys_pad, ys_pad_lens, pre_acoustic_embeds):++ """Sampler.++ Args:+ encoder_out: Encoder output tensor.+ encoder_out_lens: Encoder output lengths.+ ys_pad: TODO.+ ys_pad_lens: Lengths of ys_pad.+ pre_acoustic_embeds: TODO.+ """+ tgt_mask = (~make_pad_mask(ys_pad_lens, maxlen=ys_pad_lens.max())[:, :, None]).to(+ ys_pad.device+ )+ ys_pad_masked = ys_pad * tgt_mask[:, :, 0]+ if self.share_embedding:+ ys_pad_embed = self.decoder.output_layer.weight[ys_pad_masked]+ else:+ ys_pad_embed = self.decoder.embed(ys_pad_masked)+ with torch.no_grad():+ decoder_outs = self.decoder(+ encoder_out, encoder_out_lens, pre_acoustic_embeds, ys_pad_lens+ )+ decoder_out, _ = decoder_outs[0], decoder_outs[1]+ pred_tokens = decoder_out.argmax(-1)+ nonpad_positions = ys_pad.ne(self.ignore_id)+ seq_lens = (nonpad_positions).sum(1)+ same_num = ((pred_tokens == ys_pad) & nonpad_positions).sum(1)+ input_mask = torch.ones_like(nonpad_positions)+ bsz, seq_len = ys_pad.size()+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/e_paraformer/model.py#L425" target="_blank" class="src-more">View full source on GitHub →</a></details></div>+<div class="api-detail" id="e57"><span class="dtag badge-m">method</span><h2><code>EParaformer.sampler_with_grad(encoder_out, encoder_out_lens, ys_pad, ys_pad_lens, pre_acoustic_embeds)</code></h2><div class="dmod">funasr.models.e_paraformer.EParaformer · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/e_paraformer/model.py#L474" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Sampler with grad.</p>+<h4>Args:</h4>+<ul>+<li><code>encoder_out</code> — Encoder output tensor.</li>+<li><code>encoder_out_lens</code> — Encoder output lengths.</li>+<li><code>ys_pad</code> — TODO.</li>+<li><code>ys_pad_lens</code> — Lengths of ys_pad.</li>+<li><code>pre_acoustic_embeds</code> — TODO.</li>+</ul>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def sampler_with_grad(self, encoder_out, encoder_out_lens, ys_pad, ys_pad_lens, pre_acoustic_embeds):++ """Sampler with grad.++ Args:+ encoder_out: Encoder output tensor.+ encoder_out_lens: Encoder output lengths.+ ys_pad: TODO.+ ys_pad_lens: Lengths of ys_pad.+ pre_acoustic_embeds: TODO.+ """+ tgt_mask = (~make_pad_mask(ys_pad_lens, maxlen=ys_pad_lens.max())[:, :, None]).to(+ ys_pad.device+ )+ ys_pad_masked = ys_pad * tgt_mask[:, :, 0]+ if self.share_embedding:+ ys_pad_embed = self.decoder.output_layer.weight[ys_pad_masked]+ else:+ ys_pad_embed = self.decoder.embed(ys_pad_masked)+ decoder_outs = self.decoder(+ encoder_out, encoder_out_lens, pre_acoustic_embeds, ys_pad_lens+ )+ decoder_out, _ = decoder_outs[0], decoder_outs[1]+ pred_tokens = decoder_out.argmax(-1)+ nonpad_positions = ys_pad.ne(self.ignore_id)+ seq_lens = (nonpad_positions).sum(1)+ same_num = ((pred_tokens == ys_pad) & nonpad_positions).sum(1)+ input_mask = torch.ones_like(nonpad_positions)+ bsz, seq_len = ys_pad.size()+ for li in range(bsz):+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/e_paraformer/model.py#L474" target="_blank" class="src-more">View full source on GitHub →</a></details></div>+<div class="api-detail" id="e58"><span class="dtag badge-m">method</span><h2><code>EParaformer.init_beam_search(**kwargs)</code></h2><div class="dmod">funasr.models.e_paraformer.EParaformer · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/e_paraformer/model.py#L548" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Init beam search.</p>+<h4>Args:</h4>+<ul>+<li><code>**kwargs</code> — Additional keyword arguments.</li>+</ul>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def init_beam_search(+ self,+ **kwargs,+ ):+ """Init beam search.++ Args:+ **kwargs: Additional keyword arguments.+ """+ from funasr.models.paraformer.search import BeamSearchPara+ from funasr.models.transformer.scorers.ctc import CTCPrefixScorer+ from funasr.models.transformer.scorers.length_bonus import LengthBonus++ # 1. Build ASR model+ scorers = {}++ if self.ctc != None:+ ctc = CTCPrefixScorer(ctc=self.ctc, eos=self.eos)+ scorers.update(ctc=ctc)+ token_list = kwargs.get("token_list")+ scorers.update(+ length_bonus=LengthBonus(len(token_list)),+ )++ # 3. Build ngram model+ # ngram is not supported now+ ngram = None+ scorers["ngram"] = ngram++ weights = dict(+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/e_paraformer/model.py#L548" target="_blank" class="src-more">View full source on GitHub →</a></details></div>+<div class="api-detail" id="e59"><span class="dtag badge-m">method</span><h2><code>EParaformer.inference(data_in, data_lengths, key, tokenizer, frontend, **kwargs)</code></h2><div class="dmod">funasr.models.e_paraformer.EParaformer · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/e_paraformer/model.py#L600" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Run inference on input data.</p>+<h4>Args:</h4>+<ul>+<li><code>data_in</code> — Input data (audio samples, file paths, or text).</li>+<li><code>data_lengths</code> — Lengths of each input sample in the batch.</li>+<li><code>key</code> — Sample identifiers.</li>+<li><code>tokenizer</code> — Tokenizer instance for text encoding/decoding.</li>+<li><code>frontend</code> — Audio frontend for feature extraction.</li>+<li><code>**kwargs</code> — Additional keyword arguments.</li>+</ul>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def inference(+ self,+ data_in,+ data_lengths=None,+ key: list = None,+ tokenizer=None,+ frontend=None,+ **kwargs,+ ):+ # init beamsearch+ """Run inference on input data.++ Args:+ data_in: Input data (audio samples, file paths, or text).+ data_lengths: Lengths of each input sample in the batch.+ key: Sample identifiers.+ tokenizer: Tokenizer instance for text encoding/decoding.+ frontend: Audio frontend for feature extraction.+ **kwargs: Additional keyword arguments.+ """+ is_use_ctc = kwargs.get("decoding_ctc_weight", 0.0) > 0.00001 and self.ctc != None+ is_use_lm = (+ kwargs.get("lm_weight", 0.0) > 0.00001 and kwargs.get("lm_file", None) is not None+ )+ pred_timestamp = kwargs.get("pred_timestamp", False)+ if self.beam_search is None and (is_use_lm or is_use_ctc):+ logging.info("enable beam_search")+ self.init_beam_search(**kwargs)+ self.nbest = kwargs.get("nbest", 1)++</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/e_paraformer/model.py#L600" target="_blank" class="src-more">View full source on GitHub →</a></details></div>+<div class="api-detail" id="e60"><span class="dtag badge-m">method</span><h2><code>EParaformer.export(**kwargs)</code></h2><div class="dmod">funasr.models.e_paraformer.EParaformer · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/e_paraformer/model.py#L765" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Export.</p>+<h4>Args:</h4>+<ul>+<li><code>**kwargs</code> — Additional keyword arguments.</li>+</ul>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def export(self, **kwargs):+ """Export.++ Args:+ **kwargs: Additional keyword arguments.+ """+ from .export_meta import export_rebuild_model++ if "max_seq_len" not in kwargs:+ kwargs["max_seq_len"] = 512+ models = export_rebuild_model(model=self, **kwargs)+ return models+</code></pre></details></div>+<div class="api-detail" id="e61"><span class="dtag badge-c">class</span><h2>Emotion2vec</h2><div class="dmod">funasr.models.emotion2vec · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/emotion2vec/model.py#L40" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><ul>+<li><code>Author</code> — Ziyang Ma, Zhisheng Zheng, Jiaxin Ye, Jinchao Li, Zhifu Gao, Shiliang Zhang, Xie Chen</li>+<li><code>emotion2vec</code> — Self-Supervised Pre-Training for Speech Emotion Representation</li>+</ul>+<p>https://arxiv.org/abs/2312.15185</p>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code>class Emotion2vec(torch.nn.Module):+ """+ Author: Ziyang Ma, Zhisheng Zheng, Jiaxin Ye, Jinchao Li, Zhifu Gao, Shiliang Zhang, Xie Chen+ emotion2vec: Self-Supervised Pre-Training for Speech Emotion Representation+ https://arxiv.org/abs/2312.15185+ """++ def __init__(self, **kwargs):+ """Initialize Emotion2vec.++ Args:+ **kwargs: Additional keyword arguments.+ """+ super().__init__()+ # import pdb; pdb.set_trace()+ cfg = OmegaConf.create(kwargs["model_conf"])+ self.cfg = cfg++ make_layer_norm = partial(+ torch.nn.LayerNorm, eps=cfg.get("norm_eps"), elementwise_affine=cfg.get("norm_affine")+ )++ def make_block(drop_path, dim=None, heads=None):+ """Make block.++ Args:+ drop_path: TODO.+ dim: TODO.+ heads: TODO.+ """+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/emotion2vec/model.py#L40" target="_blank" class="src-more">View full source on GitHub →</a></details><h3 class="mt">Methods</h3><div class="mblk"><code>.forward(source, target, id, mode, padding_mask, mask, features_only, force_remove_masked, remove_extra_tokens, precomputed_mask, **kwargs)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/emotion2vec/model.py#L121" target="_blank" class="gh-link">L121</a><div class="mdoc"><p>Forward pass for training.</p>+<h4>Args:</h4>+<ul>+<li><code>source</code> — TODO.</li>+<li><code>target</code> — TODO.</li>+<li><code>id</code> — TODO.</li>+<li><code>mode</code> — TODO.</li>+<li><code>padding_mask</code> — TODO.</li>+<li><code>mask</code> — TODO.</li>+<li><code>features_only</code> — TODO.</li>+<li><code>force_remove_masked</code> — TODO.</li>+<li><code>remove_extra_tokens</code> — TODO.</li>+<li><code>precomputed_mask</code> — TODO.</li>+<li><code>**kwargs</code> — Additional keyword arguments.</li>+</ul>+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def forward(+ self,+ source,+ target=None,+ id=None,+ mode=None,+ padding_mask=None,+ mask=True,+ features_only=False,+ force_remove_masked=False,+ remove_extra_tokens=True,+ precomputed_mask=None,+ **kwargs,+ ):++ """Forward pass for training.++ Args:+ source: TODO.+ target: TODO.+ id: TODO.+ mode: TODO.+ padding_mask: TODO.+ mask: TODO.+ features_only: TODO.+ force_remove_masked: TODO.+ remove_extra_tokens: TODO.+ precomputed_mask: TODO.+ **kwargs: Additional keyword arguments.+ """+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/emotion2vec/model.py#L121" target="_blank" class="src-more">View full source →</a></details></div><div class="mblk"><code>.extract_features(source, mode, padding_mask, mask, remove_extra_tokens)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/emotion2vec/model.py#L212" target="_blank" class="gh-link">L212</a><div class="mdoc"><p>Extract features.</p>+<h4>Args:</h4>+<ul>+<li><code>source</code> — TODO.</li>+<li><code>mode</code> — TODO.</li>+<li><code>padding_mask</code> — TODO.</li>+<li><code>mask</code> — TODO.</li>+<li><code>remove_extra_tokens</code> — TODO.</li>+</ul>+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def extract_features(+ self, source, mode=None, padding_mask=None, mask=False, remove_extra_tokens=True+ ):+ """Extract features.++ Args:+ source: TODO.+ mode: TODO.+ padding_mask: TODO.+ mask: TODO.+ remove_extra_tokens: TODO.+ """+ res = self.forward(+ source,+ mode=mode,+ padding_mask=padding_mask,+ mask=mask,+ features_only=True,+ remove_extra_tokens=remove_extra_tokens,+ )+ return res+</code></pre></details></div><div class="mblk"><code>.inference(data_in, data_lengths, key, tokenizer, frontend, **kwargs)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/emotion2vec/model.py#L234" target="_blank" class="gh-link">L234</a><div class="mdoc"><p>Run inference on input data.</p>+<h4>Args:</h4>+<ul>+<li><code>data_in</code> — Input data (audio samples, file paths, or text).</li>+<li><code>data_lengths</code> — Lengths of each input sample in the batch.</li>+<li><code>key</code> — Sample identifiers.</li>+<li><code>tokenizer</code> — Tokenizer instance for text encoding/decoding.</li>+<li><code>frontend</code> — Audio frontend for feature extraction.</li>+<li><code>**kwargs</code> — Additional keyword arguments.</li>+</ul>+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def inference(+ self,+ data_in,+ data_lengths=None,+ key: list = None,+ tokenizer=None,+ frontend=None,+ **kwargs,+ ):++ # if source_file.endswith('.wav'):+ # wav, sr = sf.read(source_file)+ # channel = sf.info(source_file).channels+ # assert sr == 16e3, "Sample rate should be 16kHz, but got {}in file {}".format(sr, source_file)+ # assert channel == 1, "Channel should be 1, but got {} in file {}".format(channel, source_file)+ """Run inference on input data.++ Args:+ data_in: Input data (audio samples, file paths, or text).+ data_lengths: Lengths of each input sample in the batch.+ key: Sample identifiers.+ tokenizer: Tokenizer instance for text encoding/decoding.+ frontend: Audio frontend for feature extraction.+ **kwargs: Additional keyword arguments.+ """+ granularity = kwargs.get("granularity", "utterance")+ extract_embedding = kwargs.get("extract_embedding", True)+ if self.proj is None:+ extract_embedding = True+ meta_data = {}+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/emotion2vec/model.py#L234" target="_blank" class="src-more">View full source →</a></details></div><div class="mblk"><code>.export(**kwargs)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/emotion2vec/model.py#L320" target="_blank" class="gh-link">L320</a><div class="mdoc"><p>Export.</p>+<h4>Args:</h4>+<ul>+<li><code>**kwargs</code> — Additional keyword arguments.</li>+</ul>+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def export(self, **kwargs):+ """Export.++ Args:+ **kwargs: Additional keyword arguments.+ """+ from .export_meta import export_rebuild_model++ models = export_rebuild_model(model=self, **kwargs)+ return models+</code></pre></details></div></div>+<div class="api-detail" id="e62"><span class="dtag badge-m">method</span><h2><code>Emotion2vec.forward(source, target, id, mode, padding_mask, mask, features_only, force_remove_masked, remove_extra_tokens, precomputed_mask, **kwargs)</code></h2><div class="dmod">funasr.models.emotion2vec.Emotion2vec · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/emotion2vec/model.py#L121" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Forward pass for training.</p>+<h4>Args:</h4>+<ul>+<li><code>source</code> — TODO.</li>+<li><code>target</code> — TODO.</li>+<li><code>id</code> — TODO.</li>+<li><code>mode</code> — TODO.</li>+<li><code>padding_mask</code> — TODO.</li>+<li><code>mask</code> — TODO.</li>+<li><code>features_only</code> — TODO.</li>+<li><code>force_remove_masked</code> — TODO.</li>+<li><code>remove_extra_tokens</code> — TODO.</li>+<li><code>precomputed_mask</code> — TODO.</li>+<li><code>**kwargs</code> — Additional keyword arguments.</li>+</ul>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def forward(+ self,+ source,+ target=None,+ id=None,+ mode=None,+ padding_mask=None,+ mask=True,+ features_only=False,+ force_remove_masked=False,+ remove_extra_tokens=True,+ precomputed_mask=None,+ **kwargs,+ ):++ """Forward pass for training.++ Args:+ source: TODO.+ target: TODO.+ id: TODO.+ mode: TODO.+ padding_mask: TODO.+ mask: TODO.+ features_only: TODO.+ force_remove_masked: TODO.+ remove_extra_tokens: TODO.+ precomputed_mask: TODO.+ **kwargs: Additional keyword arguments.+ """+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/emotion2vec/model.py#L121" target="_blank" class="src-more">View full source on GitHub →</a></details></div>+<div class="api-detail" id="e63"><span class="dtag badge-m">method</span><h2><code>Emotion2vec.extract_features(source, mode, padding_mask, mask, remove_extra_tokens)</code></h2><div class="dmod">funasr.models.emotion2vec.Emotion2vec · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/emotion2vec/model.py#L212" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Extract features.</p>+<h4>Args:</h4>+<ul>+<li><code>source</code> — TODO.</li>+<li><code>mode</code> — TODO.</li>+<li><code>padding_mask</code> — TODO.</li>+<li><code>mask</code> — TODO.</li>+<li><code>remove_extra_tokens</code> — TODO.</li>+</ul>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def extract_features(+ self, source, mode=None, padding_mask=None, mask=False, remove_extra_tokens=True+ ):+ """Extract features.++ Args:+ source: TODO.+ mode: TODO.+ padding_mask: TODO.+ mask: TODO.+ remove_extra_tokens: TODO.+ """+ res = self.forward(+ source,+ mode=mode,+ padding_mask=padding_mask,+ mask=mask,+ features_only=True,+ remove_extra_tokens=remove_extra_tokens,+ )+ return res+</code></pre></details></div>+<div class="api-detail" id="e64"><span class="dtag badge-m">method</span><h2><code>Emotion2vec.inference(data_in, data_lengths, key, tokenizer, frontend, **kwargs)</code></h2><div class="dmod">funasr.models.emotion2vec.Emotion2vec · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/emotion2vec/model.py#L234" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Run inference on input data.</p>+<h4>Args:</h4>+<ul>+<li><code>data_in</code> — Input data (audio samples, file paths, or text).</li>+<li><code>data_lengths</code> — Lengths of each input sample in the batch.</li>+<li><code>key</code> — Sample identifiers.</li>+<li><code>tokenizer</code> — Tokenizer instance for text encoding/decoding.</li>+<li><code>frontend</code> — Audio frontend for feature extraction.</li>+<li><code>**kwargs</code> — Additional keyword arguments.</li>+</ul>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def inference(+ self,+ data_in,+ data_lengths=None,+ key: list = None,+ tokenizer=None,+ frontend=None,+ **kwargs,+ ):++ # if source_file.endswith('.wav'):+ # wav, sr = sf.read(source_file)+ # channel = sf.info(source_file).channels+ # assert sr == 16e3, "Sample rate should be 16kHz, but got {}in file {}".format(sr, source_file)+ # assert channel == 1, "Channel should be 1, but got {} in file {}".format(channel, source_file)+ """Run inference on input data.++ Args:+ data_in: Input data (audio samples, file paths, or text).+ data_lengths: Lengths of each input sample in the batch.+ key: Sample identifiers.+ tokenizer: Tokenizer instance for text encoding/decoding.+ frontend: Audio frontend for feature extraction.+ **kwargs: Additional keyword arguments.+ """+ granularity = kwargs.get("granularity", "utterance")+ extract_embedding = kwargs.get("extract_embedding", True)+ if self.proj is None:+ extract_embedding = True+ meta_data = {}+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/emotion2vec/model.py#L234" target="_blank" class="src-more">View full source on GitHub →</a></details></div>+<div class="api-detail" id="e65"><span class="dtag badge-m">method</span><h2><code>Emotion2vec.export(**kwargs)</code></h2><div class="dmod">funasr.models.emotion2vec.Emotion2vec · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/emotion2vec/model.py#L320" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Export.</p>+<h4>Args:</h4>+<ul>+<li><code>**kwargs</code> — Additional keyword arguments.</li>+</ul>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def export(self, **kwargs):+ """Export.++ Args:+ **kwargs: Additional keyword arguments.+ """+ from .export_meta import export_rebuild_model++ models = export_rebuild_model(model=self, **kwargs)+ return models+</code></pre></details></div>+<div class="api-detail" id="e66"><span class="dtag badge-c">class</span><h2>ERes2NetV2SV</h2><div class="dmod">funasr.models.eres2net · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/eres2net/model.py#L21" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><ul>+<li><code>ERes2NetV2</code> — Enhanced Res2Net v2 for Speaker Verification.</li>+</ul>+<p>Improved speaker embedding model based on Res2Net architecture with</p>+<ul>+<li><code>multi</code> — scale feature aggregation. Provides 192-dim speaker embeddings</li>+</ul>+<p>for speaker verification and diarization.</p>+<p>Better than CAM++ for short-duration audio (< 3s) speaker feature extraction.</p>+<ul>+<li><code>Output</code> — {"spk_embedding": Tensor of shape (1, 192)}</li>+</ul>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code>class ERes2NetV2SV(torch.nn.Module):+ """ERes2NetV2: Enhanced Res2Net v2 for Speaker Verification.++ Improved speaker embedding model based on Res2Net architecture with+ multi-scale feature aggregation. Provides 192-dim speaker embeddings+ for speaker verification and diarization.++ Better than CAM++ for short-duration audio (< 3s) speaker feature extraction.++ Output: {"spk_embedding": Tensor of shape (1, 192)}+ """++ def __init__(+ self,+ feat_dim=80,+ embedding_size=192,+ m_channels=64,+ baseWidth=26,+ scale=2,+ expansion=2,+ num_blocks=[3, 4, 6, 3],+ pooling_func="TSTP",+ two_emb_layer=False,+ **kwargs,+ ):+ """Initialize ERes2NetV2SV.++ Args:+ feat_dim: Size/dimension parameter.+ embedding_size: Size/dimension parameter.+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/eres2net/model.py#L21" target="_blank" class="src-more">View full source on GitHub →</a></details><h3 class="mt">Methods</h3><div class="mblk"><code>.forward(x)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/eres2net/model.py#L97" target="_blank" class="gh-link">L97</a><div class="mdoc"><p>Forward pass for training.</p>+<h4>Args:</h4>+<ul>+<li><code>x</code> — TODO.</li>+</ul>+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def forward(self, x):+ """Forward pass for training.++ Args:+ x: TODO.+ """+ return self.model(x)+</code></pre></details></div><div class="mblk"><code>.inference(data_in, data_lengths, key, tokenizer, frontend, **kwargs)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/eres2net/model.py#L105" target="_blank" class="gh-link">L105</a><div class="mdoc"><p>Run inference on input data.</p>+<h4>Args:</h4>+<ul>+<li><code>data_in</code> — Input data (audio samples, file paths, or text).</li>+<li><code>data_lengths</code> — Lengths of each input sample in the batch.</li>+<li><code>key</code> — Sample identifiers.</li>+<li><code>tokenizer</code> — Tokenizer instance for text encoding/decoding.</li>+<li><code>frontend</code> — Audio frontend for feature extraction.</li>+<li><code>**kwargs</code> — Additional keyword arguments.</li>+</ul>+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def inference(+ self,+ data_in,+ data_lengths=None,+ key: list = None,+ tokenizer=None,+ frontend=None,+ **kwargs,+ ):+ """Run inference on input data.++ Args:+ data_in: Input data (audio samples, file paths, or text).+ data_lengths: Lengths of each input sample in the batch.+ key: Sample identifiers.+ tokenizer: Tokenizer instance for text encoding/decoding.+ frontend: Audio frontend for feature extraction.+ **kwargs: Additional keyword arguments.+ """+ meta_data = {}+ time1 = time.perf_counter()+ audio_sample_list = load_audio_text_image_video(+ data_in, fs=16000, audio_fs=kwargs.get("fs", 16000), data_type="sound"+ )+ time2 = time.perf_counter()+ meta_data["load_data"] = f"{time2 - time1:0.3f}"+ speech, speech_lengths, speech_times = extract_feature(audio_sample_list)+ speech = speech.to(device=kwargs["device"])+ time3 = time.perf_counter()+ meta_data["extract_feat"] = f"{time3 - time2:0.3f}"+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/eres2net/model.py#L105" target="_blank" class="src-more">View full source →</a></details></div></div>+<div class="api-detail" id="e67"><span class="dtag badge-m">method</span><h2><code>ERes2NetV2SV.forward(x)</code></h2><div class="dmod">funasr.models.eres2net.ERes2NetV2SV · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/eres2net/model.py#L97" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Forward pass for training.</p>+<h4>Args:</h4>+<ul>+<li><code>x</code> — TODO.</li>+</ul>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def forward(self, x):+ """Forward pass for training.++ Args:+ x: TODO.+ """+ return self.model(x)+</code></pre></details></div>+<div class="api-detail" id="e68"><span class="dtag badge-m">method</span><h2><code>ERes2NetV2SV.inference(data_in, data_lengths, key, tokenizer, frontend, **kwargs)</code></h2><div class="dmod">funasr.models.eres2net.ERes2NetV2SV · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/eres2net/model.py#L105" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Run inference on input data.</p>+<h4>Args:</h4>+<ul>+<li><code>data_in</code> — Input data (audio samples, file paths, or text).</li>+<li><code>data_lengths</code> — Lengths of each input sample in the batch.</li>+<li><code>key</code> — Sample identifiers.</li>+<li><code>tokenizer</code> — Tokenizer instance for text encoding/decoding.</li>+<li><code>frontend</code> — Audio frontend for feature extraction.</li>+<li><code>**kwargs</code> — Additional keyword arguments.</li>+</ul>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def inference(+ self,+ data_in,+ data_lengths=None,+ key: list = None,+ tokenizer=None,+ frontend=None,+ **kwargs,+ ):+ """Run inference on input data.++ Args:+ data_in: Input data (audio samples, file paths, or text).+ data_lengths: Lengths of each input sample in the batch.+ key: Sample identifiers.+ tokenizer: Tokenizer instance for text encoding/decoding.+ frontend: Audio frontend for feature extraction.+ **kwargs: Additional keyword arguments.+ """+ meta_data = {}+ time1 = time.perf_counter()+ audio_sample_list = load_audio_text_image_video(+ data_in, fs=16000, audio_fs=kwargs.get("fs", 16000), data_type="sound"+ )+ time2 = time.perf_counter()+ meta_data["load_data"] = f"{time2 - time1:0.3f}"+ speech, speech_lengths, speech_times = extract_feature(audio_sample_list)+ speech = speech.to(device=kwargs["device"])+ time3 = time.perf_counter()+ meta_data["extract_feat"] = f"{time3 - time2:0.3f}"+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/eres2net/model.py#L105" target="_blank" class="src-more">View full source on GitHub →</a></details></div>+<div class="api-detail" id="e69"><span class="dtag badge-c">class</span><h2>FsmnKWS</h2><div class="dmod">funasr.models.fsmn_kws · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/fsmn_kws/model.py#L27" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><ul>+<li><code>FSMN</code> — KWS: Keyword Spotting model using FSMN architecture.</li>+</ul>+<p>Detects predefined keywords/wake words in audio streams.</p>+<p>Supports both offline and streaming operation.</p>+<ul>+<li><code>Output</code> — {"key": str, "value": detected_keyword_info}</li>+</ul>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code>class FsmnKWS(torch.nn.Module):+ """FSMN-KWS: Keyword Spotting model using FSMN architecture.++ Detects predefined keywords/wake words in audio streams.+ Supports both offline and streaming operation.++ Output: {"key": str, "value": detected_keyword_info}+ """++ def __init__(+ self,+ specaug: Optional[str] = None,+ specaug_conf: Optional[Dict] = None,+ normalize: str = None,+ normalize_conf: Optional[Dict] = None,+ encoder: str = None,+ encoder_conf: Optional[Dict] = None,+ ctc: str = None,+ ctc_conf: Optional[Dict] = None,+ ctc_weight: float = 1.0,+ input_size: int = 360,+ vocab_size: int = -1,+ ignore_id: int = -1,+ blank_id: int = 0,+ **kwargs,+ ):+ """Initialize FsmnKWS.++ Args:+ specaug: TODO.+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/fsmn_kws/model.py#L27" target="_blank" class="src-more">View full source on GitHub →</a></details><h3 class="mt">Methods</h3><div class="mblk"><code>.forward(speech, speech_lengths, text, text_lengths, **kwargs)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/fsmn_kws/model.py#L104" target="_blank" class="gh-link">L104</a><div class="mdoc"><p>Encoder + Decoder + Calc loss</p>+<h4>Args:</h4>+<ul>+<li><code>speech</code> — (Batch, Length, ...)</li>+<li><code>speech_lengths</code> — (Batch, )</li>+<li><code>text</code> — (Batch, Length)</li>+<li><code>text_lengths</code> — (Batch,)</li>+</ul>+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def forward(+ self,+ speech: torch.Tensor,+ speech_lengths: torch.Tensor,+ text: torch.Tensor,+ text_lengths: torch.Tensor,+ **kwargs,+ ) -> Tuple[torch.Tensor, Dict[str, torch.Tensor], torch.Tensor]:+ """Encoder + Decoder + Calc loss+ Args:+ speech: (Batch, Length, ...)+ speech_lengths: (Batch, )+ text: (Batch, Length)+ text_lengths: (Batch,)+ """+ if len(text_lengths.size()) > 1:+ text_lengths = text_lengths[:, 0]+ if len(speech_lengths.size()) > 1:+ speech_lengths = speech_lengths[:, 0]+ batch_size = speech.shape[0]++ # Encoder+ encoder_out, encoder_out_lens = self.encode(speech, speech_lengths)+ loss_ctc, cer_ctc = self._calc_ctc_loss(+ encoder_out, encoder_out_lens, text, text_lengths+ )++ # Collect CTC branch stats+ stats = dict()+ stats["loss_ctc"] = loss_ctc.detach() if loss_ctc is not None else None+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/fsmn_kws/model.py#L104" target="_blank" class="src-more">View full source →</a></details></div><div class="mblk"><code>.encode(speech, speech_lengths, **kwargs)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/fsmn_kws/model.py#L146" target="_blank" class="gh-link">L146</a><div class="mdoc"><p>Encoder. Note that this method is used by asr_inference.py</p>+<h4>Args:</h4>+<ul>+<li><code>speech</code> — (Batch, Length, ...)</li>+<li><code>speech_lengths</code> — (Batch, )</li>+<li><code>ind</code> — int</li>+</ul>+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def encode(+ self, speech: torch.Tensor, speech_lengths: torch.Tensor, **kwargs,+ ) -> Tuple[torch.Tensor, torch.Tensor]:+ """Encoder. Note that this method is used by asr_inference.py+ Args:+ speech: (Batch, Length, ...)+ speech_lengths: (Batch, )+ ind: int+ """+ with autocast(False):+ # Data augmentation+ if self.specaug is not None and self.training:+ speech, speech_lengths = self.specaug(speech, speech_lengths)++ # Normalization for feature: e.g. Global-CMVN, Utterance-CMVN+ if self.normalize is not None:+ speech, speech_lengths = self.normalize(speech, speech_lengths)++ # Forward encoder+ encoder_out = self.encoder(speech)+ encoder_out_lens = speech_lengths++ if isinstance(encoder_out, tuple):+ encoder_out = encoder_out[0]++ return encoder_out, encoder_out_lens+</code></pre></details></div><div class="mblk"><code>.inference(data_in, data_lengths, key, tokenizer, frontend, **kwargs)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/fsmn_kws/model.py#L199" target="_blank" class="gh-link">L199</a><div class="mdoc"><p>Run inference on input data.</p>+<h4>Args:</h4>+<ul>+<li><code>data_in</code> — Input data (audio samples, file paths, or text).</li>+<li><code>data_lengths</code> — Lengths of each input sample in the batch.</li>+<li><code>key</code> — Sample identifiers.</li>+<li><code>tokenizer</code> — Tokenizer instance for text encoding/decoding.</li>+<li><code>frontend</code> — Audio frontend for feature extraction.</li>+<li><code>**kwargs</code> — Additional keyword arguments.</li>+</ul>+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def inference(+ self,+ data_in,+ data_lengths=None,+ key: list=None,+ tokenizer=None,+ frontend=None,+ **kwargs,+ ):+ """Run inference on input data.++ Args:+ data_in: Input data (audio samples, file paths, or text).+ data_lengths: Lengths of each input sample in the batch.+ key: Sample identifiers.+ tokenizer: Tokenizer instance for text encoding/decoding.+ frontend: Audio frontend for feature extraction.+ **kwargs: Additional keyword arguments.+ """+ keywords = kwargs.get("keywords")+ from funasr.utils.kws_utils import KwsCtcPrefixDecoder+ self.kws_decoder = KwsCtcPrefixDecoder(+ ctc=self.ctc,+ keywords=keywords,+ token_list=tokenizer.token_list,+ seg_dict=tokenizer.seg_dict,+ )++ meta_data = {}+ if isinstance(data_in, torch.Tensor) and kwargs.get("data_type", "sound") == "fbank": # fbank+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/fsmn_kws/model.py#L199" target="_blank" class="src-more">View full source →</a></details></div></div>+<div class="api-detail" id="e70"><span class="dtag badge-m">method</span><h2><code>FsmnKWS.forward(speech, speech_lengths, text, text_lengths, **kwargs)</code></h2><div class="dmod">funasr.models.fsmn_kws.FsmnKWS · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/fsmn_kws/model.py#L104" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Encoder + Decoder + Calc loss</p>+<h4>Args:</h4>+<ul>+<li><code>speech</code> — (Batch, Length, ...)</li>+<li><code>speech_lengths</code> — (Batch, )</li>+<li><code>text</code> — (Batch, Length)</li>+<li><code>text_lengths</code> — (Batch,)</li>+</ul>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def forward(+ self,+ speech: torch.Tensor,+ speech_lengths: torch.Tensor,+ text: torch.Tensor,+ text_lengths: torch.Tensor,+ **kwargs,+ ) -> Tuple[torch.Tensor, Dict[str, torch.Tensor], torch.Tensor]:+ """Encoder + Decoder + Calc loss+ Args:+ speech: (Batch, Length, ...)+ speech_lengths: (Batch, )+ text: (Batch, Length)+ text_lengths: (Batch,)+ """+ if len(text_lengths.size()) > 1:+ text_lengths = text_lengths[:, 0]+ if len(speech_lengths.size()) > 1:+ speech_lengths = speech_lengths[:, 0]+ batch_size = speech.shape[0]++ # Encoder+ encoder_out, encoder_out_lens = self.encode(speech, speech_lengths)+ loss_ctc, cer_ctc = self._calc_ctc_loss(+ encoder_out, encoder_out_lens, text, text_lengths+ )++ # Collect CTC branch stats+ stats = dict()+ stats["loss_ctc"] = loss_ctc.detach() if loss_ctc is not None else None+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/fsmn_kws/model.py#L104" target="_blank" class="src-more">View full source on GitHub →</a></details></div>+<div class="api-detail" id="e71"><span class="dtag badge-m">method</span><h2><code>FsmnKWS.encode(speech, speech_lengths, **kwargs)</code></h2><div class="dmod">funasr.models.fsmn_kws.FsmnKWS · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/fsmn_kws/model.py#L146" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Encoder. Note that this method is used by asr_inference.py</p>+<h4>Args:</h4>+<ul>+<li><code>speech</code> — (Batch, Length, ...)</li>+<li><code>speech_lengths</code> — (Batch, )</li>+<li><code>ind</code> — int</li>+</ul>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def encode(+ self, speech: torch.Tensor, speech_lengths: torch.Tensor, **kwargs,+ ) -> Tuple[torch.Tensor, torch.Tensor]:+ """Encoder. Note that this method is used by asr_inference.py+ Args:+ speech: (Batch, Length, ...)+ speech_lengths: (Batch, )+ ind: int+ """+ with autocast(False):+ # Data augmentation+ if self.specaug is not None and self.training:+ speech, speech_lengths = self.specaug(speech, speech_lengths)++ # Normalization for feature: e.g. Global-CMVN, Utterance-CMVN+ if self.normalize is not None:+ speech, speech_lengths = self.normalize(speech, speech_lengths)++ # Forward encoder+ encoder_out = self.encoder(speech)+ encoder_out_lens = speech_lengths++ if isinstance(encoder_out, tuple):+ encoder_out = encoder_out[0]++ return encoder_out, encoder_out_lens+</code></pre></details></div>+<div class="api-detail" id="e72"><span class="dtag badge-m">method</span><h2><code>FsmnKWS.inference(data_in, data_lengths, key, tokenizer, frontend, **kwargs)</code></h2><div class="dmod">funasr.models.fsmn_kws.FsmnKWS · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/fsmn_kws/model.py#L199" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Run inference on input data.</p>+<h4>Args:</h4>+<ul>+<li><code>data_in</code> — Input data (audio samples, file paths, or text).</li>+<li><code>data_lengths</code> — Lengths of each input sample in the batch.</li>+<li><code>key</code> — Sample identifiers.</li>+<li><code>tokenizer</code> — Tokenizer instance for text encoding/decoding.</li>+<li><code>frontend</code> — Audio frontend for feature extraction.</li>+<li><code>**kwargs</code> — Additional keyword arguments.</li>+</ul>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def inference(+ self,+ data_in,+ data_lengths=None,+ key: list=None,+ tokenizer=None,+ frontend=None,+ **kwargs,+ ):+ """Run inference on input data.++ Args:+ data_in: Input data (audio samples, file paths, or text).+ data_lengths: Lengths of each input sample in the batch.+ key: Sample identifiers.+ tokenizer: Tokenizer instance for text encoding/decoding.+ frontend: Audio frontend for feature extraction.+ **kwargs: Additional keyword arguments.+ """+ keywords = kwargs.get("keywords")+ from funasr.utils.kws_utils import KwsCtcPrefixDecoder+ self.kws_decoder = KwsCtcPrefixDecoder(+ ctc=self.ctc,+ keywords=keywords,+ token_list=tokenizer.token_list,+ seg_dict=tokenizer.seg_dict,+ )++ meta_data = {}+ if isinstance(data_in, torch.Tensor) and kwargs.get("data_type", "sound") == "fbank": # fbank+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/fsmn_kws/model.py#L199" target="_blank" class="src-more">View full source on GitHub →</a></details></div>+<div class="api-detail" id="e73"><span class="dtag badge-c">class</span><h2>FsmnKWSConvert</h2><div class="dmod">funasr.models.fsmn_kws · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/fsmn_kws/model.py#L279" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><ul>+<li><code>Author</code> — Speech Lab of DAMO Academy, Alibaba Group</li>+<li><code>Deep</code> — FSMN for Large Vocabulary Continuous Speech Recognition</li>+</ul>+<p>https://arxiv.org/abs/1803.05030</p>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code>class FsmnKWSConvert(torch.nn.Module):+ """+ Author: Speech Lab of DAMO Academy, Alibaba Group+ Deep-FSMN for Large Vocabulary Continuous Speech Recognition+ https://arxiv.org/abs/1803.05030+ """++ def __init__(+ self,+ encoder: str = None,+ encoder_conf: Optional[Dict] = None,+ ctc: str = None,+ ctc_conf: Optional[Dict] = None,+ ctc_weight: float = 1.0,+ input_size: int = 360,+ vocab_size: int = -1,+ blank_id: int = 0,+ **kwargs,+ ):+ """Initialize FsmnKWSConvert.++ Args:+ encoder: TODO.+ encoder_conf: Configuration dict for encoder.+ ctc: TODO.+ ctc_conf: Configuration dict for ctc.+ ctc_weight: TODO.+ input_size: Size/dimension parameter.+ vocab_size: Size/dimension parameter.+ blank_id: TODO.+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/fsmn_kws/model.py#L279" target="_blank" class="src-more">View full source on GitHub →</a></details><h3 class="mt">Methods</h3><div class="mblk"><code>.to_kaldi_net()</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/fsmn_kws/model.py#L331" target="_blank" class="gh-link">L331</a><div class="mdoc"><p>To kaldi net.</p>+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def to_kaldi_net(self):+ """To kaldi net."""+ return self.encoder.to_kaldi_net()+</code></pre></details></div><div class="mblk"><code>.to_pytorch_net(kaldi_file)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/fsmn_kws/model.py#L336" target="_blank" class="gh-link">L336</a><div class="mdoc"><p>To pytorch net.</p>+<h4>Args:</h4>+<ul>+<li><code>kaldi_file</code> — TODO.</li>+</ul>+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def to_pytorch_net(self, kaldi_file):+ """To pytorch net.++ Args:+ kaldi_file: TODO.+ """+ return self.encoder.to_pytorch_net(kaldi_file)+</code></pre></details></div></div>+<div class="api-detail" id="e74"><span class="dtag badge-m">method</span><h2><code>FsmnKWSConvert.to_kaldi_net()</code></h2><div class="dmod">funasr.models.fsmn_kws.FsmnKWSConvert · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/fsmn_kws/model.py#L331" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>To kaldi net.</p>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def to_kaldi_net(self):+ """To kaldi net."""+ return self.encoder.to_kaldi_net()+</code></pre></details></div>+<div class="api-detail" id="e75"><span class="dtag badge-m">method</span><h2><code>FsmnKWSConvert.to_pytorch_net(kaldi_file)</code></h2><div class="dmod">funasr.models.fsmn_kws.FsmnKWSConvert · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/fsmn_kws/model.py#L336" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>To pytorch net.</p>+<h4>Args:</h4>+<ul>+<li><code>kaldi_file</code> — TODO.</li>+</ul>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def to_pytorch_net(self, kaldi_file):+ """To pytorch net.++ Args:+ kaldi_file: TODO.+ """+ return self.encoder.to_pytorch_net(kaldi_file)+</code></pre></details></div>+<div class="api-detail" id="e76"><span class="dtag badge-c">class</span><h2>FsmnKWSMT</h2><div class="dmod">funasr.models.fsmn_kws_mt · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/fsmn_kws_mt/model.py#L27" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><ul>+<li><code>FSMN</code> — KWS-MT: Multi-Task FSMN Keyword Spotting.</li>+</ul>+<p>Keyword spotting with multi-task learning: simultaneously</p>+<p>detects keywords and performs filler token classification.</p>+<p>Improves keyword detection robustness through auxiliary tasks.</p>+<ul>+<li><code>Output</code> — {"key": str, "value": keyword_detection_result}</li>+<li><code>Author</code> — Speech Lab of DAMO Academy, Alibaba Group</li>+<li><code>Deep</code> — FSMN for Large Vocabulary Continuous Speech Recognition</li>+</ul>+<p>https://arxiv.org/abs/1803.05030</p>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code>class FsmnKWSMT(torch.nn.Module):+ """FSMN-KWS-MT: Multi-Task FSMN Keyword Spotting.++ Keyword spotting with multi-task learning: simultaneously+ detects keywords and performs filler token classification.+ Improves keyword detection robustness through auxiliary tasks.++ Output: {"key": str, "value": keyword_detection_result}++ Author: Speech Lab of DAMO Academy, Alibaba Group+ Deep-FSMN for Large Vocabulary Continuous Speech Recognition+ https://arxiv.org/abs/1803.05030+ """++ def __init__(+ self,+ specaug: Optional[str] = None,+ specaug_conf: Optional[Dict] = None,+ normalize: str = None,+ normalize_conf: Optional[Dict] = None,+ encoder: str = None,+ encoder_conf: Optional[Dict] = None,+ ctc_conf: Optional[Dict] = None,+ input_size: int = 360,+ vocab_size: list = [],+ ignore_id: int = -1,+ blank_id: int = 0,+ **kwargs,+ ):+ """Initialize FsmnKWSMT.+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/fsmn_kws_mt/model.py#L27" target="_blank" class="src-more">View full source on GitHub →</a></details><h3 class="mt">Methods</h3><div class="mblk"><code>.forward(speech, speech_lengths, text, text_lengths, text2, text2_lengths, **kwargs)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/fsmn_kws_mt/model.py#L106" target="_blank" class="gh-link">L106</a><div class="mdoc"><p>Encoder + Decoder + Calc loss</p>+<h4>Args:</h4>+<ul>+<li><code>speech</code> — (Batch, Length, ...)</li>+<li><code>speech_lengths</code> — (Batch, )</li>+<li><code>text</code> — (Batch, Length)</li>+<li><code>text_lengths</code> — (Batch,)</li>+<li><code>text2</code> — (Batch, Length)</li>+<li><code>text2_lengths</code> — (Batch,)</li>+</ul>+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def forward(+ self,+ speech: torch.Tensor,+ speech_lengths: torch.Tensor,+ text: torch.Tensor,+ text_lengths: torch.Tensor,+ text2: torch.Tensor,+ text2_lengths: torch.Tensor,+ **kwargs,+ ) -> Tuple[torch.Tensor, Dict[str, torch.Tensor], torch.Tensor]:+ """Encoder + Decoder + Calc loss+ Args:+ speech: (Batch, Length, ...)+ speech_lengths: (Batch, )+ text: (Batch, Length)+ text_lengths: (Batch,)+ text2: (Batch, Length)+ text2_lengths: (Batch,)+ """+ if len(text_lengths.size()) > 1:+ text_lengths = text_lengths[:, 0]+ if len(speech_lengths.size()) > 1:+ speech_lengths = speech_lengths[:, 0]+ batch_size = speech.shape[0]++ # Encoder+ encoder_out, encoder_out2, encoder_out_lens = self.encode(speech, speech_lengths)++ loss_ctc, cer_ctc = self._calc_ctc_loss(+ encoder_out, encoder_out_lens, text, text_lengths+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/fsmn_kws_mt/model.py#L106" target="_blank" class="src-more">View full source →</a></details></div><div class="mblk"><code>.encode(speech, speech_lengths, **kwargs)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/fsmn_kws_mt/model.py#L158" target="_blank" class="gh-link">L158</a><div class="mdoc"><p>Encoder. Note that this method is used by asr_inference.py</p>+<h4>Args:</h4>+<ul>+<li><code>speech</code> — (Batch, Length, ...)</li>+<li><code>speech_lengths</code> — (Batch, )</li>+<li><code>ind</code> — int</li>+</ul>+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def encode(+ self,+ speech: torch.Tensor,+ speech_lengths: torch.Tensor,+ **kwargs,+ ) -> Tuple[torch.Tensor, torch.Tensor]:+ """Encoder. Note that this method is used by asr_inference.py+ Args:+ speech: (Batch, Length, ...)+ speech_lengths: (Batch, )+ ind: int+ """+ with autocast(False):+ # Data augmentation+ if self.specaug is not None and self.training:+ speech, speech_lengths = self.specaug(speech, speech_lengths)++ # Normalization for feature: e.g. Global-CMVN, Utterance-CMVN+ if self.normalize is not None:+ speech, speech_lengths = self.normalize(speech, speech_lengths)++ # Forward encoder+ encoder_out, encoder_out2 = self.encoder(speech)+ encoder_out_lens = speech_lengths++ if isinstance(encoder_out, tuple):+ encoder_out = encoder_out[0]++ if isinstance(encoder_out2, tuple):+ encoder_out2 = encoder_out2[0]+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/fsmn_kws_mt/model.py#L158" target="_blank" class="src-more">View full source →</a></details></div><div class="mblk"><code>.inference(data_in, data_lengths, key, tokenizer, frontend, **kwargs)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/fsmn_kws_mt/model.py#L242" target="_blank" class="gh-link">L242</a><div class="mdoc"><p>Run inference on input data.</p>+<h4>Args:</h4>+<ul>+<li><code>data_in</code> — Input data (audio samples, file paths, or text).</li>+<li><code>data_lengths</code> — Lengths of each input sample in the batch.</li>+<li><code>key</code> — Sample identifiers.</li>+<li><code>tokenizer</code> — Tokenizer instance for text encoding/decoding.</li>+<li><code>frontend</code> — Audio frontend for feature extraction.</li>+<li><code>**kwargs</code> — Additional keyword arguments.</li>+</ul>+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def inference(+ self,+ data_in,+ data_lengths=None,+ key: list=None,+ tokenizer=None,+ frontend=None,+ **kwargs,+ ):+ """Run inference on input data.++ Args:+ data_in: Input data (audio samples, file paths, or text).+ data_lengths: Lengths of each input sample in the batch.+ key: Sample identifiers.+ tokenizer: Tokenizer instance for text encoding/decoding.+ frontend: Audio frontend for feature extraction.+ **kwargs: Additional keyword arguments.+ """+ keywords = kwargs.get("keywords")+ from funasr.utils.kws_utils import KwsCtcPrefixDecoder+ self.kws_decoder = KwsCtcPrefixDecoder(+ ctc=self.ctc,+ keywords=keywords,+ token_list=tokenizer[0].token_list,+ seg_dict=tokenizer[0].seg_dict,+ )+ self.kws_decoder2 = KwsCtcPrefixDecoder(+ ctc=self.ctc2,+ keywords=keywords,+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/fsmn_kws_mt/model.py#L242" target="_blank" class="src-more">View full source →</a></details></div></div>+<div class="api-detail" id="e77"><span class="dtag badge-m">method</span><h2><code>FsmnKWSMT.forward(speech, speech_lengths, text, text_lengths, text2, text2_lengths, **kwargs)</code></h2><div class="dmod">funasr.models.fsmn_kws_mt.FsmnKWSMT · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/fsmn_kws_mt/model.py#L106" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Encoder + Decoder + Calc loss</p>+<h4>Args:</h4>+<ul>+<li><code>speech</code> — (Batch, Length, ...)</li>+<li><code>speech_lengths</code> — (Batch, )</li>+<li><code>text</code> — (Batch, Length)</li>+<li><code>text_lengths</code> — (Batch,)</li>+<li><code>text2</code> — (Batch, Length)</li>+<li><code>text2_lengths</code> — (Batch,)</li>+</ul>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def forward(+ self,+ speech: torch.Tensor,+ speech_lengths: torch.Tensor,+ text: torch.Tensor,+ text_lengths: torch.Tensor,+ text2: torch.Tensor,+ text2_lengths: torch.Tensor,+ **kwargs,+ ) -> Tuple[torch.Tensor, Dict[str, torch.Tensor], torch.Tensor]:+ """Encoder + Decoder + Calc loss+ Args:+ speech: (Batch, Length, ...)+ speech_lengths: (Batch, )+ text: (Batch, Length)+ text_lengths: (Batch,)+ text2: (Batch, Length)+ text2_lengths: (Batch,)+ """+ if len(text_lengths.size()) > 1:+ text_lengths = text_lengths[:, 0]+ if len(speech_lengths.size()) > 1:+ speech_lengths = speech_lengths[:, 0]+ batch_size = speech.shape[0]++ # Encoder+ encoder_out, encoder_out2, encoder_out_lens = self.encode(speech, speech_lengths)++ loss_ctc, cer_ctc = self._calc_ctc_loss(+ encoder_out, encoder_out_lens, text, text_lengths+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/fsmn_kws_mt/model.py#L106" target="_blank" class="src-more">View full source on GitHub →</a></details></div>+<div class="api-detail" id="e78"><span class="dtag badge-m">method</span><h2><code>FsmnKWSMT.encode(speech, speech_lengths, **kwargs)</code></h2><div class="dmod">funasr.models.fsmn_kws_mt.FsmnKWSMT · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/fsmn_kws_mt/model.py#L158" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Encoder. Note that this method is used by asr_inference.py</p>+<h4>Args:</h4>+<ul>+<li><code>speech</code> — (Batch, Length, ...)</li>+<li><code>speech_lengths</code> — (Batch, )</li>+<li><code>ind</code> — int</li>+</ul>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def encode(+ self,+ speech: torch.Tensor,+ speech_lengths: torch.Tensor,+ **kwargs,+ ) -> Tuple[torch.Tensor, torch.Tensor]:+ """Encoder. Note that this method is used by asr_inference.py+ Args:+ speech: (Batch, Length, ...)+ speech_lengths: (Batch, )+ ind: int+ """+ with autocast(False):+ # Data augmentation+ if self.specaug is not None and self.training:+ speech, speech_lengths = self.specaug(speech, speech_lengths)++ # Normalization for feature: e.g. Global-CMVN, Utterance-CMVN+ if self.normalize is not None:+ speech, speech_lengths = self.normalize(speech, speech_lengths)++ # Forward encoder+ encoder_out, encoder_out2 = self.encoder(speech)+ encoder_out_lens = speech_lengths++ if isinstance(encoder_out, tuple):+ encoder_out = encoder_out[0]++ if isinstance(encoder_out2, tuple):+ encoder_out2 = encoder_out2[0]+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/fsmn_kws_mt/model.py#L158" target="_blank" class="src-more">View full source on GitHub →</a></details></div>+<div class="api-detail" id="e79"><span class="dtag badge-m">method</span><h2><code>FsmnKWSMT.inference(data_in, data_lengths, key, tokenizer, frontend, **kwargs)</code></h2><div class="dmod">funasr.models.fsmn_kws_mt.FsmnKWSMT · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/fsmn_kws_mt/model.py#L242" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Run inference on input data.</p>+<h4>Args:</h4>+<ul>+<li><code>data_in</code> — Input data (audio samples, file paths, or text).</li>+<li><code>data_lengths</code> — Lengths of each input sample in the batch.</li>+<li><code>key</code> — Sample identifiers.</li>+<li><code>tokenizer</code> — Tokenizer instance for text encoding/decoding.</li>+<li><code>frontend</code> — Audio frontend for feature extraction.</li>+<li><code>**kwargs</code> — Additional keyword arguments.</li>+</ul>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def inference(+ self,+ data_in,+ data_lengths=None,+ key: list=None,+ tokenizer=None,+ frontend=None,+ **kwargs,+ ):+ """Run inference on input data.++ Args:+ data_in: Input data (audio samples, file paths, or text).+ data_lengths: Lengths of each input sample in the batch.+ key: Sample identifiers.+ tokenizer: Tokenizer instance for text encoding/decoding.+ frontend: Audio frontend for feature extraction.+ **kwargs: Additional keyword arguments.+ """+ keywords = kwargs.get("keywords")+ from funasr.utils.kws_utils import KwsCtcPrefixDecoder+ self.kws_decoder = KwsCtcPrefixDecoder(+ ctc=self.ctc,+ keywords=keywords,+ token_list=tokenizer[0].token_list,+ seg_dict=tokenizer[0].seg_dict,+ )+ self.kws_decoder2 = KwsCtcPrefixDecoder(+ ctc=self.ctc2,+ keywords=keywords,+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/fsmn_kws_mt/model.py#L242" target="_blank" class="src-more">View full source on GitHub →</a></details></div>+<div class="api-detail" id="e80"><span class="dtag badge-c">class</span><h2>FsmnKWSMTConvert</h2><div class="dmod">funasr.models.fsmn_kws_mt · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/fsmn_kws_mt/model.py#L352" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><ul>+<li><code>Author</code> — Speech Lab of DAMO Academy, Alibaba Group</li>+<li><code>Deep</code> — FSMN for Large Vocabulary Continuous Speech Recognition</li>+</ul>+<p>https://arxiv.org/abs/1803.05030</p>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code>class FsmnKWSMTConvert(torch.nn.Module):+ """+ Author: Speech Lab of DAMO Academy, Alibaba Group+ Deep-FSMN for Large Vocabulary Continuous Speech Recognition+ https://arxiv.org/abs/1803.05030+ """++ def __init__(+ self,+ encoder: str = None,+ encoder_conf: Optional[Dict] = None,+ ctc_conf: Optional[Dict] = None,+ ctc_weight: float = 1.0,+ input_size: int = 360,+ blank_id: int = 0,+ **kwargs,+ ):+ """Initialize FsmnKWSMTConvert.++ Args:+ encoder: TODO.+ encoder_conf: Configuration dict for encoder.+ ctc_conf: Configuration dict for ctc.+ ctc_weight: TODO.+ input_size: Size/dimension parameter.+ blank_id: TODO.+ **kwargs: Additional keyword arguments.+ """+ super().__init__()++</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/fsmn_kws_mt/model.py#L352" target="_blank" class="src-more">View full source on GitHub →</a></details><h3 class="mt">Methods</h3><div class="mblk"><code>.to_kaldi_net()</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/fsmn_kws_mt/model.py#L390" target="_blank" class="gh-link">L390</a><div class="mdoc"><p>To kaldi net.</p>+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def to_kaldi_net(self):+ """To kaldi net."""+ return self.encoder.to_kaldi_net()+</code></pre></details></div><div class="mblk"><code>.to_kaldi_net2()</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/fsmn_kws_mt/model.py#L394" target="_blank" class="gh-link">L394</a><div class="mdoc"><p>To kaldi net2.</p>+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def to_kaldi_net2(self):+ """To kaldi net2."""+ return self.encoder.to_kaldi_net2()+</code></pre></details></div><div class="mblk"><code>.to_pytorch_net(kaldi_file)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/fsmn_kws_mt/model.py#L398" target="_blank" class="gh-link">L398</a><div class="mdoc"><p>To pytorch net.</p>+<h4>Args:</h4>+<ul>+<li><code>kaldi_file</code> — TODO.</li>+</ul>+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def to_pytorch_net(self, kaldi_file):+ """To pytorch net.++ Args:+ kaldi_file: TODO.+ """+ return self.encoder.to_pytorch_net(kaldi_file)+</code></pre></details></div></div>+<div class="api-detail" id="e81"><span class="dtag badge-m">method</span><h2><code>FsmnKWSMTConvert.to_kaldi_net()</code></h2><div class="dmod">funasr.models.fsmn_kws_mt.FsmnKWSMTConvert · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/fsmn_kws_mt/model.py#L390" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>To kaldi net.</p>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def to_kaldi_net(self):+ """To kaldi net."""+ return self.encoder.to_kaldi_net()+</code></pre></details></div>+<div class="api-detail" id="e82"><span class="dtag badge-m">method</span><h2><code>FsmnKWSMTConvert.to_kaldi_net2()</code></h2><div class="dmod">funasr.models.fsmn_kws_mt.FsmnKWSMTConvert · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/fsmn_kws_mt/model.py#L394" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>To kaldi net2.</p>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def to_kaldi_net2(self):+ """To kaldi net2."""+ return self.encoder.to_kaldi_net2()+</code></pre></details></div>+<div class="api-detail" id="e83"><span class="dtag badge-m">method</span><h2><code>FsmnKWSMTConvert.to_pytorch_net(kaldi_file)</code></h2><div class="dmod">funasr.models.fsmn_kws_mt.FsmnKWSMTConvert · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/fsmn_kws_mt/model.py#L398" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>To pytorch net.</p>+<h4>Args:</h4>+<ul>+<li><code>kaldi_file</code> — TODO.</li>+</ul>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def to_pytorch_net(self, kaldi_file):+ """To pytorch net.++ Args:+ kaldi_file: TODO.+ """+ return self.encoder.to_pytorch_net(kaldi_file)+</code></pre></details></div>+<div class="api-detail" id="e84"><span class="dtag badge-c">class</span><h2>FsmnVADStreaming</h2><div class="dmod">funasr.models.fsmn_vad_streaming · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/fsmn_vad_streaming/model.py#L345" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><ul>+<li><code>FSMN</code> — based Voice Activity Detection (streaming/offline).</li>+</ul>+<p>Detects speech segments in audio, returning start/end timestamps (milliseconds).</p>+<p>Supports both offline (full audio) and streaming (chunk-by-chunk) modes.</p>+<p>Offline output: [{"key": "...", "value": [[start_ms, end_ms], ...]}]</p>+<p>Streaming output: [[beg, -1]] (start), [[-1, end]] (end), [[beg, end]] (complete), [] (no event)</p>+<ul>+<li><code>Author</code> — Speech Lab of DAMO Academy, Alibaba Group</li>+<li><code>Deep</code> — FSMN for Large Vocabulary Continuous Speech Recognition</li>+</ul>+<p>https://arxiv.org/abs/1803.05030</p>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code>class FsmnVADStreaming(nn.Module):+ """FSMN-based Voice Activity Detection (streaming/offline).++ Detects speech segments in audio, returning start/end timestamps (milliseconds).+ Supports both offline (full audio) and streaming (chunk-by-chunk) modes.++ Offline output: [{"key": "...", "value": [[start_ms, end_ms], ...]}]+ Streaming output: [[beg, -1]] (start), [[-1, end]] (end), [[beg, end]] (complete), [] (no event)++ Author: Speech Lab of DAMO Academy, Alibaba Group+ Deep-FSMN for Large Vocabulary Continuous Speech Recognition+ https://arxiv.org/abs/1803.05030+ """++ def __init__(+ self,+ encoder: str = None,+ encoder_conf: Optional[Dict] = None,+ vad_post_args: Dict[str, Any] = None,+ **kwargs,+ ):+ """Initialize FsmnVADStreaming.++ Args:+ encoder: TODO.+ encoder_conf: Configuration dict for encoder.+ vad_post_args: TODO.+ **kwargs: Additional keyword arguments.+ """+ super().__init__()+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/fsmn_vad_streaming/model.py#L345" target="_blank" class="src-more">View full source on GitHub →</a></details><h3 class="mt">Methods</h3><div class="mblk"><code>.ResetDetection(cache)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/fsmn_vad_streaming/model.py#L382" target="_blank" class="gh-link">L382</a><div class="mdoc"><p>Resetdetection.</p>+<h4>Args:</h4>+<ul>+<li><code>cache</code> — State cache dict for streaming inference.</li>+</ul>+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def ResetDetection(self, cache: dict = None):+ """Resetdetection.++ Args:+ cache: State cache dict for streaming inference.+ """+ if cache is None:+ cache = {}+ cache["stats"].continous_silence_frame_count = 0+ cache["stats"].latest_confirmed_speech_frame = 0+ cache["stats"].lastest_confirmed_silence_frame = -1+ cache["stats"].confirmed_start_frame = -1+ cache["stats"].confirmed_end_frame = -1+ cache["stats"].vad_state_machine = VadStateMachine.kVadInStateStartPointNotDetected+ cache["windows_detector"].Reset()+ cache["stats"].sil_frame = 0+ cache["stats"].frame_probs = []++ if cache["stats"].output_data_buf:+ assert cache["stats"].output_data_buf[-1].contain_seg_end_point == True+ drop_frames = int(cache["stats"].output_data_buf[-1].end_ms / self.vad_opts.frame_in_ms)+ real_drop_frames = drop_frames - cache["stats"].last_drop_frames+ cache["stats"].last_drop_frames = drop_frames+ cache["stats"].data_buf_all = cache["stats"].data_buf_all[+ real_drop_frames+ * int(self.vad_opts.frame_in_ms * self.vad_opts.sample_rate / 1000) :+ ]+ cache["stats"].decibel = cache["stats"].decibel[real_drop_frames:]+ cache["stats"].scores = cache["stats"].scores[:, real_drop_frames:, :]+</code></pre></details></div><div class="mblk"><code>.ComputeDecibel(cache)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/fsmn_vad_streaming/model.py#L412" target="_blank" class="gh-link">L412</a><div class="mdoc"><p>Computedecibel.</p>+<h4>Args:</h4>+<ul>+<li><code>cache</code> — State cache dict for streaming inference.</li>+</ul>+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def ComputeDecibel(self, cache: dict = None) -> None:+ """Computedecibel.++ Args:+ cache: State cache dict for streaming inference.+ """+ if cache is None:+ cache = {}+ frame_sample_length = int(self.vad_opts.frame_length_ms * self.vad_opts.sample_rate / 1000)+ frame_shift_length = int(self.vad_opts.frame_in_ms * self.vad_opts.sample_rate / 1000)+ if cache["stats"].data_buf_all is None:+ cache["stats"].data_buf_all = cache["stats"].waveform[+ 0+ ] # cache["stats"].data_buf is pointed to cache["stats"].waveform[0]+ cache["stats"].data_buf = cache["stats"].data_buf_all+ else:+ cache["stats"].data_buf_all = torch.cat(+ (cache["stats"].data_buf_all, cache["stats"].waveform[0])+ )++ waveform_numpy = cache["stats"].waveform.numpy()++ offsets = np.arange(0, waveform_numpy.shape[1] - frame_sample_length + 1, frame_shift_length)+ frames = waveform_numpy[0, offsets[:, np.newaxis] + np.arange(frame_sample_length)]++ decibel_numpy = 10 * np.log10(np.sum(np.square(frames), axis=1) + 0.000001)+ decibel_numpy = decibel_numpy.tolist()++ cache["stats"].decibel.extend(decibel_numpy)+</code></pre></details></div><div class="mblk"><code>.ComputeScores(feats, cache)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/fsmn_vad_streaming/model.py#L443" target="_blank" class="gh-link">L443</a><div class="mdoc"><p>Computescores.</p>+<h4>Args:</h4>+<ul>+<li><code>feats</code> — Feature tensor (e.g., fbank), shape (batch, frames, dim).</li>+<li><code>cache</code> — State cache dict for streaming inference.</li>+</ul>+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def ComputeScores(self, feats: torch.Tensor, cache: dict = None) -> None:+ """Computescores.++ Args:+ feats: Feature tensor (e.g., fbank), shape (batch, frames, dim).+ cache: State cache dict for streaming inference.+ """+ if cache is None:+ cache = {}+ scores = self.encoder(feats, cache=cache["encoder"]) # return B * T * D+ assert (+ scores.shape[1] == feats.shape[1]+ ), "The shape between feats and scores does not match"+ self.vad_opts.nn_eval_block_size = scores.shape[1]+ cache["stats"].frm_cnt += scores.shape[1] # count total frames+ if cache["stats"].scores is None:+ cache["stats"].scores = scores # the first calculation+ else:+ cache["stats"].scores = torch.cat((cache["stats"].scores, scores), dim=1)+</code></pre></details></div><div class="mblk"><code>.PopDataBufTillFrame(frame_idx, cache)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/fsmn_vad_streaming/model.py#L463" target="_blank" class="gh-link">L463</a><div class="mdoc"><p>Popdatabuftillframe.</p>+<h4>Args:</h4>+<ul>+<li><code>frame_idx</code> — TODO.</li>+<li><code>cache</code> — State cache dict for streaming inference.</li>+</ul>+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def PopDataBufTillFrame(self, frame_idx: int, cache: dict = None) -> None: # need check again+ """Popdatabuftillframe.++ Args:+ frame_idx: TODO.+ cache: State cache dict for streaming inference.+ """+ while cache["stats"].data_buf_start_frame < frame_idx:+ if len(cache["stats"].data_buf) >= int(+ self.vad_opts.frame_in_ms * self.vad_opts.sample_rate / 1000+ ):+ if cache is None:+ cache = {}+ cache["stats"].data_buf_start_frame += 1+ cache["stats"].data_buf = cache["stats"].data_buf_all[+ (cache["stats"].data_buf_start_frame - cache["stats"].last_drop_frames)+ * int(self.vad_opts.frame_in_ms * self.vad_opts.sample_rate / 1000) :+ ]+</code></pre></details></div><div class="mblk"><code>.PopDataToOutputBuf(start_frm, frm_cnt, first_frm_is_start_point, last_frm_is_end_point, end_point_is_sent_end, cache)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/fsmn_vad_streaming/model.py#L482" target="_blank" class="gh-link">L482</a><div class="mdoc"><p>Popdatatooutputbuf.</p>+<h4>Args:</h4>+<ul>+<li><code>start_frm</code> — TODO.</li>+<li><code>frm_cnt</code> — TODO.</li>+<li><code>first_frm_is_start_point</code> — TODO.</li>+<li><code>last_frm_is_end_point</code> — TODO.</li>+<li><code>end_point_is_sent_end</code> — TODO.</li>+<li><code>cache</code> — State cache dict for streaming inference.</li>+</ul>+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def PopDataToOutputBuf(+ self,+ start_frm: int,+ frm_cnt: int,+ first_frm_is_start_point: bool,+ last_frm_is_end_point: bool,+ end_point_is_sent_end: bool,+ cache: dict = None,+ ) -> None:+ """Popdatatooutputbuf.++ Args:+ start_frm: TODO.+ frm_cnt: TODO.+ first_frm_is_start_point: TODO.+ last_frm_is_end_point: TODO.+ end_point_is_sent_end: TODO.+ cache: State cache dict for streaming inference.+ """+ if cache is None:+ cache = {}+ self.PopDataBufTillFrame(start_frm, cache=cache)+ expected_sample_number = int(+ frm_cnt * self.vad_opts.sample_rate * self.vad_opts.frame_in_ms / 1000+ )+ if last_frm_is_end_point:+ extra_sample = max(+ 0,+ int(+ self.vad_opts.frame_length_ms * self.vad_opts.sample_rate / 1000+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/fsmn_vad_streaming/model.py#L482" target="_blank" class="src-more">View full source →</a></details></div><div class="mblk"><code>.OnSilenceDetected(valid_frame, cache)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/fsmn_vad_streaming/model.py#L552" target="_blank" class="gh-link">L552</a><div class="mdoc"><p>Onsilencedetected.</p>+<h4>Args:</h4>+<ul>+<li><code>valid_frame</code> — TODO.</li>+<li><code>cache</code> — State cache dict for streaming inference.</li>+</ul>+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def OnSilenceDetected(self, valid_frame: int, cache: dict = None):+ """Onsilencedetected.++ Args:+ valid_frame: TODO.+ cache: State cache dict for streaming inference.+ """+ if cache is None:+ cache = {}+ cache["stats"].lastest_confirmed_silence_frame = valid_frame+ if cache["stats"].vad_state_machine == VadStateMachine.kVadInStateStartPointNotDetected:+ self.PopDataBufTillFrame(valid_frame, cache=cache)+</code></pre></details></div><div class="mblk"><code>.OnVoiceDetected(valid_frame, cache)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/fsmn_vad_streaming/model.py#L568" target="_blank" class="gh-link">L568</a><div class="mdoc"><p>Onvoicedetected.</p>+<h4>Args:</h4>+<ul>+<li><code>valid_frame</code> — TODO.</li>+<li><code>cache</code> — State cache dict for streaming inference.</li>+</ul>+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def OnVoiceDetected(self, valid_frame: int, cache: dict = None) -> None:+ """Onvoicedetected.++ Args:+ valid_frame: TODO.+ cache: State cache dict for streaming inference.+ """+ if cache is None:+ cache = {}+ cache["stats"].latest_confirmed_speech_frame = valid_frame+ self.PopDataToOutputBuf(valid_frame, 1, False, False, False, cache=cache)+</code></pre></details></div><div class="mblk"><code>.OnVoiceStart(start_frame, fake_result, cache)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/fsmn_vad_streaming/model.py#L580" target="_blank" class="gh-link">L580</a><div class="mdoc"><p>Onvoicestart.</p>+<h4>Args:</h4>+<ul>+<li><code>start_frame</code> — TODO.</li>+<li><code>fake_result</code> — TODO.</li>+<li><code>cache</code> — State cache dict for streaming inference.</li>+</ul>+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def OnVoiceStart(self, start_frame: int, fake_result: bool = False, cache: dict = None) -> None:+ """Onvoicestart.++ Args:+ start_frame: TODO.+ fake_result: TODO.+ cache: State cache dict for streaming inference.+ """+ if cache is None:+ cache = {}+ if self.vad_opts.do_start_point_detection:+ pass+ if cache["stats"].confirmed_start_frame != -1:+ print("not reset vad properly\n")+ else:+ cache["stats"].confirmed_start_frame = start_frame++ if (+ not fake_result+ and cache["stats"].vad_state_machine == VadStateMachine.kVadInStateStartPointNotDetected+ ):+ self.PopDataToOutputBuf(+ cache["stats"].confirmed_start_frame, 1, True, False, False, cache=cache+ )+</code></pre></details></div><div class="mblk"><code>.OnVoiceEnd(end_frame, fake_result, is_last_frame, cache)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/fsmn_vad_streaming/model.py#L605" target="_blank" class="gh-link">L605</a><div class="mdoc"><p>Onvoiceend.</p>+<h4>Args:</h4>+<ul>+<li><code>end_frame</code> — TODO.</li>+<li><code>fake_result</code> — TODO.</li>+<li><code>is_last_frame</code> — Boolean flag for last frame.</li>+<li><code>cache</code> — State cache dict for streaming inference.</li>+</ul>+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def OnVoiceEnd(+ self, end_frame: int, fake_result: bool, is_last_frame: bool, cache: dict = None+ ) -> None:+ """Onvoiceend.++ Args:+ end_frame: TODO.+ fake_result: TODO.+ is_last_frame: Boolean flag for last frame.+ cache: State cache dict for streaming inference.+ """+ if cache is None:+ cache = {}+ for t in range(cache["stats"].latest_confirmed_speech_frame + 1, end_frame):+ self.OnVoiceDetected(t, cache=cache)+ if self.vad_opts.do_end_point_detection:+ pass+ if cache["stats"].confirmed_end_frame != -1:+ print("not reset vad properly\n")+ else:+ cache["stats"].confirmed_end_frame = end_frame+ if not fake_result:+ cache["stats"].sil_frame = 0+ self.PopDataToOutputBuf(+ cache["stats"].confirmed_end_frame, 1, False, True, is_last_frame, cache=cache+ )+ cache["stats"].number_end_time_detected += 1+</code></pre></details></div><div class="mblk"><code>.MaybeOnVoiceEndIfLastFrame(is_final_frame, cur_frm_idx, cache)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/fsmn_vad_streaming/model.py#L633" target="_blank" class="gh-link">L633</a><div class="mdoc"><p>Maybeonvoiceendiflastframe.</p>+<h4>Args:</h4>+<ul>+<li><code>is_final_frame</code> — Boolean flag for final frame.</li>+<li><code>cur_frm_idx</code> — TODO.</li>+<li><code>cache</code> — State cache dict for streaming inference.</li>+</ul>+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def MaybeOnVoiceEndIfLastFrame(+ self, is_final_frame: bool, cur_frm_idx: int, cache: dict = None+ ) -> None:+ """Maybeonvoiceendiflastframe.++ Args:+ is_final_frame: Boolean flag for final frame.+ cur_frm_idx: TODO.+ cache: State cache dict for streaming inference.+ """+ if cache is None:+ cache = {}+ if is_final_frame:+ self.OnVoiceEnd(cur_frm_idx, False, True, cache=cache)+ cache["stats"].vad_state_machine = VadStateMachine.kVadInStateEndPointDetected+</code></pre></details></div><div class="mblk"><code>.GetLatency(cache)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/fsmn_vad_streaming/model.py#L649" target="_blank" class="gh-link">L649</a><div class="mdoc"><p>Getlatency.</p>+<h4>Args:</h4>+<ul>+<li><code>cache</code> — State cache dict for streaming inference.</li>+</ul>+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def GetLatency(self, cache: dict = None) -> int:+ """Getlatency.++ Args:+ cache: State cache dict for streaming inference.+ """+ if cache is None:+ cache = {}+ return int(self.LatencyFrmNumAtStartPoint(cache=cache) * self.vad_opts.frame_in_ms)+</code></pre></details></div><div class="mblk"><code>.LatencyFrmNumAtStartPoint(cache)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/fsmn_vad_streaming/model.py#L659" target="_blank" class="gh-link">L659</a><div class="mdoc"><p>Latencyfrmnumatstartpoint.</p>+<h4>Args:</h4>+<ul>+<li><code>cache</code> — State cache dict for streaming inference.</li>+</ul>+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def LatencyFrmNumAtStartPoint(self, cache: dict = None) -> int:+ """Latencyfrmnumatstartpoint.++ Args:+ cache: State cache dict for streaming inference.+ """+ if cache is None:+ cache = {}+ vad_latency = cache["windows_detector"].GetWinSize()+ if self.vad_opts.do_extend:+ vad_latency += int(self.vad_opts.lookback_time_start_point / self.vad_opts.frame_in_ms)+ return vad_latency+</code></pre></details></div><div class="mblk"><code>.GetFrameState(t, cache)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/fsmn_vad_streaming/model.py#L672" target="_blank" class="gh-link">L672</a><div class="mdoc"><p>Getframestate.</p>+<h4>Args:</h4>+<ul>+<li><code>t</code> — TODO.</li>+<li><code>cache</code> — State cache dict for streaming inference.</li>+</ul>+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def GetFrameState(self, t: int, cache: dict = None):+ """Getframestate.++ Args:+ t: TODO.+ cache: State cache dict for streaming inference.+ """+ if cache is None:+ cache = {}+ frame_state = FrameState.kFrameStateInvalid+ if t >= len(cache["stats"].decibel):+ return FrameState.kFrameStateSil+ cur_decibel = cache["stats"].decibel[t]+ cur_snr = cur_decibel - cache["stats"].noise_average_decibel+ # for each frame, calc log posterior probability of each state+ if cur_decibel < self.vad_opts.decibel_thres:+ frame_state = FrameState.kFrameStateSil+ self.DetectOneFrame(frame_state, t, False, cache=cache)+ return frame_state++ sum_score = 0.0+ noise_prob = 0.0+ assert len(cache["stats"].sil_pdf_ids) == self.vad_opts.silence_pdf_num+ if len(cache["stats"].sil_pdf_ids) > 0:+ assert len(cache["stats"].scores) == 1 # 只支持batch_size = 1的测试+ """+ - Change type of `sum_score` to float. The reason is that `sum_score` is a tensor with single element.+ and `torch.Tensor` is slower `float` when tensor has only one element.+ - Put the iteration of `sil_pdf_ids` inside `sum()` to reduce the overhead of creating a new list.+ - The default `sil_pdf_ids` is [0], the `if` statement is used to reduce the overhead of expression+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/fsmn_vad_streaming/model.py#L672" target="_blank" class="src-more">View full source →</a></details></div><div class="mblk"><code>.forward(feats, waveform, cache, is_final, **kwargs)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/fsmn_vad_streaming/model.py#L737" target="_blank" class="gh-link">L737</a><div class="mdoc"><p>Forward pass for training.</p>+<h4>Args:</h4>+<ul>+<li><code>feats</code> — Feature tensor (e.g., fbank), shape (batch, frames, dim).</li>+<li><code>waveform</code> — TODO.</li>+<li><code>cache</code> — State cache dict for streaming inference.</li>+<li><code>is_final</code> — Whether this is the final chunk in streaming.</li>+<li><code>**kwargs</code> — Additional keyword arguments.</li>+</ul>+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def forward(+ self,+ feats: torch.Tensor,+ waveform: torch.tensor,+ cache: dict = None,+ is_final: bool = False,+ **kwargs,+ ):+ """Forward pass for training.++ Args:+ feats: Feature tensor (e.g., fbank), shape (batch, frames, dim).+ waveform: TODO.+ cache: State cache dict for streaming inference.+ is_final: Whether this is the final chunk in streaming.+ **kwargs: Additional keyword arguments.+ """+ if cache is None:+ cache = {}+ # if len(cache) == 0:+ # self.AllResetDetection()+ # self.waveform = waveform # compute decibel for each frame+ cache["stats"].waveform = waveform+ is_streaming_input = kwargs.get("is_streaming_input", True)+ self.ComputeDecibel(cache=cache)+ self.ComputeScores(feats, cache=cache)+ if not is_final:+ self.DetectCommonFrames(cache=cache)+ else:+ self.DetectLastFrames(cache=cache)+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/fsmn_vad_streaming/model.py#L737" target="_blank" class="src-more">View full source →</a></details></div><div class="mblk"><code>.init_cache(cache, **kwargs)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/fsmn_vad_streaming/model.py#L820" target="_blank" class="gh-link">L820</a><div class="mdoc"><p>Init cache.</p>+<h4>Args:</h4>+<ul>+<li><code>cache</code> — State cache dict for streaming inference.</li>+<li><code>**kwargs</code> — Additional keyword arguments.</li>+</ul>+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def init_cache(self, cache: dict = None, **kwargs):+ """Init cache.++ Args:+ cache: State cache dict for streaming inference.+ **kwargs: Additional keyword arguments.+ """+ if cache is None:+ cache = {}++ cache["frontend"] = {}+ cache["prev_samples"] = torch.empty(0)+ cache["encoder"] = {}++ if kwargs.get("max_end_silence_time") is not None:+ # update the max_end_silence_time+ self.vad_opts.max_end_silence_time = kwargs.get("max_end_silence_time")++ windows_detector = WindowDetector(+ self.vad_opts.window_size_ms,+ self.vad_opts.sil_to_speech_time_thres,+ self.vad_opts.speech_to_sil_time_thres,+ self.vad_opts.frame_in_ms,+ )+ windows_detector.Reset()++ stats = Stats(+ sil_pdf_ids=self.vad_opts.sil_pdf_ids,+ max_end_sil_frame_cnt_thresh=self.vad_opts.max_end_silence_time+ - self.vad_opts.speech_to_sil_time_thres,+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/fsmn_vad_streaming/model.py#L820" target="_blank" class="src-more">View full source →</a></details></div><div class="mblk"><code>.inference(data_in, data_lengths, key, tokenizer, frontend, cache, **kwargs)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/fsmn_vad_streaming/model.py#L856" target="_blank" class="gh-link">L856</a><div class="mdoc"><p>Run inference on input data.</p>+<h4>Args:</h4>+<ul>+<li><code>data_in</code> — Input data (audio samples, file paths, or text).</li>+<li><code>data_lengths</code> — Lengths of each input sample in the batch.</li>+<li><code>key</code> — Sample identifiers.</li>+<li><code>tokenizer</code> — Tokenizer instance for text encoding/decoding.</li>+<li><code>frontend</code> — Audio frontend for feature extraction.</li>+<li><code>cache</code> — State cache dict for streaming inference.</li>+<li><code>**kwargs</code> — Additional keyword arguments.</li>+</ul>+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def inference(+ self,+ data_in,+ data_lengths=None,+ key: list = None,+ tokenizer=None,+ frontend=None,+ cache: dict = None,+ **kwargs,+ ):+ """Run inference on input data.++ Args:+ data_in: Input data (audio samples, file paths, or text).+ data_lengths: Lengths of each input sample in the batch.+ key: Sample identifiers.+ tokenizer: Tokenizer instance for text encoding/decoding.+ frontend: Audio frontend for feature extraction.+ cache: State cache dict for streaming inference.+ **kwargs: Additional keyword arguments.+ """+ if cache is None:+ cache = {}+ if len(cache) == 0:+ self.init_cache(cache, **kwargs)++ meta_data = {}+ chunk_size = kwargs.get("chunk_size", 60000) # 50ms+ chunk_stride_samples = int(chunk_size * frontend.fs / 1000)++</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/fsmn_vad_streaming/model.py#L856" target="_blank" class="src-more">View full source →</a></details></div><div class="mblk"><code>.export(**kwargs)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/fsmn_vad_streaming/model.py#L970" target="_blank" class="gh-link">L970</a><div class="mdoc"><p>Export.</p>+<h4>Args:</h4>+<ul>+<li><code>**kwargs</code> — Additional keyword arguments.</li>+</ul>+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def export(self, **kwargs):++ """Export.++ Args:+ **kwargs: Additional keyword arguments.+ """+ from .export_meta import export_rebuild_model++ models = export_rebuild_model(model=self, **kwargs)+ return models+</code></pre></details></div><div class="mblk"><code>.DetectCommonFrames(cache)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/fsmn_vad_streaming/model.py#L982" target="_blank" class="gh-link">L982</a><div class="mdoc"><p>Detectcommonframes.</p>+<h4>Args:</h4>+<ul>+<li><code>cache</code> — State cache dict for streaming inference.</li>+</ul>+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def DetectCommonFrames(self, cache: dict = None) -> int:+ """Detectcommonframes.++ Args:+ cache: State cache dict for streaming inference.+ """+ if cache is None:+ cache = {}+ if cache["stats"].vad_state_machine == VadStateMachine.kVadInStateEndPointDetected:+ return 0+ for i in range(self.vad_opts.nn_eval_block_size - 1, -1, -1):+ frame_state = FrameState.kFrameStateInvalid+ frame_state = self.GetFrameState(+ cache["stats"].frm_cnt - 1 - i - cache["stats"].last_drop_frames, cache=cache+ )+ self.DetectOneFrame(frame_state, cache["stats"].frm_cnt - 1 - i, False, cache=cache)++ return 0+</code></pre></details></div><div class="mblk"><code>.DetectLastFrames(cache)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/fsmn_vad_streaming/model.py#L1001" target="_blank" class="gh-link">L1001</a><div class="mdoc"><p>Detectlastframes.</p>+<h4>Args:</h4>+<ul>+<li><code>cache</code> — State cache dict for streaming inference.</li>+</ul>+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def DetectLastFrames(self, cache: dict = None) -> int:+ """Detectlastframes.++ Args:+ cache: State cache dict for streaming inference.+ """+ if cache is None:+ cache = {}+ if cache["stats"].vad_state_machine == VadStateMachine.kVadInStateEndPointDetected:+ return 0+ for i in range(self.vad_opts.nn_eval_block_size - 1, -1, -1):+ frame_state = FrameState.kFrameStateInvalid+ frame_state = self.GetFrameState(+ cache["stats"].frm_cnt - 1 - i - cache["stats"].last_drop_frames, cache=cache+ )+ if i != 0:+ self.DetectOneFrame(frame_state, cache["stats"].frm_cnt - 1 - i, False, cache=cache)+ else:+ self.DetectOneFrame(frame_state, cache["stats"].frm_cnt - 1, True, cache=cache)++ return 0+</code></pre></details></div><div class="mblk"><code>.DetectOneFrame(cur_frm_state, cur_frm_idx, is_final_frame, cache)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/fsmn_vad_streaming/model.py#L1023" target="_blank" class="gh-link">L1023</a><div class="mdoc"><p>Detectoneframe.</p>+<h4>Args:</h4>+<ul>+<li><code>cur_frm_state</code> — TODO.</li>+<li><code>cur_frm_idx</code> — TODO.</li>+<li><code>is_final_frame</code> — Boolean flag for final frame.</li>+<li><code>cache</code> — State cache dict for streaming inference.</li>+</ul>+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def DetectOneFrame(+ self, cur_frm_state: FrameState, cur_frm_idx: int, is_final_frame: bool, cache: dict = None+ ) -> None:+ """Detectoneframe.++ Args:+ cur_frm_state: TODO.+ cur_frm_idx: TODO.+ is_final_frame: Boolean flag for final frame.+ cache: State cache dict for streaming inference.+ """+ if cache is None:+ cache = {}+ tmp_cur_frm_state = FrameState.kFrameStateInvalid+ if cur_frm_state == FrameState.kFrameStateSpeech:+ if math.fabs(1.0) > self.vad_opts.fe_prior_thres:+ tmp_cur_frm_state = FrameState.kFrameStateSpeech+ else:+ tmp_cur_frm_state = FrameState.kFrameStateSil+ elif cur_frm_state == FrameState.kFrameStateSil:+ tmp_cur_frm_state = FrameState.kFrameStateSil+ state_change = cache["windows_detector"].DetectOneFrame(+ tmp_cur_frm_state, cur_frm_idx, cache=cache+ )+ frm_shift_in_ms = self.vad_opts.frame_in_ms+ if AudioChangeState.kChangeStateSil2Speech == state_change:+ silence_frame_count = cache["stats"].continous_silence_frame_count+ cache["stats"].continous_silence_frame_count = 0+ cache["stats"].pre_end_silence_detected = False+ start_frame = 0+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/fsmn_vad_streaming/model.py#L1023" target="_blank" class="src-more">View full source →</a></details></div></div>+<div class="api-detail" id="e85"><span class="dtag badge-m">method</span><h2><code>FsmnVADStreaming.ResetDetection(cache)</code></h2><div class="dmod">funasr.models.fsmn_vad_streaming.FsmnVADStreaming · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/fsmn_vad_streaming/model.py#L382" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Resetdetection.</p>+<h4>Args:</h4>+<ul>+<li><code>cache</code> — State cache dict for streaming inference.</li>+</ul>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def ResetDetection(self, cache: dict = None):+ """Resetdetection.++ Args:+ cache: State cache dict for streaming inference.+ """+ if cache is None:+ cache = {}+ cache["stats"].continous_silence_frame_count = 0+ cache["stats"].latest_confirmed_speech_frame = 0+ cache["stats"].lastest_confirmed_silence_frame = -1+ cache["stats"].confirmed_start_frame = -1+ cache["stats"].confirmed_end_frame = -1+ cache["stats"].vad_state_machine = VadStateMachine.kVadInStateStartPointNotDetected+ cache["windows_detector"].Reset()+ cache["stats"].sil_frame = 0+ cache["stats"].frame_probs = []++ if cache["stats"].output_data_buf:+ assert cache["stats"].output_data_buf[-1].contain_seg_end_point == True+ drop_frames = int(cache["stats"].output_data_buf[-1].end_ms / self.vad_opts.frame_in_ms)+ real_drop_frames = drop_frames - cache["stats"].last_drop_frames+ cache["stats"].last_drop_frames = drop_frames+ cache["stats"].data_buf_all = cache["stats"].data_buf_all[+ real_drop_frames+ * int(self.vad_opts.frame_in_ms * self.vad_opts.sample_rate / 1000) :+ ]+ cache["stats"].decibel = cache["stats"].decibel[real_drop_frames:]+ cache["stats"].scores = cache["stats"].scores[:, real_drop_frames:, :]+</code></pre></details></div>+<div class="api-detail" id="e86"><span class="dtag badge-m">method</span><h2><code>FsmnVADStreaming.ComputeDecibel(cache)</code></h2><div class="dmod">funasr.models.fsmn_vad_streaming.FsmnVADStreaming · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/fsmn_vad_streaming/model.py#L412" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Computedecibel.</p>+<h4>Args:</h4>+<ul>+<li><code>cache</code> — State cache dict for streaming inference.</li>+</ul>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def ComputeDecibel(self, cache: dict = None) -> None:+ """Computedecibel.++ Args:+ cache: State cache dict for streaming inference.+ """+ if cache is None:+ cache = {}+ frame_sample_length = int(self.vad_opts.frame_length_ms * self.vad_opts.sample_rate / 1000)+ frame_shift_length = int(self.vad_opts.frame_in_ms * self.vad_opts.sample_rate / 1000)+ if cache["stats"].data_buf_all is None:+ cache["stats"].data_buf_all = cache["stats"].waveform[+ 0+ ] # cache["stats"].data_buf is pointed to cache["stats"].waveform[0]+ cache["stats"].data_buf = cache["stats"].data_buf_all+ else:+ cache["stats"].data_buf_all = torch.cat(+ (cache["stats"].data_buf_all, cache["stats"].waveform[0])+ )++ waveform_numpy = cache["stats"].waveform.numpy()++ offsets = np.arange(0, waveform_numpy.shape[1] - frame_sample_length + 1, frame_shift_length)+ frames = waveform_numpy[0, offsets[:, np.newaxis] + np.arange(frame_sample_length)]++ decibel_numpy = 10 * np.log10(np.sum(np.square(frames), axis=1) + 0.000001)+ decibel_numpy = decibel_numpy.tolist()++ cache["stats"].decibel.extend(decibel_numpy)+</code></pre></details></div>+<div class="api-detail" id="e87"><span class="dtag badge-m">method</span><h2><code>FsmnVADStreaming.ComputeScores(feats, cache)</code></h2><div class="dmod">funasr.models.fsmn_vad_streaming.FsmnVADStreaming · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/fsmn_vad_streaming/model.py#L443" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Computescores.</p>+<h4>Args:</h4>+<ul>+<li><code>feats</code> — Feature tensor (e.g., fbank), shape (batch, frames, dim).</li>+<li><code>cache</code> — State cache dict for streaming inference.</li>+</ul>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def ComputeScores(self, feats: torch.Tensor, cache: dict = None) -> None:+ """Computescores.++ Args:+ feats: Feature tensor (e.g., fbank), shape (batch, frames, dim).+ cache: State cache dict for streaming inference.+ """+ if cache is None:+ cache = {}+ scores = self.encoder(feats, cache=cache["encoder"]) # return B * T * D+ assert (+ scores.shape[1] == feats.shape[1]+ ), "The shape between feats and scores does not match"+ self.vad_opts.nn_eval_block_size = scores.shape[1]+ cache["stats"].frm_cnt += scores.shape[1] # count total frames+ if cache["stats"].scores is None:+ cache["stats"].scores = scores # the first calculation+ else:+ cache["stats"].scores = torch.cat((cache["stats"].scores, scores), dim=1)+</code></pre></details></div>+<div class="api-detail" id="e88"><span class="dtag badge-m">method</span><h2><code>FsmnVADStreaming.PopDataBufTillFrame(frame_idx, cache)</code></h2><div class="dmod">funasr.models.fsmn_vad_streaming.FsmnVADStreaming · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/fsmn_vad_streaming/model.py#L463" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Popdatabuftillframe.</p>+<h4>Args:</h4>+<ul>+<li><code>frame_idx</code> — TODO.</li>+<li><code>cache</code> — State cache dict for streaming inference.</li>+</ul>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def PopDataBufTillFrame(self, frame_idx: int, cache: dict = None) -> None: # need check again+ """Popdatabuftillframe.++ Args:+ frame_idx: TODO.+ cache: State cache dict for streaming inference.+ """+ while cache["stats"].data_buf_start_frame < frame_idx:+ if len(cache["stats"].data_buf) >= int(+ self.vad_opts.frame_in_ms * self.vad_opts.sample_rate / 1000+ ):+ if cache is None:+ cache = {}+ cache["stats"].data_buf_start_frame += 1+ cache["stats"].data_buf = cache["stats"].data_buf_all[+ (cache["stats"].data_buf_start_frame - cache["stats"].last_drop_frames)+ * int(self.vad_opts.frame_in_ms * self.vad_opts.sample_rate / 1000) :+ ]+</code></pre></details></div>+<div class="api-detail" id="e89"><span class="dtag badge-m">method</span><h2><code>FsmnVADStreaming.PopDataToOutputBuf(start_frm, frm_cnt, first_frm_is_start_point, last_frm_is_end_point, end_point_is_sent_end, cache)</code></h2><div class="dmod">funasr.models.fsmn_vad_streaming.FsmnVADStreaming · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/fsmn_vad_streaming/model.py#L482" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Popdatatooutputbuf.</p>+<h4>Args:</h4>+<ul>+<li><code>start_frm</code> — TODO.</li>+<li><code>frm_cnt</code> — TODO.</li>+<li><code>first_frm_is_start_point</code> — TODO.</li>+<li><code>last_frm_is_end_point</code> — TODO.</li>+<li><code>end_point_is_sent_end</code> — TODO.</li>+<li><code>cache</code> — State cache dict for streaming inference.</li>+</ul>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def PopDataToOutputBuf(+ self,+ start_frm: int,+ frm_cnt: int,+ first_frm_is_start_point: bool,+ last_frm_is_end_point: bool,+ end_point_is_sent_end: bool,+ cache: dict = None,+ ) -> None:+ """Popdatatooutputbuf.++ Args:+ start_frm: TODO.+ frm_cnt: TODO.+ first_frm_is_start_point: TODO.+ last_frm_is_end_point: TODO.+ end_point_is_sent_end: TODO.+ cache: State cache dict for streaming inference.+ """+ if cache is None:+ cache = {}+ self.PopDataBufTillFrame(start_frm, cache=cache)+ expected_sample_number = int(+ frm_cnt * self.vad_opts.sample_rate * self.vad_opts.frame_in_ms / 1000+ )+ if last_frm_is_end_point:+ extra_sample = max(+ 0,+ int(+ self.vad_opts.frame_length_ms * self.vad_opts.sample_rate / 1000+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/fsmn_vad_streaming/model.py#L482" target="_blank" class="src-more">View full source on GitHub →</a></details></div>+<div class="api-detail" id="e90"><span class="dtag badge-m">method</span><h2><code>FsmnVADStreaming.OnSilenceDetected(valid_frame, cache)</code></h2><div class="dmod">funasr.models.fsmn_vad_streaming.FsmnVADStreaming · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/fsmn_vad_streaming/model.py#L552" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Onsilencedetected.</p>+<h4>Args:</h4>+<ul>+<li><code>valid_frame</code> — TODO.</li>+<li><code>cache</code> — State cache dict for streaming inference.</li>+</ul>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def OnSilenceDetected(self, valid_frame: int, cache: dict = None):+ """Onsilencedetected.++ Args:+ valid_frame: TODO.+ cache: State cache dict for streaming inference.+ """+ if cache is None:+ cache = {}+ cache["stats"].lastest_confirmed_silence_frame = valid_frame+ if cache["stats"].vad_state_machine == VadStateMachine.kVadInStateStartPointNotDetected:+ self.PopDataBufTillFrame(valid_frame, cache=cache)+</code></pre></details></div>+<div class="api-detail" id="e91"><span class="dtag badge-m">method</span><h2><code>FsmnVADStreaming.OnVoiceDetected(valid_frame, cache)</code></h2><div class="dmod">funasr.models.fsmn_vad_streaming.FsmnVADStreaming · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/fsmn_vad_streaming/model.py#L568" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Onvoicedetected.</p>+<h4>Args:</h4>+<ul>+<li><code>valid_frame</code> — TODO.</li>+<li><code>cache</code> — State cache dict for streaming inference.</li>+</ul>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def OnVoiceDetected(self, valid_frame: int, cache: dict = None) -> None:+ """Onvoicedetected.++ Args:+ valid_frame: TODO.+ cache: State cache dict for streaming inference.+ """+ if cache is None:+ cache = {}+ cache["stats"].latest_confirmed_speech_frame = valid_frame+ self.PopDataToOutputBuf(valid_frame, 1, False, False, False, cache=cache)+</code></pre></details></div>+<div class="api-detail" id="e92"><span class="dtag badge-m">method</span><h2><code>FsmnVADStreaming.OnVoiceStart(start_frame, fake_result, cache)</code></h2><div class="dmod">funasr.models.fsmn_vad_streaming.FsmnVADStreaming · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/fsmn_vad_streaming/model.py#L580" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Onvoicestart.</p>+<h4>Args:</h4>+<ul>+<li><code>start_frame</code> — TODO.</li>+<li><code>fake_result</code> — TODO.</li>+<li><code>cache</code> — State cache dict for streaming inference.</li>+</ul>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def OnVoiceStart(self, start_frame: int, fake_result: bool = False, cache: dict = None) -> None:+ """Onvoicestart.++ Args:+ start_frame: TODO.+ fake_result: TODO.+ cache: State cache dict for streaming inference.+ """+ if cache is None:+ cache = {}+ if self.vad_opts.do_start_point_detection:+ pass+ if cache["stats"].confirmed_start_frame != -1:+ print("not reset vad properly\n")+ else:+ cache["stats"].confirmed_start_frame = start_frame++ if (+ not fake_result+ and cache["stats"].vad_state_machine == VadStateMachine.kVadInStateStartPointNotDetected+ ):+ self.PopDataToOutputBuf(+ cache["stats"].confirmed_start_frame, 1, True, False, False, cache=cache+ )+</code></pre></details></div>+<div class="api-detail" id="e93"><span class="dtag badge-m">method</span><h2><code>FsmnVADStreaming.OnVoiceEnd(end_frame, fake_result, is_last_frame, cache)</code></h2><div class="dmod">funasr.models.fsmn_vad_streaming.FsmnVADStreaming · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/fsmn_vad_streaming/model.py#L605" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Onvoiceend.</p>+<h4>Args:</h4>+<ul>+<li><code>end_frame</code> — TODO.</li>+<li><code>fake_result</code> — TODO.</li>+<li><code>is_last_frame</code> — Boolean flag for last frame.</li>+<li><code>cache</code> — State cache dict for streaming inference.</li>+</ul>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def OnVoiceEnd(+ self, end_frame: int, fake_result: bool, is_last_frame: bool, cache: dict = None+ ) -> None:+ """Onvoiceend.++ Args:+ end_frame: TODO.+ fake_result: TODO.+ is_last_frame: Boolean flag for last frame.+ cache: State cache dict for streaming inference.+ """+ if cache is None:+ cache = {}+ for t in range(cache["stats"].latest_confirmed_speech_frame + 1, end_frame):+ self.OnVoiceDetected(t, cache=cache)+ if self.vad_opts.do_end_point_detection:+ pass+ if cache["stats"].confirmed_end_frame != -1:+ print("not reset vad properly\n")+ else:+ cache["stats"].confirmed_end_frame = end_frame+ if not fake_result:+ cache["stats"].sil_frame = 0+ self.PopDataToOutputBuf(+ cache["stats"].confirmed_end_frame, 1, False, True, is_last_frame, cache=cache+ )+ cache["stats"].number_end_time_detected += 1+</code></pre></details></div>+<div class="api-detail" id="e94"><span class="dtag badge-m">method</span><h2><code>FsmnVADStreaming.MaybeOnVoiceEndIfLastFrame(is_final_frame, cur_frm_idx, cache)</code></h2><div class="dmod">funasr.models.fsmn_vad_streaming.FsmnVADStreaming · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/fsmn_vad_streaming/model.py#L633" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Maybeonvoiceendiflastframe.</p>+<h4>Args:</h4>+<ul>+<li><code>is_final_frame</code> — Boolean flag for final frame.</li>+<li><code>cur_frm_idx</code> — TODO.</li>+<li><code>cache</code> — State cache dict for streaming inference.</li>+</ul>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def MaybeOnVoiceEndIfLastFrame(+ self, is_final_frame: bool, cur_frm_idx: int, cache: dict = None+ ) -> None:+ """Maybeonvoiceendiflastframe.++ Args:+ is_final_frame: Boolean flag for final frame.+ cur_frm_idx: TODO.+ cache: State cache dict for streaming inference.+ """+ if cache is None:+ cache = {}+ if is_final_frame:+ self.OnVoiceEnd(cur_frm_idx, False, True, cache=cache)+ cache["stats"].vad_state_machine = VadStateMachine.kVadInStateEndPointDetected+</code></pre></details></div>+<div class="api-detail" id="e95"><span class="dtag badge-m">method</span><h2><code>FsmnVADStreaming.GetLatency(cache)</code></h2><div class="dmod">funasr.models.fsmn_vad_streaming.FsmnVADStreaming · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/fsmn_vad_streaming/model.py#L649" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Getlatency.</p>+<h4>Args:</h4>+<ul>+<li><code>cache</code> — State cache dict for streaming inference.</li>+</ul>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def GetLatency(self, cache: dict = None) -> int:+ """Getlatency.++ Args:+ cache: State cache dict for streaming inference.+ """+ if cache is None:+ cache = {}+ return int(self.LatencyFrmNumAtStartPoint(cache=cache) * self.vad_opts.frame_in_ms)+</code></pre></details></div>+<div class="api-detail" id="e96"><span class="dtag badge-m">method</span><h2><code>FsmnVADStreaming.LatencyFrmNumAtStartPoint(cache)</code></h2><div class="dmod">funasr.models.fsmn_vad_streaming.FsmnVADStreaming · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/fsmn_vad_streaming/model.py#L659" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Latencyfrmnumatstartpoint.</p>+<h4>Args:</h4>+<ul>+<li><code>cache</code> — State cache dict for streaming inference.</li>+</ul>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def LatencyFrmNumAtStartPoint(self, cache: dict = None) -> int:+ """Latencyfrmnumatstartpoint.++ Args:+ cache: State cache dict for streaming inference.+ """+ if cache is None:+ cache = {}+ vad_latency = cache["windows_detector"].GetWinSize()+ if self.vad_opts.do_extend:+ vad_latency += int(self.vad_opts.lookback_time_start_point / self.vad_opts.frame_in_ms)+ return vad_latency+</code></pre></details></div>+<div class="api-detail" id="e97"><span class="dtag badge-m">method</span><h2><code>FsmnVADStreaming.GetFrameState(t, cache)</code></h2><div class="dmod">funasr.models.fsmn_vad_streaming.FsmnVADStreaming · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/fsmn_vad_streaming/model.py#L672" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Getframestate.</p>+<h4>Args:</h4>+<ul>+<li><code>t</code> — TODO.</li>+<li><code>cache</code> — State cache dict for streaming inference.</li>+</ul>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def GetFrameState(self, t: int, cache: dict = None):+ """Getframestate.++ Args:+ t: TODO.+ cache: State cache dict for streaming inference.+ """+ if cache is None:+ cache = {}+ frame_state = FrameState.kFrameStateInvalid+ if t >= len(cache["stats"].decibel):+ return FrameState.kFrameStateSil+ cur_decibel = cache["stats"].decibel[t]+ cur_snr = cur_decibel - cache["stats"].noise_average_decibel+ # for each frame, calc log posterior probability of each state+ if cur_decibel < self.vad_opts.decibel_thres:+ frame_state = FrameState.kFrameStateSil+ self.DetectOneFrame(frame_state, t, False, cache=cache)+ return frame_state++ sum_score = 0.0+ noise_prob = 0.0+ assert len(cache["stats"].sil_pdf_ids) == self.vad_opts.silence_pdf_num+ if len(cache["stats"].sil_pdf_ids) > 0:+ assert len(cache["stats"].scores) == 1 # 只支持batch_size = 1的测试+ """+ - Change type of `sum_score` to float. The reason is that `sum_score` is a tensor with single element.+ and `torch.Tensor` is slower `float` when tensor has only one element.+ - Put the iteration of `sil_pdf_ids` inside `sum()` to reduce the overhead of creating a new list.+ - The default `sil_pdf_ids` is [0], the `if` statement is used to reduce the overhead of expression+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/fsmn_vad_streaming/model.py#L672" target="_blank" class="src-more">View full source on GitHub →</a></details></div>+<div class="api-detail" id="e98"><span class="dtag badge-m">method</span><h2><code>FsmnVADStreaming.forward(feats, waveform, cache, is_final, **kwargs)</code></h2><div class="dmod">funasr.models.fsmn_vad_streaming.FsmnVADStreaming · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/fsmn_vad_streaming/model.py#L737" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Forward pass for training.</p>+<h4>Args:</h4>+<ul>+<li><code>feats</code> — Feature tensor (e.g., fbank), shape (batch, frames, dim).</li>+<li><code>waveform</code> — TODO.</li>+<li><code>cache</code> — State cache dict for streaming inference.</li>+<li><code>is_final</code> — Whether this is the final chunk in streaming.</li>+<li><code>**kwargs</code> — Additional keyword arguments.</li>+</ul>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def forward(+ self,+ feats: torch.Tensor,+ waveform: torch.tensor,+ cache: dict = None,+ is_final: bool = False,+ **kwargs,+ ):+ """Forward pass for training.++ Args:+ feats: Feature tensor (e.g., fbank), shape (batch, frames, dim).+ waveform: TODO.+ cache: State cache dict for streaming inference.+ is_final: Whether this is the final chunk in streaming.+ **kwargs: Additional keyword arguments.+ """+ if cache is None:+ cache = {}+ # if len(cache) == 0:+ # self.AllResetDetection()+ # self.waveform = waveform # compute decibel for each frame+ cache["stats"].waveform = waveform+ is_streaming_input = kwargs.get("is_streaming_input", True)+ self.ComputeDecibel(cache=cache)+ self.ComputeScores(feats, cache=cache)+ if not is_final:+ self.DetectCommonFrames(cache=cache)+ else:+ self.DetectLastFrames(cache=cache)+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/fsmn_vad_streaming/model.py#L737" target="_blank" class="src-more">View full source on GitHub →</a></details></div>+<div class="api-detail" id="e99"><span class="dtag badge-m">method</span><h2><code>FsmnVADStreaming.init_cache(cache, **kwargs)</code></h2><div class="dmod">funasr.models.fsmn_vad_streaming.FsmnVADStreaming · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/fsmn_vad_streaming/model.py#L820" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Init cache.</p>+<h4>Args:</h4>+<ul>+<li><code>cache</code> — State cache dict for streaming inference.</li>+<li><code>**kwargs</code> — Additional keyword arguments.</li>+</ul>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def init_cache(self, cache: dict = None, **kwargs):+ """Init cache.++ Args:+ cache: State cache dict for streaming inference.+ **kwargs: Additional keyword arguments.+ """+ if cache is None:+ cache = {}++ cache["frontend"] = {}+ cache["prev_samples"] = torch.empty(0)+ cache["encoder"] = {}++ if kwargs.get("max_end_silence_time") is not None:+ # update the max_end_silence_time+ self.vad_opts.max_end_silence_time = kwargs.get("max_end_silence_time")++ windows_detector = WindowDetector(+ self.vad_opts.window_size_ms,+ self.vad_opts.sil_to_speech_time_thres,+ self.vad_opts.speech_to_sil_time_thres,+ self.vad_opts.frame_in_ms,+ )+ windows_detector.Reset()++ stats = Stats(+ sil_pdf_ids=self.vad_opts.sil_pdf_ids,+ max_end_sil_frame_cnt_thresh=self.vad_opts.max_end_silence_time+ - self.vad_opts.speech_to_sil_time_thres,+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/fsmn_vad_streaming/model.py#L820" target="_blank" class="src-more">View full source on GitHub →</a></details></div>+<div class="api-detail" id="e100"><span class="dtag badge-m">method</span><h2><code>FsmnVADStreaming.inference(data_in, data_lengths, key, tokenizer, frontend, cache, **kwargs)</code></h2><div class="dmod">funasr.models.fsmn_vad_streaming.FsmnVADStreaming · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/fsmn_vad_streaming/model.py#L856" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Run inference on input data.</p>+<h4>Args:</h4>+<ul>+<li><code>data_in</code> — Input data (audio samples, file paths, or text).</li>+<li><code>data_lengths</code> — Lengths of each input sample in the batch.</li>+<li><code>key</code> — Sample identifiers.</li>+<li><code>tokenizer</code> — Tokenizer instance for text encoding/decoding.</li>+<li><code>frontend</code> — Audio frontend for feature extraction.</li>+<li><code>cache</code> — State cache dict for streaming inference.</li>+<li><code>**kwargs</code> — Additional keyword arguments.</li>+</ul>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def inference(+ self,+ data_in,+ data_lengths=None,+ key: list = None,+ tokenizer=None,+ frontend=None,+ cache: dict = None,+ **kwargs,+ ):+ """Run inference on input data.++ Args:+ data_in: Input data (audio samples, file paths, or text).+ data_lengths: Lengths of each input sample in the batch.+ key: Sample identifiers.+ tokenizer: Tokenizer instance for text encoding/decoding.+ frontend: Audio frontend for feature extraction.+ cache: State cache dict for streaming inference.+ **kwargs: Additional keyword arguments.+ """+ if cache is None:+ cache = {}+ if len(cache) == 0:+ self.init_cache(cache, **kwargs)++ meta_data = {}+ chunk_size = kwargs.get("chunk_size", 60000) # 50ms+ chunk_stride_samples = int(chunk_size * frontend.fs / 1000)++</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/fsmn_vad_streaming/model.py#L856" target="_blank" class="src-more">View full source on GitHub →</a></details></div>+<div class="api-detail" id="e101"><span class="dtag badge-m">method</span><h2><code>FsmnVADStreaming.export(**kwargs)</code></h2><div class="dmod">funasr.models.fsmn_vad_streaming.FsmnVADStreaming · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/fsmn_vad_streaming/model.py#L970" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Export.</p>+<h4>Args:</h4>+<ul>+<li><code>**kwargs</code> — Additional keyword arguments.</li>+</ul>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def export(self, **kwargs):++ """Export.++ Args:+ **kwargs: Additional keyword arguments.+ """+ from .export_meta import export_rebuild_model++ models = export_rebuild_model(model=self, **kwargs)+ return models+</code></pre></details></div>+<div class="api-detail" id="e102"><span class="dtag badge-m">method</span><h2><code>FsmnVADStreaming.DetectCommonFrames(cache)</code></h2><div class="dmod">funasr.models.fsmn_vad_streaming.FsmnVADStreaming · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/fsmn_vad_streaming/model.py#L982" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Detectcommonframes.</p>+<h4>Args:</h4>+<ul>+<li><code>cache</code> — State cache dict for streaming inference.</li>+</ul>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def DetectCommonFrames(self, cache: dict = None) -> int:+ """Detectcommonframes.++ Args:+ cache: State cache dict for streaming inference.+ """+ if cache is None:+ cache = {}+ if cache["stats"].vad_state_machine == VadStateMachine.kVadInStateEndPointDetected:+ return 0+ for i in range(self.vad_opts.nn_eval_block_size - 1, -1, -1):+ frame_state = FrameState.kFrameStateInvalid+ frame_state = self.GetFrameState(+ cache["stats"].frm_cnt - 1 - i - cache["stats"].last_drop_frames, cache=cache+ )+ self.DetectOneFrame(frame_state, cache["stats"].frm_cnt - 1 - i, False, cache=cache)++ return 0+</code></pre></details></div>+<div class="api-detail" id="e103"><span class="dtag badge-m">method</span><h2><code>FsmnVADStreaming.DetectLastFrames(cache)</code></h2><div class="dmod">funasr.models.fsmn_vad_streaming.FsmnVADStreaming · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/fsmn_vad_streaming/model.py#L1001" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Detectlastframes.</p>+<h4>Args:</h4>+<ul>+<li><code>cache</code> — State cache dict for streaming inference.</li>+</ul>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def DetectLastFrames(self, cache: dict = None) -> int:+ """Detectlastframes.++ Args:+ cache: State cache dict for streaming inference.+ """+ if cache is None:+ cache = {}+ if cache["stats"].vad_state_machine == VadStateMachine.kVadInStateEndPointDetected:+ return 0+ for i in range(self.vad_opts.nn_eval_block_size - 1, -1, -1):+ frame_state = FrameState.kFrameStateInvalid+ frame_state = self.GetFrameState(+ cache["stats"].frm_cnt - 1 - i - cache["stats"].last_drop_frames, cache=cache+ )+ if i != 0:+ self.DetectOneFrame(frame_state, cache["stats"].frm_cnt - 1 - i, False, cache=cache)+ else:+ self.DetectOneFrame(frame_state, cache["stats"].frm_cnt - 1, True, cache=cache)++ return 0+</code></pre></details></div>+<div class="api-detail" id="e104"><span class="dtag badge-m">method</span><h2><code>FsmnVADStreaming.DetectOneFrame(cur_frm_state, cur_frm_idx, is_final_frame, cache)</code></h2><div class="dmod">funasr.models.fsmn_vad_streaming.FsmnVADStreaming · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/fsmn_vad_streaming/model.py#L1023" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Detectoneframe.</p>+<h4>Args:</h4>+<ul>+<li><code>cur_frm_state</code> — TODO.</li>+<li><code>cur_frm_idx</code> — TODO.</li>+<li><code>is_final_frame</code> — Boolean flag for final frame.</li>+<li><code>cache</code> — State cache dict for streaming inference.</li>+</ul>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def DetectOneFrame(+ self, cur_frm_state: FrameState, cur_frm_idx: int, is_final_frame: bool, cache: dict = None+ ) -> None:+ """Detectoneframe.++ Args:+ cur_frm_state: TODO.+ cur_frm_idx: TODO.+ is_final_frame: Boolean flag for final frame.+ cache: State cache dict for streaming inference.+ """+ if cache is None:+ cache = {}+ tmp_cur_frm_state = FrameState.kFrameStateInvalid+ if cur_frm_state == FrameState.kFrameStateSpeech:+ if math.fabs(1.0) > self.vad_opts.fe_prior_thres:+ tmp_cur_frm_state = FrameState.kFrameStateSpeech+ else:+ tmp_cur_frm_state = FrameState.kFrameStateSil+ elif cur_frm_state == FrameState.kFrameStateSil:+ tmp_cur_frm_state = FrameState.kFrameStateSil+ state_change = cache["windows_detector"].DetectOneFrame(+ tmp_cur_frm_state, cur_frm_idx, cache=cache+ )+ frm_shift_in_ms = self.vad_opts.frame_in_ms+ if AudioChangeState.kChangeStateSil2Speech == state_change:+ silence_frame_count = cache["stats"].continous_silence_frame_count+ cache["stats"].continous_silence_frame_count = 0+ cache["stats"].pre_end_silence_detected = False+ start_frame = 0+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/fsmn_vad_streaming/model.py#L1023" target="_blank" class="src-more">View full source on GitHub →</a></details></div>+<div class="api-detail" id="e105"><span class="dtag badge-c">class</span><h2>FunASRNano</h2><div class="dmod">funasr.models.fun_asr_nano · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/fun_asr_nano/model.py#L27" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><ul>+<li><code>Fun</code> — ASR-Nano: End-to-End ASR Large Model.</li>+</ul>+<p>Trained on tens of millions of hours of real speech data.</p>+<p>Supports 31 languages including Chinese dialects and regional accents.</p>+<h4>Features:</h4>+<ul>+<li>Character-level timestamps (via CTC forced alignment)</li>+<li>Hotword customization</li>+<li>Speaker diarization (when combined with spk_model)</li>+<li>Lyrics and rap recognition</li>+<li>Streaming chunk-by-chunk inference (demo2.py)</li>+<li><code>Output</code> — {"key": ..., "text": ..., "timestamps": [{"token", "start_time", "end_time"}, ...],</li>+<li class="sub">"ctc_timestamps": [...]}</li>+<li><code>Note</code> — Outputs punctuation natively — punc_model is NOT needed.</li>+<li><code>Requirements</code> — pip install tiktoken huggingface_hub</li>+</ul>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code>class FunASRNano(nn.Module):+ """Fun-ASR-Nano: End-to-End ASR Large Model.++ Trained on tens of millions of hours of real speech data.+ Supports 31 languages including Chinese dialects and regional accents.++ Features:+ - Character-level timestamps (via CTC forced alignment)+ - Hotword customization+ - Speaker diarization (when combined with spk_model)+ - Lyrics and rap recognition+ - Streaming chunk-by-chunk inference (demo2.py)++ Output: {"key": ..., "text": ..., "timestamps": [{"token", "start_time", "end_time"}, ...],+ "ctc_timestamps": [...]}++ Note: Outputs punctuation natively — punc_model is NOT needed.++ Requirements: pip install tiktoken huggingface_hub+ """++ def __init__(+ self,+ audio_encoder: str = None,+ audio_encoder_conf: dict = None,+ audio_adaptor: str = None,+ audio_adaptor_conf: dict = None,+ llm: str = None,+ llm_conf: dict = None,+ input_size: int = 80,+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/fun_asr_nano/model.py#L27" target="_blank" class="src-more">View full source on GitHub →</a></details><h3 class="mt">Methods</h3><div class="mblk"><code>.forward(speech, speech_lengths, input_ids, attention_mask, labels_ids, fbank_beg, fbank_mask, **kwargs)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/fun_asr_nano/model.py#L194" target="_blank" class="gh-link">L194</a><div class="mdoc"><p>Forward pass for training.</p>+<h4>Args:</h4>+<ul>+<li><code>speech</code> — Speech audio tensor, shape (batch, time).</li>+<li><code>speech_lengths</code> — Length of each speech sample.</li>+<li><code>input_ids</code> — TODO.</li>+<li><code>attention_mask</code> — TODO.</li>+<li><code>labels_ids</code> — TODO.</li>+<li><code>fbank_beg</code> — TODO.</li>+<li><code>fbank_mask</code> — TODO.</li>+<li><code>**kwargs</code> — Additional keyword arguments.</li>+</ul>+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def forward(+ self,+ speech: torch.Tensor = None,+ speech_lengths: torch.Tensor = None,+ input_ids: torch.Tensor = None,+ attention_mask: torch.Tensor = None,+ labels_ids: torch.Tensor = None,+ fbank_beg: torch.Tensor = None,+ fbank_mask: torch.Tensor = None,+ **kwargs,+ ):+ """Forward pass for training.++ Args:+ speech: Speech audio tensor, shape (batch, time).+ speech_lengths: Length of each speech sample.+ input_ids: TODO.+ attention_mask: TODO.+ labels_ids: TODO.+ fbank_beg: TODO.+ fbank_mask: TODO.+ **kwargs: Additional keyword arguments.+ """+ batch_size, token_num = input_ids.shape+ stats = {}+ input_ids[input_ids < 0] = 0+ inputs_embeds = self.llm.model.get_input_embeddings()(input_ids)+ if speech is not None:+ if len(speech_lengths.size()) > 1:+ speech_lengths = speech_lengths[:, 0]+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/fun_asr_nano/model.py#L194" target="_blank" class="src-more">View full source →</a></details></div><div class="mblk"><code>.forward_export(speech, speech_lengths, **kwargs)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/fun_asr_nano/model.py#L317" target="_blank" class="gh-link">L317</a><div class="mdoc"><p>Forward export.</p>+<h4>Args:</h4>+<ul>+<li><code>speech</code> — Speech audio tensor, shape (batch, time).</li>+<li><code>speech_lengths</code> — Length of each speech sample.</li>+<li><code>**kwargs</code> — Additional keyword arguments.</li>+</ul>+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def forward_export(self, speech, speech_lengths, **kwargs):+ """Forward export.++ Args:+ speech: Speech audio tensor, shape (batch, time).+ speech_lengths: Length of each speech sample.+ **kwargs: Additional keyword arguments.+ """+ x, olens = self.audio_encoder(speech, speech_lengths)+ encoder_out, encoder_out_lens = self.audio_adaptor(x, olens)+ return encoder_out, encoder_out_lens+</code></pre></details></div><div class="mblk"><code>.encode(speech, speech_lengths)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/fun_asr_nano/model.py#L329" target="_blank" class="gh-link">L329</a><div class="mdoc"><p>Encode.</p>+<h4>Args:</h4>+<ul>+<li><code>speech</code> — Speech audio tensor, shape (batch, time).</li>+<li><code>speech_lengths</code> — Length of each speech sample.</li>+</ul>+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def encode(self, speech, speech_lengths):+ # audio encoder+ """Encode.++ Args:+ speech: Speech audio tensor, shape (batch, time).+ speech_lengths: Length of each speech sample.+ """+ encoder_out, encoder_out_lens = self.audio_encoder(speech, speech_lengths)++ return encoder_out, encoder_out_lens+</code></pre></details></div><div class="mblk"><code>.data_template(data)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/fun_asr_nano/model.py#L341" target="_blank" class="gh-link">L341</a><div class="mdoc"><p>Data template.</p>+<h4>Args:</h4>+<ul>+<li><code>data</code> — TODO.</li>+</ul>+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def data_template(self, data):+ """Data template.++ Args:+ data: TODO.+ """+ system, user, assistant = [], [], []+ for i, item in enumerate(data):+ role = item["role"]+ content = item["content"]+ if role == "system":+ system.append(content)+ elif role == "user":+ if "audio" in item:+ audio = item["audio"]+ content = [content, audio]+ user.append(content)+ elif role == "assistant":+ assistant.append(content)++ system = system * len(user)++ contents = {+ "system": system,+ "user": user,+ "assistant": assistant,+ }++ return contents+</code></pre></details></div><div class="mblk"><code>.data_load_speech(contents, tokenizer, frontend, meta_data, **kwargs)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/fun_asr_nano/model.py#L371" target="_blank" class="gh-link">L371</a><div class="mdoc"><p>Data load speech.</p>+<h4>Args:</h4>+<ul>+<li><code>contents</code> — TODO.</li>+<li><code>tokenizer</code> — Tokenizer instance for text encoding/decoding.</li>+<li><code>frontend</code> — Audio frontend for feature extraction.</li>+<li><code>meta_data</code> — TODO.</li>+<li><code>**kwargs</code> — Additional keyword arguments.</li>+</ul>+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def data_load_speech(self, contents: dict, tokenizer, frontend, meta_data={}, **kwargs):+ """Data load speech.++ Args:+ contents: TODO.+ tokenizer: Tokenizer instance for text encoding/decoding.+ frontend: Audio frontend for feature extraction.+ meta_data: TODO.+ **kwargs: Additional keyword arguments.+ """+ system = contents["system"]+ user = contents["user"]+ assistant = contents["assistant"]+ pattern = re.compile(r"(<\|startofspeech\|>.*?<\|endofspeech\|>)")+ do_think = True+ sys_prompt = True+ if "dataset_conf" in kwargs:+ do_think = kwargs["dataset_conf"].get("do_think", True)+ sys_prompt = kwargs["dataset_conf"].get("sys_prompt", True)++ input_ids, labels, fbank, fbank_lens, fbank_mask, fbank_beg, fake_token_len = (+ [],+ [],+ [],+ [],+ [],+ [],+ [],+ )+ input_source_ids = []+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/fun_asr_nano/model.py#L371" target="_blank" class="src-more">View full source →</a></details></div><div class="mblk"><code>.inference_prepare(data_in, data_lengths, key, tokenizer, frontend, **kwargs)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/fun_asr_nano/model.py#L533" target="_blank" class="gh-link">L533</a><div class="mdoc"><p>Inference prepare.</p>+<h4>Args:</h4>+<ul>+<li><code>data_in</code> — Input data (audio samples, file paths, or text).</li>+<li><code>data_lengths</code> — Lengths of each input sample in the batch.</li>+<li><code>key</code> — Sample identifiers.</li>+<li><code>tokenizer</code> — Tokenizer instance for text encoding/decoding.</li>+<li><code>frontend</code> — Audio frontend for feature extraction.</li>+<li><code>**kwargs</code> — Additional keyword arguments.</li>+</ul>+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def inference_prepare(+ self,+ data_in,+ data_lengths=None,+ key: list = None,+ tokenizer=None,+ frontend=None,+ **kwargs,+ ):+ """Inference prepare.++ Args:+ data_in: Input data (audio samples, file paths, or text).+ data_lengths: Lengths of each input sample in the batch.+ key: Sample identifiers.+ tokenizer: Tokenizer instance for text encoding/decoding.+ frontend: Audio frontend for feature extraction.+ **kwargs: Additional keyword arguments.+ """+ meta_data = {}++ if len(data_in) > 1:+ raise NotImplementedError("batch decoding is not implemented")++ contents = self.data_template(data_in[0])+ output = self.data_load_speech(contents, tokenizer, frontend, meta_data=meta_data, **kwargs)+ batch = to_device(output, kwargs["device"])++ # audio encoder+ speech = batch["speech"]+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/fun_asr_nano/model.py#L533" target="_blank" class="src-more">View full source →</a></details></div><div class="mblk"><code>.get_prompt(hotwords, language, itn)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/fun_asr_nano/model.py#L632" target="_blank" class="gh-link">L632</a><div class="mdoc"><p>Get prompt.</p>+<h4>Args:</h4>+<ul>+<li><code>hotwords</code> — TODO.</li>+<li><code>language</code> — Language identifier.</li>+<li><code>itn</code> — TODO.</li>+</ul>+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def get_prompt(self, hotwords: list[str], language: str = None, itn: bool = True):+ """Get prompt.++ Args:+ hotwords: TODO.+ language: Language identifier.+ itn: TODO.+ """+ if len(hotwords) > 0:+ hotwords = ", ".join(hotwords)+ prompt = f"请结合上下文信息,更加准确地完成语音转写任务。如果没有相关信息,我们会留空。\n\n\n**上下文信息:**\n\n\n"+ prompt += f"热词列表:[{hotwords}]\n"+ else:+ prompt = ""+ if language is None:+ prompt += "语音转写"+ else:+ prompt += f"语音转写成{language}"+ if not itn:+ prompt += ",不进行文本规整"+ return prompt + ":"+</code></pre></details></div><div class="mblk"><code>.generate_chatml(prompt, data)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/fun_asr_nano/model.py#L654" target="_blank" class="gh-link">L654</a><div class="mdoc"><p>Generate chatml.</p>+<h4>Args:</h4>+<ul>+<li><code>prompt</code> — TODO.</li>+<li><code>data</code> — TODO.</li>+</ul>+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def generate_chatml(self, prompt: str, data: Union[str, torch.Tensor]):+ """Generate chatml.++ Args:+ prompt: TODO.+ data: TODO.+ """+ if isinstance(data, str):+ return [+ {"role": "system", "content": "You are a helpful assistant."},+ {"role": "user", "content": f"{prompt}<|startofspeech|>!{data}<|endofspeech|>"},+ {"role": "assistant", "content": "null"},+ ]+ elif isinstance(data, torch.Tensor):+ return [+ {"role": "system", "content": "You are a helpful assistant."},+ {+ "role": "user",+ "content": f"{prompt}<|startofspeech|>!!<|endofspeech|>",+ "audio": data,+ },+ {"role": "assistant", "content": "null"},+ ]+</code></pre></details></div><div class="mblk"><code>.inference(data_in, data_lengths, key, tokenizer, frontend, **kwargs)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/fun_asr_nano/model.py#L678" target="_blank" class="gh-link">L678</a><div class="mdoc"><p>Run inference on input data.</p>+<h4>Args:</h4>+<ul>+<li><code>data_in</code> — Input data (audio samples, file paths, or text).</li>+<li><code>data_lengths</code> — Lengths of each input sample in the batch.</li>+<li><code>key</code> — Sample identifiers.</li>+<li><code>tokenizer</code> — Tokenizer instance for text encoding/decoding.</li>+<li><code>frontend</code> — Audio frontend for feature extraction.</li>+<li><code>**kwargs</code> — Additional keyword arguments.</li>+</ul>+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def inference(+ self,+ data_in,+ data_lengths=None,+ key: list = None,+ tokenizer=None,+ frontend=None,+ **kwargs,+ ):+ """Run inference on input data.++ Args:+ data_in: Input data (audio samples, file paths, or text).+ data_lengths: Lengths of each input sample in the batch.+ key: Sample identifiers.+ tokenizer: Tokenizer instance for text encoding/decoding.+ frontend: Audio frontend for feature extraction.+ **kwargs: Additional keyword arguments.+ """+ prompt = self.get_prompt(+ kwargs.get("hotwords", []), kwargs.get("language", None), kwargs.get("itn", True)+ )+ data_in = [self.generate_chatml(prompt, data) for data in data_in]++ if key is None:+ key = []+ for _ in data_in:+ chars = string.ascii_letters + string.digits+ key.append("rand_key_" + "".join(random.choice(chars) for _ in range(13)))++</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/fun_asr_nano/model.py#L678" target="_blank" class="src-more">View full source →</a></details></div><div class="mblk"><code>.inference_llm(data_in, data_lengths, key, tokenizer, frontend, **kwargs)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/fun_asr_nano/model.py#L717" target="_blank" class="gh-link">L717</a><div class="mdoc"><p>Inference llm.</p>+<h4>Args:</h4>+<ul>+<li><code>data_in</code> — Input data (audio samples, file paths, or text).</li>+<li><code>data_lengths</code> — Lengths of each input sample in the batch.</li>+<li><code>key</code> — Sample identifiers.</li>+<li><code>tokenizer</code> — Tokenizer instance for text encoding/decoding.</li>+<li><code>frontend</code> — Audio frontend for feature extraction.</li>+<li><code>**kwargs</code> — Additional keyword arguments.</li>+</ul>+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def inference_llm(+ self,+ data_in,+ data_lengths=None,+ key: list = None,+ tokenizer=None,+ frontend=None,+ **kwargs,+ ):+ """Inference llm.++ Args:+ data_in: Input data (audio samples, file paths, or text).+ data_lengths: Lengths of each input sample in the batch.+ key: Sample identifiers.+ tokenizer: Tokenizer instance for text encoding/decoding.+ frontend: Audio frontend for feature extraction.+ **kwargs: Additional keyword arguments.+ """+ inputs_embeds, contents, batch, source_ids, meta_data = self.inference_prepare(+ data_in, data_lengths, key, tokenizer, frontend, **kwargs+ )++ ctc_results = []+ if self.ctc_decoder is not None:+ encoder_out = meta_data["encoder_out"]+ encoder_out_lens = meta_data["encoder_out_lens"]+ decoder_out, decoder_out_lens = self.ctc_decoder(encoder_out, encoder_out_lens)+ ctc_logits = self.ctc.log_softmax(decoder_out)++</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/fun_asr_nano/model.py#L717" target="_blank" class="src-more">View full source →</a></details></div><div class="mblk"><code>.from_pretrained(model, **kwargs)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/fun_asr_nano/model.py#L856" target="_blank" class="gh-link">L856</a><div class="mdoc"><p>From pretrained.</p>+<h4>Args:</h4>+<ul>+<li><code>model</code> — Model instance or model name.</li>+<li><code>**kwargs</code> — Additional keyword arguments.</li>+</ul>+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def from_pretrained(model: str = None, **kwargs):+ """From pretrained.++ Args:+ model: Model instance or model name.+ **kwargs: Additional keyword arguments.+ """+ from funasr import AutoModel++ model, kwargs = AutoModel.build_model(model=model, trust_remote_code=True, **kwargs)++ return model, kwargs+</code></pre></details></div></div>+<div class="api-detail" id="e106"><span class="dtag badge-m">method</span><h2><code>FunASRNano.forward(speech, speech_lengths, input_ids, attention_mask, labels_ids, fbank_beg, fbank_mask, **kwargs)</code></h2><div class="dmod">funasr.models.fun_asr_nano.FunASRNano · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/fun_asr_nano/model.py#L194" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Forward pass for training.</p>+<h4>Args:</h4>+<ul>+<li><code>speech</code> — Speech audio tensor, shape (batch, time).</li>+<li><code>speech_lengths</code> — Length of each speech sample.</li>+<li><code>input_ids</code> — TODO.</li>+<li><code>attention_mask</code> — TODO.</li>+<li><code>labels_ids</code> — TODO.</li>+<li><code>fbank_beg</code> — TODO.</li>+<li><code>fbank_mask</code> — TODO.</li>+<li><code>**kwargs</code> — Additional keyword arguments.</li>+</ul>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def forward(+ self,+ speech: torch.Tensor = None,+ speech_lengths: torch.Tensor = None,+ input_ids: torch.Tensor = None,+ attention_mask: torch.Tensor = None,+ labels_ids: torch.Tensor = None,+ fbank_beg: torch.Tensor = None,+ fbank_mask: torch.Tensor = None,+ **kwargs,+ ):+ """Forward pass for training.++ Args:+ speech: Speech audio tensor, shape (batch, time).+ speech_lengths: Length of each speech sample.+ input_ids: TODO.+ attention_mask: TODO.+ labels_ids: TODO.+ fbank_beg: TODO.+ fbank_mask: TODO.+ **kwargs: Additional keyword arguments.+ """+ batch_size, token_num = input_ids.shape+ stats = {}+ input_ids[input_ids < 0] = 0+ inputs_embeds = self.llm.model.get_input_embeddings()(input_ids)+ if speech is not None:+ if len(speech_lengths.size()) > 1:+ speech_lengths = speech_lengths[:, 0]+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/fun_asr_nano/model.py#L194" target="_blank" class="src-more">View full source on GitHub →</a></details></div>+<div class="api-detail" id="e107"><span class="dtag badge-m">method</span><h2><code>FunASRNano.forward_export(speech, speech_lengths, **kwargs)</code></h2><div class="dmod">funasr.models.fun_asr_nano.FunASRNano · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/fun_asr_nano/model.py#L317" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Forward export.</p>+<h4>Args:</h4>+<ul>+<li><code>speech</code> — Speech audio tensor, shape (batch, time).</li>+<li><code>speech_lengths</code> — Length of each speech sample.</li>+<li><code>**kwargs</code> — Additional keyword arguments.</li>+</ul>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def forward_export(self, speech, speech_lengths, **kwargs):+ """Forward export.++ Args:+ speech: Speech audio tensor, shape (batch, time).+ speech_lengths: Length of each speech sample.+ **kwargs: Additional keyword arguments.+ """+ x, olens = self.audio_encoder(speech, speech_lengths)+ encoder_out, encoder_out_lens = self.audio_adaptor(x, olens)+ return encoder_out, encoder_out_lens+</code></pre></details></div>+<div class="api-detail" id="e108"><span class="dtag badge-m">method</span><h2><code>FunASRNano.encode(speech, speech_lengths)</code></h2><div class="dmod">funasr.models.fun_asr_nano.FunASRNano · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/fun_asr_nano/model.py#L329" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Encode.</p>+<h4>Args:</h4>+<ul>+<li><code>speech</code> — Speech audio tensor, shape (batch, time).</li>+<li><code>speech_lengths</code> — Length of each speech sample.</li>+</ul>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def encode(self, speech, speech_lengths):+ # audio encoder+ """Encode.++ Args:+ speech: Speech audio tensor, shape (batch, time).+ speech_lengths: Length of each speech sample.+ """+ encoder_out, encoder_out_lens = self.audio_encoder(speech, speech_lengths)++ return encoder_out, encoder_out_lens+</code></pre></details></div>+<div class="api-detail" id="e109"><span class="dtag badge-m">method</span><h2><code>FunASRNano.data_template(data)</code></h2><div class="dmod">funasr.models.fun_asr_nano.FunASRNano · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/fun_asr_nano/model.py#L341" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Data template.</p>+<h4>Args:</h4>+<ul>+<li><code>data</code> — TODO.</li>+</ul>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def data_template(self, data):+ """Data template.++ Args:+ data: TODO.+ """+ system, user, assistant = [], [], []+ for i, item in enumerate(data):+ role = item["role"]+ content = item["content"]+ if role == "system":+ system.append(content)+ elif role == "user":+ if "audio" in item:+ audio = item["audio"]+ content = [content, audio]+ user.append(content)+ elif role == "assistant":+ assistant.append(content)++ system = system * len(user)++ contents = {+ "system": system,+ "user": user,+ "assistant": assistant,+ }++ return contents+</code></pre></details></div>+<div class="api-detail" id="e110"><span class="dtag badge-m">method</span><h2><code>FunASRNano.data_load_speech(contents, tokenizer, frontend, meta_data, **kwargs)</code></h2><div class="dmod">funasr.models.fun_asr_nano.FunASRNano · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/fun_asr_nano/model.py#L371" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Data load speech.</p>+<h4>Args:</h4>+<ul>+<li><code>contents</code> — TODO.</li>+<li><code>tokenizer</code> — Tokenizer instance for text encoding/decoding.</li>+<li><code>frontend</code> — Audio frontend for feature extraction.</li>+<li><code>meta_data</code> — TODO.</li>+<li><code>**kwargs</code> — Additional keyword arguments.</li>+</ul>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def data_load_speech(self, contents: dict, tokenizer, frontend, meta_data={}, **kwargs):+ """Data load speech.++ Args:+ contents: TODO.+ tokenizer: Tokenizer instance for text encoding/decoding.+ frontend: Audio frontend for feature extraction.+ meta_data: TODO.+ **kwargs: Additional keyword arguments.+ """+ system = contents["system"]+ user = contents["user"]+ assistant = contents["assistant"]+ pattern = re.compile(r"(<\|startofspeech\|>.*?<\|endofspeech\|>)")+ do_think = True+ sys_prompt = True+ if "dataset_conf" in kwargs:+ do_think = kwargs["dataset_conf"].get("do_think", True)+ sys_prompt = kwargs["dataset_conf"].get("sys_prompt", True)++ input_ids, labels, fbank, fbank_lens, fbank_mask, fbank_beg, fake_token_len = (+ [],+ [],+ [],+ [],+ [],+ [],+ [],+ )+ input_source_ids = []+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/fun_asr_nano/model.py#L371" target="_blank" class="src-more">View full source on GitHub →</a></details></div>+<div class="api-detail" id="e111"><span class="dtag badge-m">method</span><h2><code>FunASRNano.inference_prepare(data_in, data_lengths, key, tokenizer, frontend, **kwargs)</code></h2><div class="dmod">funasr.models.fun_asr_nano.FunASRNano · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/fun_asr_nano/model.py#L533" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Inference prepare.</p>+<h4>Args:</h4>+<ul>+<li><code>data_in</code> — Input data (audio samples, file paths, or text).</li>+<li><code>data_lengths</code> — Lengths of each input sample in the batch.</li>+<li><code>key</code> — Sample identifiers.</li>+<li><code>tokenizer</code> — Tokenizer instance for text encoding/decoding.</li>+<li><code>frontend</code> — Audio frontend for feature extraction.</li>+<li><code>**kwargs</code> — Additional keyword arguments.</li>+</ul>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def inference_prepare(+ self,+ data_in,+ data_lengths=None,+ key: list = None,+ tokenizer=None,+ frontend=None,+ **kwargs,+ ):+ """Inference prepare.++ Args:+ data_in: Input data (audio samples, file paths, or text).+ data_lengths: Lengths of each input sample in the batch.+ key: Sample identifiers.+ tokenizer: Tokenizer instance for text encoding/decoding.+ frontend: Audio frontend for feature extraction.+ **kwargs: Additional keyword arguments.+ """+ meta_data = {}++ if len(data_in) > 1:+ raise NotImplementedError("batch decoding is not implemented")++ contents = self.data_template(data_in[0])+ output = self.data_load_speech(contents, tokenizer, frontend, meta_data=meta_data, **kwargs)+ batch = to_device(output, kwargs["device"])++ # audio encoder+ speech = batch["speech"]+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/fun_asr_nano/model.py#L533" target="_blank" class="src-more">View full source on GitHub →</a></details></div>+<div class="api-detail" id="e112"><span class="dtag badge-m">method</span><h2><code>FunASRNano.get_prompt(hotwords, language, itn)</code></h2><div class="dmod">funasr.models.fun_asr_nano.FunASRNano · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/fun_asr_nano/model.py#L632" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Get prompt.</p>+<h4>Args:</h4>+<ul>+<li><code>hotwords</code> — TODO.</li>+<li><code>language</code> — Language identifier.</li>+<li><code>itn</code> — TODO.</li>+</ul>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def get_prompt(self, hotwords: list[str], language: str = None, itn: bool = True):+ """Get prompt.++ Args:+ hotwords: TODO.+ language: Language identifier.+ itn: TODO.+ """+ if len(hotwords) > 0:+ hotwords = ", ".join(hotwords)+ prompt = f"请结合上下文信息,更加准确地完成语音转写任务。如果没有相关信息,我们会留空。\n\n\n**上下文信息:**\n\n\n"+ prompt += f"热词列表:[{hotwords}]\n"+ else:+ prompt = ""+ if language is None:+ prompt += "语音转写"+ else:+ prompt += f"语音转写成{language}"+ if not itn:+ prompt += ",不进行文本规整"+ return prompt + ":"+</code></pre></details></div>+<div class="api-detail" id="e113"><span class="dtag badge-m">method</span><h2><code>FunASRNano.generate_chatml(prompt, data)</code></h2><div class="dmod">funasr.models.fun_asr_nano.FunASRNano · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/fun_asr_nano/model.py#L654" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Generate chatml.</p>+<h4>Args:</h4>+<ul>+<li><code>prompt</code> — TODO.</li>+<li><code>data</code> — TODO.</li>+</ul>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def generate_chatml(self, prompt: str, data: Union[str, torch.Tensor]):+ """Generate chatml.++ Args:+ prompt: TODO.+ data: TODO.+ """+ if isinstance(data, str):+ return [+ {"role": "system", "content": "You are a helpful assistant."},+ {"role": "user", "content": f"{prompt}<|startofspeech|>!{data}<|endofspeech|>"},+ {"role": "assistant", "content": "null"},+ ]+ elif isinstance(data, torch.Tensor):+ return [+ {"role": "system", "content": "You are a helpful assistant."},+ {+ "role": "user",+ "content": f"{prompt}<|startofspeech|>!!<|endofspeech|>",+ "audio": data,+ },+ {"role": "assistant", "content": "null"},+ ]+</code></pre></details></div>+<div class="api-detail" id="e114"><span class="dtag badge-m">method</span><h2><code>FunASRNano.inference(data_in, data_lengths, key, tokenizer, frontend, **kwargs)</code></h2><div class="dmod">funasr.models.fun_asr_nano.FunASRNano · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/fun_asr_nano/model.py#L678" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Run inference on input data.</p>+<h4>Args:</h4>+<ul>+<li><code>data_in</code> — Input data (audio samples, file paths, or text).</li>+<li><code>data_lengths</code> — Lengths of each input sample in the batch.</li>+<li><code>key</code> — Sample identifiers.</li>+<li><code>tokenizer</code> — Tokenizer instance for text encoding/decoding.</li>+<li><code>frontend</code> — Audio frontend for feature extraction.</li>+<li><code>**kwargs</code> — Additional keyword arguments.</li>+</ul>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def inference(+ self,+ data_in,+ data_lengths=None,+ key: list = None,+ tokenizer=None,+ frontend=None,+ **kwargs,+ ):+ """Run inference on input data.++ Args:+ data_in: Input data (audio samples, file paths, or text).+ data_lengths: Lengths of each input sample in the batch.+ key: Sample identifiers.+ tokenizer: Tokenizer instance for text encoding/decoding.+ frontend: Audio frontend for feature extraction.+ **kwargs: Additional keyword arguments.+ """+ prompt = self.get_prompt(+ kwargs.get("hotwords", []), kwargs.get("language", None), kwargs.get("itn", True)+ )+ data_in = [self.generate_chatml(prompt, data) for data in data_in]++ if key is None:+ key = []+ for _ in data_in:+ chars = string.ascii_letters + string.digits+ key.append("rand_key_" + "".join(random.choice(chars) for _ in range(13)))++</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/fun_asr_nano/model.py#L678" target="_blank" class="src-more">View full source on GitHub →</a></details></div>+<div class="api-detail" id="e115"><span class="dtag badge-m">method</span><h2><code>FunASRNano.inference_llm(data_in, data_lengths, key, tokenizer, frontend, **kwargs)</code></h2><div class="dmod">funasr.models.fun_asr_nano.FunASRNano · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/fun_asr_nano/model.py#L717" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Inference llm.</p>+<h4>Args:</h4>+<ul>+<li><code>data_in</code> — Input data (audio samples, file paths, or text).</li>+<li><code>data_lengths</code> — Lengths of each input sample in the batch.</li>+<li><code>key</code> — Sample identifiers.</li>+<li><code>tokenizer</code> — Tokenizer instance for text encoding/decoding.</li>+<li><code>frontend</code> — Audio frontend for feature extraction.</li>+<li><code>**kwargs</code> — Additional keyword arguments.</li>+</ul>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def inference_llm(+ self,+ data_in,+ data_lengths=None,+ key: list = None,+ tokenizer=None,+ frontend=None,+ **kwargs,+ ):+ """Inference llm.++ Args:+ data_in: Input data (audio samples, file paths, or text).+ data_lengths: Lengths of each input sample in the batch.+ key: Sample identifiers.+ tokenizer: Tokenizer instance for text encoding/decoding.+ frontend: Audio frontend for feature extraction.+ **kwargs: Additional keyword arguments.+ """+ inputs_embeds, contents, batch, source_ids, meta_data = self.inference_prepare(+ data_in, data_lengths, key, tokenizer, frontend, **kwargs+ )++ ctc_results = []+ if self.ctc_decoder is not None:+ encoder_out = meta_data["encoder_out"]+ encoder_out_lens = meta_data["encoder_out_lens"]+ decoder_out, decoder_out_lens = self.ctc_decoder(encoder_out, encoder_out_lens)+ ctc_logits = self.ctc.log_softmax(decoder_out)++</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/fun_asr_nano/model.py#L717" target="_blank" class="src-more">View full source on GitHub →</a></details></div>+<div class="api-detail" id="e116"><span class="dtag badge-m">method</span><h2><code>FunASRNano.from_pretrained(model, **kwargs)</code></h2><div class="dmod">funasr.models.fun_asr_nano.FunASRNano · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/fun_asr_nano/model.py#L856" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>From pretrained.</p>+<h4>Args:</h4>+<ul>+<li><code>model</code> — Model instance or model name.</li>+<li><code>**kwargs</code> — Additional keyword arguments.</li>+</ul>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def from_pretrained(model: str = None, **kwargs):+ """From pretrained.++ Args:+ model: Model instance or model name.+ **kwargs: Additional keyword arguments.+ """+ from funasr import AutoModel++ model, kwargs = AutoModel.build_model(model=model, trust_remote_code=True, **kwargs)++ return model, kwargs+</code></pre></details></div>+<div class="api-detail" id="e117"><span class="dtag badge-c">class</span><h2>GLMASR</h2><div class="dmod">funasr.models.glm_asr · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/glm_asr/model.py#L14" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p class="muted">No documentation yet.</p></div><details class="src-block"><summary>📄 Source code</summary><pre><code>class GLMASR(nn.Module):++ def __init__(self, **kwargs):+ """Initialize GLMASR.++ Args:+ **kwargs: Additional keyword arguments.+ """+ super().__init__()+ model_path = kwargs.get("model_path", kwargs.get("model", "zai-org/GLM-ASR-Nano-2512"))+ device = kwargs.get("device", "cuda:0")+ dtype = kwargs.get("dtype", "bf16")+ hub = kwargs.get("hub", "ms")+ self._max_new_tokens = kwargs.get("max_new_tokens", 512)++ self._dtype_map = {"bf16": torch.bfloat16, "fp16": torch.float16, "fp32": torch.float32}+ self._device = device+ self._torch_dtype = self._dtype_map.get(dtype, torch.bfloat16)+ self._placeholder = nn.Parameter(torch.empty(0))++ model_path = self._resolve_model_path(model_path, hub, kwargs)+ self.model_path = model_path++ from transformers import AutoModel as HFAutoModel+ from transformers import AutoProcessor++ self.processor = AutoProcessor.from_pretrained(model_path, trust_remote_code=True)+ self.glm_model = HFAutoModel.from_pretrained(+ model_path,+ dtype=self._torch_dtype,+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/glm_asr/model.py#L14" target="_blank" class="src-more">View full source on GitHub →</a></details><h3 class="mt">Methods</h3><div class="mblk"><code>.forward(**kwargs)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/glm_asr/model.py#L73" target="_blank" class="gh-link">L73</a><div class="mdoc"><p>Forward pass for training.</p>+<h4>Args:</h4>+<ul>+<li><code>**kwargs</code> — Additional keyword arguments.</li>+</ul>+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def forward(self, **kwargs):+ """Forward pass for training.++ Args:+ **kwargs: Additional keyword arguments.+ """+ raise NotImplementedError("GLMASR only supports inference mode")+</code></pre></details></div><div class="mblk"><code>.inference(data_in, data_lengths, key, tokenizer, frontend, **kwargs)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/glm_asr/model.py#L81" target="_blank" class="gh-link">L81</a><div class="mdoc"><p>Run inference on input data.</p>+<h4>Args:</h4>+<ul>+<li><code>data_in</code> — Input data (audio samples, file paths, or text).</li>+<li><code>data_lengths</code> — Lengths of each input sample in the batch.</li>+<li><code>key</code> — Sample identifiers.</li>+<li><code>tokenizer</code> — Tokenizer instance for text encoding/decoding.</li>+<li><code>frontend</code> — Audio frontend for feature extraction.</li>+<li><code>**kwargs</code> — Additional keyword arguments.</li>+</ul>+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def inference(+ self,+ data_in,+ data_lengths=None,+ key: list = None,+ tokenizer=None,+ frontend=None,+ **kwargs,+ ):+ """Run inference on input data.++ Args:+ data_in: Input data (audio samples, file paths, or text).+ data_lengths: Lengths of each input sample in the batch.+ key: Sample identifiers.+ tokenizer: Tokenizer instance for text encoding/decoding.+ frontend: Audio frontend for feature extraction.+ **kwargs: Additional keyword arguments.+ """+ meta_data = {}+ time1 = time.perf_counter()++ prompt = kwargs.get("prompt", "Please transcribe this audio into text")++ if isinstance(data_in, (list, tuple)):+ audio_list = list(data_in)+ elif isinstance(data_in, str):+ audio_list = [data_in]+ else:+ audio_list = [data_in]+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/glm_asr/model.py#L81" target="_blank" class="src-more">View full source →</a></details></div></div>+<div class="api-detail" id="e118"><span class="dtag badge-m">method</span><h2><code>GLMASR.forward(**kwargs)</code></h2><div class="dmod">funasr.models.glm_asr.GLMASR · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/glm_asr/model.py#L73" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Forward pass for training.</p>+<h4>Args:</h4>+<ul>+<li><code>**kwargs</code> — Additional keyword arguments.</li>+</ul>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def forward(self, **kwargs):+ """Forward pass for training.++ Args:+ **kwargs: Additional keyword arguments.+ """+ raise NotImplementedError("GLMASR only supports inference mode")+</code></pre></details></div>+<div class="api-detail" id="e119"><span class="dtag badge-m">method</span><h2><code>GLMASR.inference(data_in, data_lengths, key, tokenizer, frontend, **kwargs)</code></h2><div class="dmod">funasr.models.glm_asr.GLMASR · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/glm_asr/model.py#L81" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Run inference on input data.</p>+<h4>Args:</h4>+<ul>+<li><code>data_in</code> — Input data (audio samples, file paths, or text).</li>+<li><code>data_lengths</code> — Lengths of each input sample in the batch.</li>+<li><code>key</code> — Sample identifiers.</li>+<li><code>tokenizer</code> — Tokenizer instance for text encoding/decoding.</li>+<li><code>frontend</code> — Audio frontend for feature extraction.</li>+<li><code>**kwargs</code> — Additional keyword arguments.</li>+</ul>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def inference(+ self,+ data_in,+ data_lengths=None,+ key: list = None,+ tokenizer=None,+ frontend=None,+ **kwargs,+ ):+ """Run inference on input data.++ Args:+ data_in: Input data (audio samples, file paths, or text).+ data_lengths: Lengths of each input sample in the batch.+ key: Sample identifiers.+ tokenizer: Tokenizer instance for text encoding/decoding.+ frontend: Audio frontend for feature extraction.+ **kwargs: Additional keyword arguments.+ """+ meta_data = {}+ time1 = time.perf_counter()++ prompt = kwargs.get("prompt", "Please transcribe this audio into text")++ if isinstance(data_in, (list, tuple)):+ audio_list = list(data_in)+ elif isinstance(data_in, str):+ audio_list = [data_in]+ else:+ audio_list = [data_in]+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/glm_asr/model.py#L81" target="_blank" class="src-more">View full source on GitHub →</a></details></div>+<div class="api-detail" id="e120"><span class="dtag badge-c">class</span><h2>LCBNet</h2><div class="dmod">funasr.models.lcbnet · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/lcbnet/model.py#L28" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><ul>+<li><code>LCBNet</code> — Lightweight Convolutional Block Network for ASR.</li>+</ul>+<p>Efficient model design using depthwise separable convolutions</p>+<p>for low-resource deployment scenarios.</p>+<p>Inherits Paraformer pipeline.</p>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code>class LCBNet(nn.Module):+ """LCBNet: Lightweight Convolutional Block Network for ASR.++ Efficient model design using depthwise separable convolutions+ for low-resource deployment scenarios.++ Inherits Paraformer pipeline.+ """++ def __init__(+ self,+ specaug: str = None,+ specaug_conf: dict = None,+ normalize: str = None,+ normalize_conf: dict = None,+ encoder: str = None,+ encoder_conf: dict = None,+ decoder: str = None,+ decoder_conf: dict = None,+ text_encoder: str = None,+ text_encoder_conf: dict = None,+ bias_predictor: str = None,+ bias_predictor_conf: dict = None,+ fusion_encoder: str = None,+ fusion_encoder_conf: dict = None,+ ctc: str = None,+ ctc_conf: dict = None,+ ctc_weight: float = 0.5,+ interctc_weight: float = 0.0,+ select_num: int = 2,+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/lcbnet/model.py#L28" target="_blank" class="src-more">View full source on GitHub →</a></details><h3 class="mt">Methods</h3><div class="mblk"><code>.forward(speech, speech_lengths, text, text_lengths, **kwargs)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/lcbnet/model.py#L208" target="_blank" class="gh-link">L208</a><div class="mdoc"><p>Encoder + Decoder + Calc loss</p>+<h4>Args:</h4>+<ul>+<li><code>speech</code> — (Batch, Length, ...)</li>+<li><code>speech_lengths</code> — (Batch, )</li>+<li><code>text</code> — (Batch, Length)</li>+<li><code>text_lengths</code> — (Batch,)</li>+</ul>+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def forward(+ self,+ speech: torch.Tensor,+ speech_lengths: torch.Tensor,+ text: torch.Tensor,+ text_lengths: torch.Tensor,+ **kwargs,+ ) -> Tuple[torch.Tensor, Dict[str, torch.Tensor], torch.Tensor]:+ """Encoder + Decoder + Calc loss+ Args:+ speech: (Batch, Length, ...)+ speech_lengths: (Batch, )+ text: (Batch, Length)+ text_lengths: (Batch,)+ """++ if len(text_lengths.size()) > 1:+ text_lengths = text_lengths[:, 0]+ if len(speech_lengths.size()) > 1:+ speech_lengths = speech_lengths[:, 0]++ batch_size = speech.shape[0]++ # 1. Encoder+ encoder_out, encoder_out_lens = self.encode(speech, speech_lengths)+ intermediate_outs = None+ if isinstance(encoder_out, tuple):+ intermediate_outs = encoder_out[1]+ encoder_out = encoder_out[0]++</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/lcbnet/model.py#L208" target="_blank" class="src-more">View full source →</a></details></div><div class="mblk"><code>.encode(speech, speech_lengths, **kwargs)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/lcbnet/model.py#L302" target="_blank" class="gh-link">L302</a><div class="mdoc"><p>Frontend + Encoder. Note that this method is used by asr_inference.py</p>+<h4>Args:</h4>+<ul>+<li><code>speech</code> — (Batch, Length, ...)</li>+<li><code>speech_lengths</code> — (Batch, )</li>+<li><code>ind</code> — int</li>+</ul>+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def encode(+ self,+ speech: torch.Tensor,+ speech_lengths: torch.Tensor,+ **kwargs,+ ) -> Tuple[torch.Tensor, torch.Tensor]:+ """Frontend + Encoder. Note that this method is used by asr_inference.py+ Args:+ speech: (Batch, Length, ...)+ speech_lengths: (Batch, )+ ind: int+ """+ with autocast(False):+ # Data augmentation+ if self.specaug is not None and self.training:+ speech, speech_lengths = self.specaug(speech, speech_lengths)+ # Normalization for feature: e.g. Global-CMVN, Utterance-CMVN+ if self.normalize is not None:+ speech, speech_lengths = self.normalize(speech, speech_lengths)+ # Forward encoder+ # feats: (Batch, Length, Dim)+ # -> encoder_out: (Batch, Length2, Dim2)+ if self.encoder.interctc_use_conditioning:+ encoder_out, encoder_out_lens, _ = self.encoder(speech, speech_lengths, ctc=self.ctc)+ else:+ encoder_out, encoder_out_lens, _ = self.encoder(speech, speech_lengths)+ intermediate_outs = None+ if isinstance(encoder_out, tuple):+ intermediate_outs = encoder_out[1]+ encoder_out = encoder_out[0]+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/lcbnet/model.py#L302" target="_blank" class="src-more">View full source →</a></details></div><div class="mblk"><code>.init_beam_search(**kwargs)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/lcbnet/model.py#L400" target="_blank" class="gh-link">L400</a><div class="mdoc"><p>Init beam search.</p>+<h4>Args:</h4>+<ul>+<li><code>**kwargs</code> — Additional keyword arguments.</li>+</ul>+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def init_beam_search(+ self,+ **kwargs,+ ):+ """Init beam search.++ Args:+ **kwargs: Additional keyword arguments.+ """+ from funasr.models.transformer.search import BeamSearch+ from funasr.models.transformer.scorers.ctc import CTCPrefixScorer+ from funasr.models.transformer.scorers.length_bonus import LengthBonus++ # 1. Build ASR model+ scorers = {}++ if self.ctc != None:+ ctc = CTCPrefixScorer(ctc=self.ctc, eos=self.eos)+ scorers.update(ctc=ctc)+ token_list = kwargs.get("token_list")+ scorers.update(+ decoder=self.decoder,+ length_bonus=LengthBonus(len(token_list)),+ )++ # 3. Build ngram model+ # ngram is not supported now+ ngram = None+ scorers["ngram"] = ngram++</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/lcbnet/model.py#L400" target="_blank" class="src-more">View full source →</a></details></div><div class="mblk"><code>.inference(data_in, data_lengths, key, tokenizer, frontend, **kwargs)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/lcbnet/model.py#L450" target="_blank" class="gh-link">L450</a><div class="mdoc"><p>Run inference on input data.</p>+<h4>Args:</h4>+<ul>+<li><code>data_in</code> — Input data (audio samples, file paths, or text).</li>+<li><code>data_lengths</code> — Lengths of each input sample in the batch.</li>+<li><code>key</code> — Sample identifiers.</li>+<li><code>tokenizer</code> — Tokenizer instance for text encoding/decoding.</li>+<li><code>frontend</code> — Audio frontend for feature extraction.</li>+<li><code>**kwargs</code> — Additional keyword arguments.</li>+</ul>+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def inference(+ self,+ data_in,+ data_lengths=None,+ key: list = None,+ tokenizer=None,+ frontend=None,+ **kwargs,+ ):++ """Run inference on input data.++ Args:+ data_in: Input data (audio samples, file paths, or text).+ data_lengths: Lengths of each input sample in the batch.+ key: Sample identifiers.+ tokenizer: Tokenizer instance for text encoding/decoding.+ frontend: Audio frontend for feature extraction.+ **kwargs: Additional keyword arguments.+ """+ if kwargs.get("batch_size", 1) > 1:+ raise NotImplementedError("batch decoding is not implemented")++ # init beamsearch+ if self.beam_search is None:+ logging.info("enable beam_search")+ self.init_beam_search(**kwargs)+ self.nbest = kwargs.get("nbest", 1)++ meta_data = {}+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/lcbnet/model.py#L450" target="_blank" class="src-more">View full source →</a></details></div></div>+<div class="api-detail" id="e121"><span class="dtag badge-m">method</span><h2><code>LCBNet.forward(speech, speech_lengths, text, text_lengths, **kwargs)</code></h2><div class="dmod">funasr.models.lcbnet.LCBNet · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/lcbnet/model.py#L208" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Encoder + Decoder + Calc loss</p>+<h4>Args:</h4>+<ul>+<li><code>speech</code> — (Batch, Length, ...)</li>+<li><code>speech_lengths</code> — (Batch, )</li>+<li><code>text</code> — (Batch, Length)</li>+<li><code>text_lengths</code> — (Batch,)</li>+</ul>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def forward(+ self,+ speech: torch.Tensor,+ speech_lengths: torch.Tensor,+ text: torch.Tensor,+ text_lengths: torch.Tensor,+ **kwargs,+ ) -> Tuple[torch.Tensor, Dict[str, torch.Tensor], torch.Tensor]:+ """Encoder + Decoder + Calc loss+ Args:+ speech: (Batch, Length, ...)+ speech_lengths: (Batch, )+ text: (Batch, Length)+ text_lengths: (Batch,)+ """++ if len(text_lengths.size()) > 1:+ text_lengths = text_lengths[:, 0]+ if len(speech_lengths.size()) > 1:+ speech_lengths = speech_lengths[:, 0]++ batch_size = speech.shape[0]++ # 1. Encoder+ encoder_out, encoder_out_lens = self.encode(speech, speech_lengths)+ intermediate_outs = None+ if isinstance(encoder_out, tuple):+ intermediate_outs = encoder_out[1]+ encoder_out = encoder_out[0]++</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/lcbnet/model.py#L208" target="_blank" class="src-more">View full source on GitHub →</a></details></div>+<div class="api-detail" id="e122"><span class="dtag badge-m">method</span><h2><code>LCBNet.encode(speech, speech_lengths, **kwargs)</code></h2><div class="dmod">funasr.models.lcbnet.LCBNet · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/lcbnet/model.py#L302" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Frontend + Encoder. Note that this method is used by asr_inference.py</p>+<h4>Args:</h4>+<ul>+<li><code>speech</code> — (Batch, Length, ...)</li>+<li><code>speech_lengths</code> — (Batch, )</li>+<li><code>ind</code> — int</li>+</ul>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def encode(+ self,+ speech: torch.Tensor,+ speech_lengths: torch.Tensor,+ **kwargs,+ ) -> Tuple[torch.Tensor, torch.Tensor]:+ """Frontend + Encoder. Note that this method is used by asr_inference.py+ Args:+ speech: (Batch, Length, ...)+ speech_lengths: (Batch, )+ ind: int+ """+ with autocast(False):+ # Data augmentation+ if self.specaug is not None and self.training:+ speech, speech_lengths = self.specaug(speech, speech_lengths)+ # Normalization for feature: e.g. Global-CMVN, Utterance-CMVN+ if self.normalize is not None:+ speech, speech_lengths = self.normalize(speech, speech_lengths)+ # Forward encoder+ # feats: (Batch, Length, Dim)+ # -> encoder_out: (Batch, Length2, Dim2)+ if self.encoder.interctc_use_conditioning:+ encoder_out, encoder_out_lens, _ = self.encoder(speech, speech_lengths, ctc=self.ctc)+ else:+ encoder_out, encoder_out_lens, _ = self.encoder(speech, speech_lengths)+ intermediate_outs = None+ if isinstance(encoder_out, tuple):+ intermediate_outs = encoder_out[1]+ encoder_out = encoder_out[0]+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/lcbnet/model.py#L302" target="_blank" class="src-more">View full source on GitHub →</a></details></div>+<div class="api-detail" id="e123"><span class="dtag badge-m">method</span><h2><code>LCBNet.init_beam_search(**kwargs)</code></h2><div class="dmod">funasr.models.lcbnet.LCBNet · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/lcbnet/model.py#L400" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Init beam search.</p>+<h4>Args:</h4>+<ul>+<li><code>**kwargs</code> — Additional keyword arguments.</li>+</ul>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def init_beam_search(+ self,+ **kwargs,+ ):+ """Init beam search.++ Args:+ **kwargs: Additional keyword arguments.+ """+ from funasr.models.transformer.search import BeamSearch+ from funasr.models.transformer.scorers.ctc import CTCPrefixScorer+ from funasr.models.transformer.scorers.length_bonus import LengthBonus++ # 1. Build ASR model+ scorers = {}++ if self.ctc != None:+ ctc = CTCPrefixScorer(ctc=self.ctc, eos=self.eos)+ scorers.update(ctc=ctc)+ token_list = kwargs.get("token_list")+ scorers.update(+ decoder=self.decoder,+ length_bonus=LengthBonus(len(token_list)),+ )++ # 3. Build ngram model+ # ngram is not supported now+ ngram = None+ scorers["ngram"] = ngram++</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/lcbnet/model.py#L400" target="_blank" class="src-more">View full source on GitHub →</a></details></div>+<div class="api-detail" id="e124"><span class="dtag badge-m">method</span><h2><code>LCBNet.inference(data_in, data_lengths, key, tokenizer, frontend, **kwargs)</code></h2><div class="dmod">funasr.models.lcbnet.LCBNet · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/lcbnet/model.py#L450" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Run inference on input data.</p>+<h4>Args:</h4>+<ul>+<li><code>data_in</code> — Input data (audio samples, file paths, or text).</li>+<li><code>data_lengths</code> — Lengths of each input sample in the batch.</li>+<li><code>key</code> — Sample identifiers.</li>+<li><code>tokenizer</code> — Tokenizer instance for text encoding/decoding.</li>+<li><code>frontend</code> — Audio frontend for feature extraction.</li>+<li><code>**kwargs</code> — Additional keyword arguments.</li>+</ul>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def inference(+ self,+ data_in,+ data_lengths=None,+ key: list = None,+ tokenizer=None,+ frontend=None,+ **kwargs,+ ):++ """Run inference on input data.++ Args:+ data_in: Input data (audio samples, file paths, or text).+ data_lengths: Lengths of each input sample in the batch.+ key: Sample identifiers.+ tokenizer: Tokenizer instance for text encoding/decoding.+ frontend: Audio frontend for feature extraction.+ **kwargs: Additional keyword arguments.+ """+ if kwargs.get("batch_size", 1) > 1:+ raise NotImplementedError("batch decoding is not implemented")++ # init beamsearch+ if self.beam_search is None:+ logging.info("enable beam_search")+ self.init_beam_search(**kwargs)+ self.nbest = kwargs.get("nbest", 1)++ meta_data = {}+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/lcbnet/model.py#L450" target="_blank" class="src-more">View full source on GitHub →</a></details></div>+<div class="api-detail" id="e125"><span class="dtag badge-c">class</span><h2>LLMASR</h2><div class="dmod">funasr.models.llm_asr · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/llm_asr/model.py#L28" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><ul>+<li><code>LLM</code> — ASR: Large Language Model based Speech Recognition.</li>+</ul>+<p>Combines audio encoder with LLM decoder for speech-to-text.</p>+<ul>+<li><code>Output</code> — {"key": str, "text": str}</li>+</ul>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code>class LLMASR(nn.Module):+ """LLM-ASR: Large Language Model based Speech Recognition.++ Combines audio encoder with LLM decoder for speech-to-text.+ Output: {"key": str, "text": str}+ """++ def __init__(+ self,+ specaug: str = None,+ specaug_conf: dict = None,+ normalize: str = None,+ normalize_conf: dict = None,+ audio_encoder: str = None,+ audio_encoder_conf: dict = None,+ audio_adaptor: str = None,+ audio_adaptor_conf: dict = None,+ decoder: str = None,+ decoder_conf: dict = None,+ ctc: str = None,+ ctc_conf: dict = None,+ ctc_weight: float = 0.5,+ llm: str = None,+ llm_conf: dict = None,+ input_size: int = 80,+ vocab_size: int = -1,+ ignore_id: int = -1,+ blank_id: int = 0,+ sos: int = 1,+ eos: int = 2,+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/llm_asr/model.py#L28" target="_blank" class="src-more">View full source on GitHub →</a></details><h3 class="mt">Methods</h3><div class="mblk"><code>.forward(speech, speech_lengths, text, text_lengths, input_ids, attention_mask, labels_ids, label_mask, audio_mask, **kwargs)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/llm_asr/model.py#L188" target="_blank" class="gh-link">L188</a><div class="mdoc"><p>Encoder + Decoder + Calc loss</p>+<h4>Args:</h4>+<ul>+<li><code>speech</code> — (Batch, Length, ...)</li>+<li><code>speech_lengths</code> — (Batch, )</li>+<li><code>text</code> — (Batch, Length)</li>+<li><code>text_lengths</code> — (Batch,)</li>+</ul>+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def forward(+ self,+ speech: torch.Tensor,+ speech_lengths: torch.Tensor,+ text: torch.Tensor,+ text_lengths: torch.Tensor,+ input_ids: torch.Tensor,+ attention_mask: torch.Tensor,+ labels_ids: torch.Tensor,+ label_mask: torch.Tensor,+ audio_mask: torch.Tensor,+ **kwargs,+ ) -> Tuple[torch.Tensor, Dict[str, torch.Tensor], torch.Tensor]:+ """Encoder + Decoder + Calc loss+ Args:+ speech: (Batch, Length, ...)+ speech_lengths: (Batch, )+ text: (Batch, Length)+ text_lengths: (Batch,)+ """+ if len(text_lengths.size()) > 1:+ text_lengths = text_lengths[:, 0]+ if len(speech_lengths.size()) > 1:+ speech_lengths = speech_lengths[:, 0]++ batch_size = speech.shape[0]++ # audio encoder+ encoder_out, encoder_out_lens = self.encode(speech, speech_lengths)++</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/llm_asr/model.py#L188" target="_blank" class="src-more">View full source →</a></details></div><div class="mblk"><code>.encode(speech, speech_lengths, **kwargs)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/llm_asr/model.py#L258" target="_blank" class="gh-link">L258</a><div class="mdoc"><p>Encode.</p>+<h4>Args:</h4>+<ul>+<li><code>speech</code> — Speech audio tensor, shape (batch, time).</li>+<li><code>speech_lengths</code> — Length of each speech sample.</li>+<li><code>**kwargs</code> — Additional keyword arguments.</li>+</ul>+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def encode(+ self,+ speech: torch.Tensor,+ speech_lengths: torch.Tensor,+ **kwargs,+ ):+ """Encode.++ Args:+ speech: Speech audio tensor, shape (batch, time).+ speech_lengths: Length of each speech sample.+ **kwargs: Additional keyword arguments.+ """+ speech = speech.permute(0, 2, 1)+ res = self.audio_encoder(speech)+ if isinstance(res, (list, tuple)):+ encoder_out, encoder_out_lens = res[0], res[1]+ else:+ encoder_out, encoder_out_lens = res, speech_lengths+ return encoder_out, encoder_out_lens+</code></pre></details></div><div class="mblk"><code>.inference(data_in, data_lengths, key, tokenizer, frontend, **kwargs)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/llm_asr/model.py#L279" target="_blank" class="gh-link">L279</a><div class="mdoc"><p>Run inference on input data.</p>+<h4>Args:</h4>+<ul>+<li><code>data_in</code> — Input data (audio samples, file paths, or text).</li>+<li><code>data_lengths</code> — Lengths of each input sample in the batch.</li>+<li><code>key</code> — Sample identifiers.</li>+<li><code>tokenizer</code> — Tokenizer instance for text encoding/decoding.</li>+<li><code>frontend</code> — Audio frontend for feature extraction.</li>+<li><code>**kwargs</code> — Additional keyword arguments.</li>+</ul>+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def inference(+ self,+ data_in,+ data_lengths=None,+ key: list = None,+ tokenizer=None,+ frontend=None,+ **kwargs,+ ):++ """Run inference on input data.++ Args:+ data_in: Input data (audio samples, file paths, or text).+ data_lengths: Lengths of each input sample in the batch.+ key: Sample identifiers.+ tokenizer: Tokenizer instance for text encoding/decoding.+ frontend: Audio frontend for feature extraction.+ **kwargs: Additional keyword arguments.+ """+ prompt = kwargs.get("prompt", "Transcribe speech to text.")++ if kwargs.get("batch_size", 1) > 1:+ raise NotImplementedError("batch decoding is not implemented")++ meta_data = {}+ if (+ isinstance(data_in, torch.Tensor) and kwargs.get("data_type", "sound") == "fbank"+ ): # fbank+ speech, speech_lengths = data_in, data_lengths+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/llm_asr/model.py#L279" target="_blank" class="src-more">View full source →</a></details></div></div>+<div class="api-detail" id="e126"><span class="dtag badge-m">method</span><h2><code>LLMASR.forward(speech, speech_lengths, text, text_lengths, input_ids, attention_mask, labels_ids, label_mask, audio_mask, **kwargs)</code></h2><div class="dmod">funasr.models.llm_asr.LLMASR · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/llm_asr/model.py#L188" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Encoder + Decoder + Calc loss</p>+<h4>Args:</h4>+<ul>+<li><code>speech</code> — (Batch, Length, ...)</li>+<li><code>speech_lengths</code> — (Batch, )</li>+<li><code>text</code> — (Batch, Length)</li>+<li><code>text_lengths</code> — (Batch,)</li>+</ul>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def forward(+ self,+ speech: torch.Tensor,+ speech_lengths: torch.Tensor,+ text: torch.Tensor,+ text_lengths: torch.Tensor,+ input_ids: torch.Tensor,+ attention_mask: torch.Tensor,+ labels_ids: torch.Tensor,+ label_mask: torch.Tensor,+ audio_mask: torch.Tensor,+ **kwargs,+ ) -> Tuple[torch.Tensor, Dict[str, torch.Tensor], torch.Tensor]:+ """Encoder + Decoder + Calc loss+ Args:+ speech: (Batch, Length, ...)+ speech_lengths: (Batch, )+ text: (Batch, Length)+ text_lengths: (Batch,)+ """+ if len(text_lengths.size()) > 1:+ text_lengths = text_lengths[:, 0]+ if len(speech_lengths.size()) > 1:+ speech_lengths = speech_lengths[:, 0]++ batch_size = speech.shape[0]++ # audio encoder+ encoder_out, encoder_out_lens = self.encode(speech, speech_lengths)++</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/llm_asr/model.py#L188" target="_blank" class="src-more">View full source on GitHub →</a></details></div>+<div class="api-detail" id="e127"><span class="dtag badge-m">method</span><h2><code>LLMASR.encode(speech, speech_lengths, **kwargs)</code></h2><div class="dmod">funasr.models.llm_asr.LLMASR · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/llm_asr/model.py#L258" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Encode.</p>+<h4>Args:</h4>+<ul>+<li><code>speech</code> — Speech audio tensor, shape (batch, time).</li>+<li><code>speech_lengths</code> — Length of each speech sample.</li>+<li><code>**kwargs</code> — Additional keyword arguments.</li>+</ul>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def encode(+ self,+ speech: torch.Tensor,+ speech_lengths: torch.Tensor,+ **kwargs,+ ):+ """Encode.++ Args:+ speech: Speech audio tensor, shape (batch, time).+ speech_lengths: Length of each speech sample.+ **kwargs: Additional keyword arguments.+ """+ speech = speech.permute(0, 2, 1)+ res = self.audio_encoder(speech)+ if isinstance(res, (list, tuple)):+ encoder_out, encoder_out_lens = res[0], res[1]+ else:+ encoder_out, encoder_out_lens = res, speech_lengths+ return encoder_out, encoder_out_lens+</code></pre></details></div>+<div class="api-detail" id="e128"><span class="dtag badge-m">method</span><h2><code>LLMASR.inference(data_in, data_lengths, key, tokenizer, frontend, **kwargs)</code></h2><div class="dmod">funasr.models.llm_asr.LLMASR · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/llm_asr/model.py#L279" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Run inference on input data.</p>+<h4>Args:</h4>+<ul>+<li><code>data_in</code> — Input data (audio samples, file paths, or text).</li>+<li><code>data_lengths</code> — Lengths of each input sample in the batch.</li>+<li><code>key</code> — Sample identifiers.</li>+<li><code>tokenizer</code> — Tokenizer instance for text encoding/decoding.</li>+<li><code>frontend</code> — Audio frontend for feature extraction.</li>+<li><code>**kwargs</code> — Additional keyword arguments.</li>+</ul>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def inference(+ self,+ data_in,+ data_lengths=None,+ key: list = None,+ tokenizer=None,+ frontend=None,+ **kwargs,+ ):++ """Run inference on input data.++ Args:+ data_in: Input data (audio samples, file paths, or text).+ data_lengths: Lengths of each input sample in the batch.+ key: Sample identifiers.+ tokenizer: Tokenizer instance for text encoding/decoding.+ frontend: Audio frontend for feature extraction.+ **kwargs: Additional keyword arguments.+ """+ prompt = kwargs.get("prompt", "Transcribe speech to text.")++ if kwargs.get("batch_size", 1) > 1:+ raise NotImplementedError("batch decoding is not implemented")++ meta_data = {}+ if (+ isinstance(data_in, torch.Tensor) and kwargs.get("data_type", "sound") == "fbank"+ ): # fbank+ speech, speech_lengths = data_in, data_lengths+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/llm_asr/model.py#L279" target="_blank" class="src-more">View full source on GitHub →</a></details></div>+<div class="api-detail" id="e129"><span class="dtag badge-c">class</span><h2>LLMASR2</h2><div class="dmod">funasr.models.llm_asr · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/llm_asr/model.py#L403" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p class="muted">No documentation yet.</p></div><details class="src-block"><summary>📄 Source code</summary><pre><code>class LLMASR2(nn.Module):+ """ """++ def __init__(+ self,+ specaug: str = None,+ specaug_conf: dict = None,+ normalize: str = None,+ normalize_conf: dict = None,+ audio_encoder: str = None,+ audio_encoder_conf: dict = None,+ audio_adaptor: str = None,+ audio_adaptor_conf: dict = None,+ decoder: str = None,+ decoder_conf: dict = None,+ ctc: str = None,+ ctc_conf: dict = None,+ ctc_weight: float = 0.5,+ llm: str = None,+ llm_conf: dict = None,+ input_size: int = 80,+ vocab_size: int = -1,+ ignore_id: int = -1,+ blank_id: int = 0,+ sos: int = 1,+ eos: int = 2,+ lsm_weight: float = 0.0,+ length_normalized_loss: bool = False,+ report_cer: bool = True,+ report_wer: bool = True,+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/llm_asr/model.py#L403" target="_blank" class="src-more">View full source on GitHub →</a></details><h3 class="mt">Methods</h3><div class="mblk"><code>.forward(speech, speech_lengths, input_ids, attention_mask, labels_ids, fbank_beg, fbank_mask, **kwargs)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/llm_asr/model.py#L561" target="_blank" class="gh-link">L561</a><div class="mdoc"><p>Encoder + Decoder + Calc loss</p>+<h4>Args:</h4>+<ul>+<li><code>speech</code> — (Batch, Length, ...)</li>+<li><code>speech_lengths</code> — (Batch, )</li>+<li><code>text</code> — (Batch, Length)</li>+<li><code>text_lengths</code> — (Batch,)</li>+</ul>+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def forward(+ self,+ speech: torch.Tensor,+ speech_lengths: torch.Tensor,+ input_ids: torch.Tensor,+ attention_mask: torch.Tensor,+ labels_ids: torch.Tensor,+ fbank_beg: torch.Tensor,+ fbank_mask: torch.Tensor,+ **kwargs,+ ) -> Tuple[torch.Tensor, Dict[str, torch.Tensor], torch.Tensor]:+ """Encoder + Decoder + Calc loss+ Args:+ speech: (Batch, Length, ...)+ speech_lengths: (Batch, )+ text: (Batch, Length)+ text_lengths: (Batch,)+ """+ # import pdb;+ # pdb.set_trace()+ if len(speech_lengths.size()) > 1:+ speech_lengths = speech_lengths[:, 0]++ batch_size, frames, _ = speech.shape++ with torch.cuda.amp.autocast(enabled=False):+ # audio encoder+ encoder_out, encoder_out_lens = self.encode(speech, speech_lengths)++ # audio_adaptor+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/llm_asr/model.py#L561" target="_blank" class="src-more">View full source →</a></details></div><div class="mblk"><code>.encode(speech, speech_lengths)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/llm_asr/model.py#L654" target="_blank" class="gh-link">L654</a><div class="mdoc"><p>Encode.</p>+<h4>Args:</h4>+<ul>+<li><code>speech</code> — Speech audio tensor, shape (batch, time).</li>+<li><code>speech_lengths</code> — Length of each speech sample.</li>+</ul>+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def encode(self, speech, speech_lengths):+ # audio encoder+ """Encode.++ Args:+ speech: Speech audio tensor, shape (batch, time).+ speech_lengths: Length of each speech sample.+ """+ encoder_out, encoder_out_lens = self.audio_encoder(speech.permute(0, 2, 1), speech_lengths)++ return encoder_out, encoder_out_lens+</code></pre></details></div><div class="mblk"><code>.data_template(data)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/llm_asr/model.py#L666" target="_blank" class="gh-link">L666</a><div class="mdoc"><p>Data template.</p>+<h4>Args:</h4>+<ul>+<li><code>data</code> — TODO.</li>+</ul>+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def data_template(self, data):+ """Data template.++ Args:+ data: TODO.+ """+ system, user, assistant = [], [], []+ for i, item in enumerate(data):+ role = item["role"]+ content = item["content"]+ if role == "system":+ system.append(content)+ elif role == "user":+ user.append(content)+ elif role == "assistant":+ assistant.append(content)++ system = system * len(user)++ contents = {+ "system": system,+ "user": user,+ "assistant": assistant,+ }++ return contents+</code></pre></details></div><div class="mblk"><code>.data_load_speech(contents, tokenizer, frontend, meta_data, **kwargs)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/llm_asr/model.py#L693" target="_blank" class="gh-link">L693</a><div class="mdoc"><p>Data load speech.</p>+<h4>Args:</h4>+<ul>+<li><code>contents</code> — TODO.</li>+<li><code>tokenizer</code> — Tokenizer instance for text encoding/decoding.</li>+<li><code>frontend</code> — Audio frontend for feature extraction.</li>+<li><code>meta_data</code> — TODO.</li>+<li><code>**kwargs</code> — Additional keyword arguments.</li>+</ul>+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def data_load_speech(self, contents: dict, tokenizer, frontend, meta_data={}, **kwargs):++ """Data load speech.++ Args:+ contents: TODO.+ tokenizer: Tokenizer instance for text encoding/decoding.+ frontend: Audio frontend for feature extraction.+ meta_data: TODO.+ **kwargs: Additional keyword arguments.+ """+ system = contents["system"]+ user = contents["user"]+ assistant = contents["assistant"]+ pattern = re.compile(r"(<\|startofspeech\|>.*?<\|endofspeech\|>)")+ input_ids, labels, source_ids, target_ids, fbank, fbank_lens, fbank_mask, fbank_beg = (+ [],+ [],+ [],+ [],+ [],+ [],+ [],+ [],+ )++ for i, (system_prompt, user_prompt, target_out) in enumerate(zip(system, user, assistant)):++ source_input = f"<|im_start|>system\n{system_prompt}<|im_end|>\n<|im_start|>user\n{user_prompt}<|im_end|>\n<|im_start|>assistant\n"++</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/llm_asr/model.py#L693" target="_blank" class="src-more">View full source →</a></details></div><div class="mblk"><code>.inference(data_in, data_lengths, key, tokenizer, frontend, **kwargs)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/llm_asr/model.py#L820" target="_blank" class="gh-link">L820</a><div class="mdoc"><p>Run inference on input data.</p>+<h4>Args:</h4>+<ul>+<li><code>data_in</code> — Input data (audio samples, file paths, or text).</li>+<li><code>data_lengths</code> — Lengths of each input sample in the batch.</li>+<li><code>key</code> — Sample identifiers.</li>+<li><code>tokenizer</code> — Tokenizer instance for text encoding/decoding.</li>+<li><code>frontend</code> — Audio frontend for feature extraction.</li>+<li><code>**kwargs</code> — Additional keyword arguments.</li>+</ul>+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def inference(+ self,+ data_in,+ data_lengths=None,+ key: list = None,+ tokenizer=None,+ frontend=None,+ **kwargs,+ ):++ """Run inference on input data.++ Args:+ data_in: Input data (audio samples, file paths, or text).+ data_lengths: Lengths of each input sample in the batch.+ key: Sample identifiers.+ tokenizer: Tokenizer instance for text encoding/decoding.+ frontend: Audio frontend for feature extraction.+ **kwargs: Additional keyword arguments.+ """+ meta_data = {}+ prompt = kwargs.get("prompt", None)++ if kwargs.get("batch_size", 1) > 1:+ raise NotImplementedError("batch decoding is not implemented")++ contents = self.data_template(data_in[0])+ output = self.data_load_speech(contents, tokenizer, frontend, meta_data=meta_data, **kwargs)+ batch = to_device(output, kwargs["device"])++</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/llm_asr/model.py#L820" target="_blank" class="src-more">View full source →</a></details></div></div>+<div class="api-detail" id="e130"><span class="dtag badge-m">method</span><h2><code>LLMASR2.forward(speech, speech_lengths, input_ids, attention_mask, labels_ids, fbank_beg, fbank_mask, **kwargs)</code></h2><div class="dmod">funasr.models.llm_asr.LLMASR2 · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/llm_asr/model.py#L561" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Encoder + Decoder + Calc loss</p>+<h4>Args:</h4>+<ul>+<li><code>speech</code> — (Batch, Length, ...)</li>+<li><code>speech_lengths</code> — (Batch, )</li>+<li><code>text</code> — (Batch, Length)</li>+<li><code>text_lengths</code> — (Batch,)</li>+</ul>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def forward(+ self,+ speech: torch.Tensor,+ speech_lengths: torch.Tensor,+ input_ids: torch.Tensor,+ attention_mask: torch.Tensor,+ labels_ids: torch.Tensor,+ fbank_beg: torch.Tensor,+ fbank_mask: torch.Tensor,+ **kwargs,+ ) -> Tuple[torch.Tensor, Dict[str, torch.Tensor], torch.Tensor]:+ """Encoder + Decoder + Calc loss+ Args:+ speech: (Batch, Length, ...)+ speech_lengths: (Batch, )+ text: (Batch, Length)+ text_lengths: (Batch,)+ """+ # import pdb;+ # pdb.set_trace()+ if len(speech_lengths.size()) > 1:+ speech_lengths = speech_lengths[:, 0]++ batch_size, frames, _ = speech.shape++ with torch.cuda.amp.autocast(enabled=False):+ # audio encoder+ encoder_out, encoder_out_lens = self.encode(speech, speech_lengths)++ # audio_adaptor+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/llm_asr/model.py#L561" target="_blank" class="src-more">View full source on GitHub →</a></details></div>+<div class="api-detail" id="e131"><span class="dtag badge-m">method</span><h2><code>LLMASR2.encode(speech, speech_lengths)</code></h2><div class="dmod">funasr.models.llm_asr.LLMASR2 · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/llm_asr/model.py#L654" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Encode.</p>+<h4>Args:</h4>+<ul>+<li><code>speech</code> — Speech audio tensor, shape (batch, time).</li>+<li><code>speech_lengths</code> — Length of each speech sample.</li>+</ul>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def encode(self, speech, speech_lengths):+ # audio encoder+ """Encode.++ Args:+ speech: Speech audio tensor, shape (batch, time).+ speech_lengths: Length of each speech sample.+ """+ encoder_out, encoder_out_lens = self.audio_encoder(speech.permute(0, 2, 1), speech_lengths)++ return encoder_out, encoder_out_lens+</code></pre></details></div>+<div class="api-detail" id="e132"><span class="dtag badge-m">method</span><h2><code>LLMASR2.data_template(data)</code></h2><div class="dmod">funasr.models.llm_asr.LLMASR2 · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/llm_asr/model.py#L666" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Data template.</p>+<h4>Args:</h4>+<ul>+<li><code>data</code> — TODO.</li>+</ul>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def data_template(self, data):+ """Data template.++ Args:+ data: TODO.+ """+ system, user, assistant = [], [], []+ for i, item in enumerate(data):+ role = item["role"]+ content = item["content"]+ if role == "system":+ system.append(content)+ elif role == "user":+ user.append(content)+ elif role == "assistant":+ assistant.append(content)++ system = system * len(user)++ contents = {+ "system": system,+ "user": user,+ "assistant": assistant,+ }++ return contents+</code></pre></details></div>+<div class="api-detail" id="e133"><span class="dtag badge-m">method</span><h2><code>LLMASR2.data_load_speech(contents, tokenizer, frontend, meta_data, **kwargs)</code></h2><div class="dmod">funasr.models.llm_asr.LLMASR2 · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/llm_asr/model.py#L693" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Data load speech.</p>+<h4>Args:</h4>+<ul>+<li><code>contents</code> — TODO.</li>+<li><code>tokenizer</code> — Tokenizer instance for text encoding/decoding.</li>+<li><code>frontend</code> — Audio frontend for feature extraction.</li>+<li><code>meta_data</code> — TODO.</li>+<li><code>**kwargs</code> — Additional keyword arguments.</li>+</ul>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def data_load_speech(self, contents: dict, tokenizer, frontend, meta_data={}, **kwargs):++ """Data load speech.++ Args:+ contents: TODO.+ tokenizer: Tokenizer instance for text encoding/decoding.+ frontend: Audio frontend for feature extraction.+ meta_data: TODO.+ **kwargs: Additional keyword arguments.+ """+ system = contents["system"]+ user = contents["user"]+ assistant = contents["assistant"]+ pattern = re.compile(r"(<\|startofspeech\|>.*?<\|endofspeech\|>)")+ input_ids, labels, source_ids, target_ids, fbank, fbank_lens, fbank_mask, fbank_beg = (+ [],+ [],+ [],+ [],+ [],+ [],+ [],+ [],+ )++ for i, (system_prompt, user_prompt, target_out) in enumerate(zip(system, user, assistant)):++ source_input = f"<|im_start|>system\n{system_prompt}<|im_end|>\n<|im_start|>user\n{user_prompt}<|im_end|>\n<|im_start|>assistant\n"++</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/llm_asr/model.py#L693" target="_blank" class="src-more">View full source on GitHub →</a></details></div>+<div class="api-detail" id="e134"><span class="dtag badge-m">method</span><h2><code>LLMASR2.inference(data_in, data_lengths, key, tokenizer, frontend, **kwargs)</code></h2><div class="dmod">funasr.models.llm_asr.LLMASR2 · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/llm_asr/model.py#L820" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Run inference on input data.</p>+<h4>Args:</h4>+<ul>+<li><code>data_in</code> — Input data (audio samples, file paths, or text).</li>+<li><code>data_lengths</code> — Lengths of each input sample in the batch.</li>+<li><code>key</code> — Sample identifiers.</li>+<li><code>tokenizer</code> — Tokenizer instance for text encoding/decoding.</li>+<li><code>frontend</code> — Audio frontend for feature extraction.</li>+<li><code>**kwargs</code> — Additional keyword arguments.</li>+</ul>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def inference(+ self,+ data_in,+ data_lengths=None,+ key: list = None,+ tokenizer=None,+ frontend=None,+ **kwargs,+ ):++ """Run inference on input data.++ Args:+ data_in: Input data (audio samples, file paths, or text).+ data_lengths: Lengths of each input sample in the batch.+ key: Sample identifiers.+ tokenizer: Tokenizer instance for text encoding/decoding.+ frontend: Audio frontend for feature extraction.+ **kwargs: Additional keyword arguments.+ """+ meta_data = {}+ prompt = kwargs.get("prompt", None)++ if kwargs.get("batch_size", 1) > 1:+ raise NotImplementedError("batch decoding is not implemented")++ contents = self.data_template(data_in[0])+ output = self.data_load_speech(contents, tokenizer, frontend, meta_data=meta_data, **kwargs)+ batch = to_device(output, kwargs["device"])++</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/llm_asr/model.py#L820" target="_blank" class="src-more">View full source on GitHub →</a></details></div>+<div class="api-detail" id="e135"><span class="dtag badge-c">class</span><h2>LLMASR3</h2><div class="dmod">funasr.models.llm_asr · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/llm_asr/model.py#L947" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p class="muted">No documentation yet.</p></div><details class="src-block"><summary>📄 Source code</summary><pre><code>class LLMASR3(LLMASR2):+ """ """++ def __init__(+ self,+ *args,+ **kwargs,+ ):++ """Initialize LLMASR3.++ Args:+ *args: Variable positional arguments.+ **kwargs: Additional keyword arguments.+ """+ super().__init__(*args, **kwargs)++ def encode(self, speech, speech_lengths):+ # audio encoder+ """Encode.++ Args:+ speech: Speech audio tensor, shape (batch, time).+ speech_lengths: Length of each speech sample.+ """+ encoder_out, encoder_out_lens = self.audio_encoder(speech, speech_lengths)+ return encoder_out, encoder_out_lens+</code></pre></details><h3 class="mt">Methods</h3><div class="mblk"><code>.encode(speech, speech_lengths)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/llm_asr/model.py#L964" target="_blank" class="gh-link">L964</a><div class="mdoc"><p>Encode.</p>+<h4>Args:</h4>+<ul>+<li><code>speech</code> — Speech audio tensor, shape (batch, time).</li>+<li><code>speech_lengths</code> — Length of each speech sample.</li>+</ul>+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def encode(self, speech, speech_lengths):+ # audio encoder+ """Encode.++ Args:+ speech: Speech audio tensor, shape (batch, time).+ speech_lengths: Length of each speech sample.+ """+ encoder_out, encoder_out_lens = self.audio_encoder(speech, speech_lengths)+ return encoder_out, encoder_out_lens+</code></pre></details></div></div>+<div class="api-detail" id="e136"><span class="dtag badge-m">method</span><h2><code>LLMASR3.encode(speech, speech_lengths)</code></h2><div class="dmod">funasr.models.llm_asr.LLMASR3 · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/llm_asr/model.py#L964" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Encode.</p>+<h4>Args:</h4>+<ul>+<li><code>speech</code> — Speech audio tensor, shape (batch, time).</li>+<li><code>speech_lengths</code> — Length of each speech sample.</li>+</ul>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def encode(self, speech, speech_lengths):+ # audio encoder+ """Encode.++ Args:+ speech: Speech audio tensor, shape (batch, time).+ speech_lengths: Length of each speech sample.+ """+ encoder_out, encoder_out_lens = self.audio_encoder(speech, speech_lengths)+ return encoder_out, encoder_out_lens+</code></pre></details></div>+<div class="api-detail" id="e137"><span class="dtag badge-c">class</span><h2>LLMASR4</h2><div class="dmod">funasr.models.llm_asr · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/llm_asr/model.py#L977" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p class="muted">No documentation yet.</p></div><details class="src-block"><summary>📄 Source code</summary><pre><code>class LLMASR4(nn.Module):+ """ """++ def __init__(+ self,+ specaug: str = None,+ specaug_conf: dict = None,+ normalize: str = None,+ normalize_conf: dict = None,+ audio_encoder: str = None,+ audio_encoder_conf: dict = None,+ audio_adaptor: str = None,+ audio_adaptor_conf: dict = None,+ decoder: str = None,+ decoder_conf: dict = None,+ ctc: str = None,+ ctc_conf: dict = None,+ ctc_weight: float = 0.5,+ llm: str = None,+ llm_conf: dict = None,+ input_size: int = 80,+ vocab_size: int = -1,+ ignore_id: int = -1,+ blank_id: int = 0,+ sos: int = 1,+ eos: int = 2,+ lsm_weight: float = 0.0,+ length_normalized_loss: bool = False,+ report_cer: bool = True,+ report_wer: bool = True,+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/llm_asr/model.py#L977" target="_blank" class="src-more">View full source on GitHub →</a></details><h3 class="mt">Methods</h3><div class="mblk"><code>.forward(speech, speech_lengths, input_ids, attention_mask, labels_ids, fbank_beg, fbank_mask, **kwargs)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/llm_asr/model.py#L1135" target="_blank" class="gh-link">L1135</a><div class="mdoc"><p>Encoder + Decoder + Calc loss</p>+<h4>Args:</h4>+<ul>+<li><code>speech</code> — (Batch, Length, ...)</li>+<li><code>speech_lengths</code> — (Batch, )</li>+<li><code>text</code> — (Batch, Length)</li>+<li><code>text_lengths</code> — (Batch,)</li>+</ul>+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def forward(+ self,+ speech: torch.Tensor,+ speech_lengths: torch.Tensor,+ input_ids: torch.Tensor,+ attention_mask: torch.Tensor,+ labels_ids: torch.Tensor,+ fbank_beg: torch.Tensor,+ fbank_mask: torch.Tensor,+ **kwargs,+ ) -> Tuple[torch.Tensor, Dict[str, torch.Tensor], torch.Tensor]:+ """Encoder + Decoder + Calc loss+ Args:+ speech: (Batch, Length, ...)+ speech_lengths: (Batch, )+ text: (Batch, Length)+ text_lengths: (Batch,)+ """+ # import pdb+ #+ # pdb.set_trace()+ if len(speech_lengths.size()) > 1:+ speech_lengths = speech_lengths[:, 0]++ batch_size_speech, frames, _ = speech.shape+ batch_size, token_num = input_ids.shape++ with torch.cuda.amp.autocast(enabled=False):+ # audio encoder+ encoder_out, encoder_out_lens = self.encode(speech, speech_lengths)+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/llm_asr/model.py#L1135" target="_blank" class="src-more">View full source →</a></details></div><div class="mblk"><code>.encode(speech, speech_lengths)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/llm_asr/model.py#L1246" target="_blank" class="gh-link">L1246</a><div class="mdoc"><p>Encode.</p>+<h4>Args:</h4>+<ul>+<li><code>speech</code> — Speech audio tensor, shape (batch, time).</li>+<li><code>speech_lengths</code> — Length of each speech sample.</li>+</ul>+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def encode(self, speech, speech_lengths):+ # audio encoder+ """Encode.++ Args:+ speech: Speech audio tensor, shape (batch, time).+ speech_lengths: Length of each speech sample.+ """+ encoder_out, encoder_out_lens = self.audio_encoder(speech.permute(0, 2, 1), speech_lengths)++ return encoder_out, encoder_out_lens+</code></pre></details></div><div class="mblk"><code>.data_template(data)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/llm_asr/model.py#L1258" target="_blank" class="gh-link">L1258</a><div class="mdoc"><p>Data template.</p>+<h4>Args:</h4>+<ul>+<li><code>data</code> — TODO.</li>+</ul>+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def data_template(self, data):+ """Data template.++ Args:+ data: TODO.+ """+ system, user, assistant = [], [], []+ for i, item in enumerate(data):+ role = item["role"]+ content = item["content"]+ if role == "system":+ system.append(content)+ elif role == "user":+ user.append(content)+ elif role == "assistant":+ assistant.append(content)++ system = system * len(user)++ contents = {+ "system": system,+ "user": user,+ "assistant": assistant,+ }++ return contents+</code></pre></details></div><div class="mblk"><code>.data_load_speech(contents, tokenizer, frontend, meta_data, **kwargs)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/llm_asr/model.py#L1285" target="_blank" class="gh-link">L1285</a><div class="mdoc"><p>Data load speech.</p>+<h4>Args:</h4>+<ul>+<li><code>contents</code> — TODO.</li>+<li><code>tokenizer</code> — Tokenizer instance for text encoding/decoding.</li>+<li><code>frontend</code> — Audio frontend for feature extraction.</li>+<li><code>meta_data</code> — TODO.</li>+<li><code>**kwargs</code> — Additional keyword arguments.</li>+</ul>+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def data_load_speech(self, contents: dict, tokenizer, frontend, meta_data={}, **kwargs):++ """Data load speech.++ Args:+ contents: TODO.+ tokenizer: Tokenizer instance for text encoding/decoding.+ frontend: Audio frontend for feature extraction.+ meta_data: TODO.+ **kwargs: Additional keyword arguments.+ """+ system = contents["system"]+ user = contents["user"]+ assistant = contents["assistant"]+ pattern = re.compile(r"(<\|startofspeech\|>.*?<\|endofspeech\|>)")++ input_ids, labels, fbank, fbank_lens, fbank_mask, fbank_beg, fake_token_len = (+ [],+ [],+ [],+ [],+ [],+ [],+ [],+ )+ input_source_ids = []+ for i, (system_prompt, user_prompt, target_out) in enumerate(zip(system, user, assistant)):+ if i >= kwargs.get("multiturn_num_max", 5):+ break+ if len(input_ids) > kwargs.get("max_token_length", 1500):+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/llm_asr/model.py#L1285" target="_blank" class="src-more">View full source →</a></details></div><div class="mblk"><code>.inference_prepare(data_in, data_lengths, key, tokenizer, frontend, **kwargs)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/llm_asr/model.py#L1433" target="_blank" class="gh-link">L1433</a><div class="mdoc"><p>Inference prepare.</p>+<h4>Args:</h4>+<ul>+<li><code>data_in</code> — Input data (audio samples, file paths, or text).</li>+<li><code>data_lengths</code> — Lengths of each input sample in the batch.</li>+<li><code>key</code> — Sample identifiers.</li>+<li><code>tokenizer</code> — Tokenizer instance for text encoding/decoding.</li>+<li><code>frontend</code> — Audio frontend for feature extraction.</li>+<li><code>**kwargs</code> — Additional keyword arguments.</li>+</ul>+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def inference_prepare(+ self,+ data_in,+ data_lengths=None,+ key: list = None,+ tokenizer=None,+ frontend=None,+ **kwargs,+ ):++ """Inference prepare.++ Args:+ data_in: Input data (audio samples, file paths, or text).+ data_lengths: Lengths of each input sample in the batch.+ key: Sample identifiers.+ tokenizer: Tokenizer instance for text encoding/decoding.+ frontend: Audio frontend for feature extraction.+ **kwargs: Additional keyword arguments.+ """+ meta_data = {}+ prompt = kwargs.get("prompt", None)++ if kwargs.get("batch_size", 1) > 1:+ raise NotImplementedError("batch decoding is not implemented")++ contents = self.data_template(data_in[0])+ output = self.data_load_speech(contents, tokenizer, frontend, meta_data=meta_data, **kwargs)+ batch = to_device(output, kwargs["device"])++</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/llm_asr/model.py#L1433" target="_blank" class="src-more">View full source →</a></details></div><div class="mblk"><code>.inference(data_in, data_lengths, key, tokenizer, frontend, **kwargs)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/llm_asr/model.py#L1525" target="_blank" class="gh-link">L1525</a><div class="mdoc"><p>Run inference on input data.</p>+<h4>Args:</h4>+<ul>+<li><code>data_in</code> — Input data (audio samples, file paths, or text).</li>+<li><code>data_lengths</code> — Lengths of each input sample in the batch.</li>+<li><code>key</code> — Sample identifiers.</li>+<li><code>tokenizer</code> — Tokenizer instance for text encoding/decoding.</li>+<li><code>frontend</code> — Audio frontend for feature extraction.</li>+<li><code>**kwargs</code> — Additional keyword arguments.</li>+</ul>+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def inference(+ self,+ data_in,+ data_lengths=None,+ key: list = None,+ tokenizer=None,+ frontend=None,+ **kwargs,+ ):++ """Run inference on input data.++ Args:+ data_in: Input data (audio samples, file paths, or text).+ data_lengths: Lengths of each input sample in the batch.+ key: Sample identifiers.+ tokenizer: Tokenizer instance for text encoding/decoding.+ frontend: Audio frontend for feature extraction.+ **kwargs: Additional keyword arguments.+ """+ inputs_embeds, contents, batch, source_ids, meta_data = self.inference_prepare(+ data_in, data_lengths, key, tokenizer, frontend, **kwargs+ )++ llm_dtype = kwargs.get("llm_dtype", "fp32")+ if llm_dtype == "fp32":+ llm_dtype = "fp16" if kwargs.get("fp16", False) else llm_dtype+ llm_dtype = "bf16" if kwargs.get("bf16", False) else llm_dtype++ with torch.cuda.amp.autocast(+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/llm_asr/model.py#L1525" target="_blank" class="src-more">View full source →</a></details></div></div>+<div class="api-detail" id="e138"><span class="dtag badge-m">method</span><h2><code>LLMASR4.forward(speech, speech_lengths, input_ids, attention_mask, labels_ids, fbank_beg, fbank_mask, **kwargs)</code></h2><div class="dmod">funasr.models.llm_asr.LLMASR4 · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/llm_asr/model.py#L1135" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Encoder + Decoder + Calc loss</p>+<h4>Args:</h4>+<ul>+<li><code>speech</code> — (Batch, Length, ...)</li>+<li><code>speech_lengths</code> — (Batch, )</li>+<li><code>text</code> — (Batch, Length)</li>+<li><code>text_lengths</code> — (Batch,)</li>+</ul>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def forward(+ self,+ speech: torch.Tensor,+ speech_lengths: torch.Tensor,+ input_ids: torch.Tensor,+ attention_mask: torch.Tensor,+ labels_ids: torch.Tensor,+ fbank_beg: torch.Tensor,+ fbank_mask: torch.Tensor,+ **kwargs,+ ) -> Tuple[torch.Tensor, Dict[str, torch.Tensor], torch.Tensor]:+ """Encoder + Decoder + Calc loss+ Args:+ speech: (Batch, Length, ...)+ speech_lengths: (Batch, )+ text: (Batch, Length)+ text_lengths: (Batch,)+ """+ # import pdb+ #+ # pdb.set_trace()+ if len(speech_lengths.size()) > 1:+ speech_lengths = speech_lengths[:, 0]++ batch_size_speech, frames, _ = speech.shape+ batch_size, token_num = input_ids.shape++ with torch.cuda.amp.autocast(enabled=False):+ # audio encoder+ encoder_out, encoder_out_lens = self.encode(speech, speech_lengths)+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/llm_asr/model.py#L1135" target="_blank" class="src-more">View full source on GitHub →</a></details></div>+<div class="api-detail" id="e139"><span class="dtag badge-m">method</span><h2><code>LLMASR4.encode(speech, speech_lengths)</code></h2><div class="dmod">funasr.models.llm_asr.LLMASR4 · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/llm_asr/model.py#L1246" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Encode.</p>+<h4>Args:</h4>+<ul>+<li><code>speech</code> — Speech audio tensor, shape (batch, time).</li>+<li><code>speech_lengths</code> — Length of each speech sample.</li>+</ul>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def encode(self, speech, speech_lengths):+ # audio encoder+ """Encode.++ Args:+ speech: Speech audio tensor, shape (batch, time).+ speech_lengths: Length of each speech sample.+ """+ encoder_out, encoder_out_lens = self.audio_encoder(speech.permute(0, 2, 1), speech_lengths)++ return encoder_out, encoder_out_lens+</code></pre></details></div>+<div class="api-detail" id="e140"><span class="dtag badge-m">method</span><h2><code>LLMASR4.data_template(data)</code></h2><div class="dmod">funasr.models.llm_asr.LLMASR4 · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/llm_asr/model.py#L1258" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Data template.</p>+<h4>Args:</h4>+<ul>+<li><code>data</code> — TODO.</li>+</ul>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def data_template(self, data):+ """Data template.++ Args:+ data: TODO.+ """+ system, user, assistant = [], [], []+ for i, item in enumerate(data):+ role = item["role"]+ content = item["content"]+ if role == "system":+ system.append(content)+ elif role == "user":+ user.append(content)+ elif role == "assistant":+ assistant.append(content)++ system = system * len(user)++ contents = {+ "system": system,+ "user": user,+ "assistant": assistant,+ }++ return contents+</code></pre></details></div>+<div class="api-detail" id="e141"><span class="dtag badge-m">method</span><h2><code>LLMASR4.data_load_speech(contents, tokenizer, frontend, meta_data, **kwargs)</code></h2><div class="dmod">funasr.models.llm_asr.LLMASR4 · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/llm_asr/model.py#L1285" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Data load speech.</p>+<h4>Args:</h4>+<ul>+<li><code>contents</code> — TODO.</li>+<li><code>tokenizer</code> — Tokenizer instance for text encoding/decoding.</li>+<li><code>frontend</code> — Audio frontend for feature extraction.</li>+<li><code>meta_data</code> — TODO.</li>+<li><code>**kwargs</code> — Additional keyword arguments.</li>+</ul>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def data_load_speech(self, contents: dict, tokenizer, frontend, meta_data={}, **kwargs):++ """Data load speech.++ Args:+ contents: TODO.+ tokenizer: Tokenizer instance for text encoding/decoding.+ frontend: Audio frontend for feature extraction.+ meta_data: TODO.+ **kwargs: Additional keyword arguments.+ """+ system = contents["system"]+ user = contents["user"]+ assistant = contents["assistant"]+ pattern = re.compile(r"(<\|startofspeech\|>.*?<\|endofspeech\|>)")++ input_ids, labels, fbank, fbank_lens, fbank_mask, fbank_beg, fake_token_len = (+ [],+ [],+ [],+ [],+ [],+ [],+ [],+ )+ input_source_ids = []+ for i, (system_prompt, user_prompt, target_out) in enumerate(zip(system, user, assistant)):+ if i >= kwargs.get("multiturn_num_max", 5):+ break+ if len(input_ids) > kwargs.get("max_token_length", 1500):+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/llm_asr/model.py#L1285" target="_blank" class="src-more">View full source on GitHub →</a></details></div>+<div class="api-detail" id="e142"><span class="dtag badge-m">method</span><h2><code>LLMASR4.inference_prepare(data_in, data_lengths, key, tokenizer, frontend, **kwargs)</code></h2><div class="dmod">funasr.models.llm_asr.LLMASR4 · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/llm_asr/model.py#L1433" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Inference prepare.</p>+<h4>Args:</h4>+<ul>+<li><code>data_in</code> — Input data (audio samples, file paths, or text).</li>+<li><code>data_lengths</code> — Lengths of each input sample in the batch.</li>+<li><code>key</code> — Sample identifiers.</li>+<li><code>tokenizer</code> — Tokenizer instance for text encoding/decoding.</li>+<li><code>frontend</code> — Audio frontend for feature extraction.</li>+<li><code>**kwargs</code> — Additional keyword arguments.</li>+</ul>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def inference_prepare(+ self,+ data_in,+ data_lengths=None,+ key: list = None,+ tokenizer=None,+ frontend=None,+ **kwargs,+ ):++ """Inference prepare.++ Args:+ data_in: Input data (audio samples, file paths, or text).+ data_lengths: Lengths of each input sample in the batch.+ key: Sample identifiers.+ tokenizer: Tokenizer instance for text encoding/decoding.+ frontend: Audio frontend for feature extraction.+ **kwargs: Additional keyword arguments.+ """+ meta_data = {}+ prompt = kwargs.get("prompt", None)++ if kwargs.get("batch_size", 1) > 1:+ raise NotImplementedError("batch decoding is not implemented")++ contents = self.data_template(data_in[0])+ output = self.data_load_speech(contents, tokenizer, frontend, meta_data=meta_data, **kwargs)+ batch = to_device(output, kwargs["device"])++</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/llm_asr/model.py#L1433" target="_blank" class="src-more">View full source on GitHub →</a></details></div>+<div class="api-detail" id="e143"><span class="dtag badge-m">method</span><h2><code>LLMASR4.inference(data_in, data_lengths, key, tokenizer, frontend, **kwargs)</code></h2><div class="dmod">funasr.models.llm_asr.LLMASR4 · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/llm_asr/model.py#L1525" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Run inference on input data.</p>+<h4>Args:</h4>+<ul>+<li><code>data_in</code> — Input data (audio samples, file paths, or text).</li>+<li><code>data_lengths</code> — Lengths of each input sample in the batch.</li>+<li><code>key</code> — Sample identifiers.</li>+<li><code>tokenizer</code> — Tokenizer instance for text encoding/decoding.</li>+<li><code>frontend</code> — Audio frontend for feature extraction.</li>+<li><code>**kwargs</code> — Additional keyword arguments.</li>+</ul>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def inference(+ self,+ data_in,+ data_lengths=None,+ key: list = None,+ tokenizer=None,+ frontend=None,+ **kwargs,+ ):++ """Run inference on input data.++ Args:+ data_in: Input data (audio samples, file paths, or text).+ data_lengths: Lengths of each input sample in the batch.+ key: Sample identifiers.+ tokenizer: Tokenizer instance for text encoding/decoding.+ frontend: Audio frontend for feature extraction.+ **kwargs: Additional keyword arguments.+ """+ inputs_embeds, contents, batch, source_ids, meta_data = self.inference_prepare(+ data_in, data_lengths, key, tokenizer, frontend, **kwargs+ )++ llm_dtype = kwargs.get("llm_dtype", "fp32")+ if llm_dtype == "fp32":+ llm_dtype = "fp16" if kwargs.get("fp16", False) else llm_dtype+ llm_dtype = "bf16" if kwargs.get("bf16", False) else llm_dtype++ with torch.cuda.amp.autocast(+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/llm_asr/model.py#L1525" target="_blank" class="src-more">View full source on GitHub →</a></details></div>+<div class="api-detail" id="e144"><span class="dtag badge-c">class</span><h2>LLMASRNAR</h2><div class="dmod">funasr.models.llm_asr_nar · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/llm_asr_nar/model.py#L26" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p class="muted">No documentation yet.</p></div><details class="src-block"><summary>📄 Source code</summary><pre><code>class LLMASRNAR(nn.Module):+ """ """++ def __init__(+ self,+ specaug: str = None,+ specaug_conf: dict = None,+ normalize: str = None,+ normalize_conf: dict = None,+ encoder: str = None,+ encoder_conf: dict = None,+ decoder: str = None,+ decoder_conf: dict = None,+ ctc: str = None,+ ctc_conf: dict = None,+ ctc_weight: float = 0.5,+ llm: str = None,+ llm_conf: dict = None,+ adaptor: str = None,+ adaptor_conf: dict = None,+ input_size: int = 80,+ vocab_size: int = -1,+ ignore_id: int = -1,+ blank_id: int = 0,+ sos: int = 1,+ eos: int = 2,+ lsm_weight: float = 0.0,+ length_normalized_loss: bool = False,+ report_cer: bool = True,+ report_wer: bool = True,+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/llm_asr_nar/model.py#L26" target="_blank" class="src-more">View full source on GitHub →</a></details><h3 class="mt">Methods</h3><div class="mblk"><code>.forward(speech, speech_lengths, text, text_lengths, input_ids, attention_mask, labels_ids, label_mask, audio_mask, **kwargs)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/llm_asr_nar/model.py#L182" target="_blank" class="gh-link">L182</a><div class="mdoc"><p>Encoder + Decoder + Calc loss</p>+<h4>Args:</h4>+<ul>+<li><code>speech</code> — (Batch, Length, ...)</li>+<li><code>speech_lengths</code> — (Batch, )</li>+<li><code>text</code> — (Batch, Length)</li>+<li><code>text_lengths</code> — (Batch,)</li>+</ul>+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def forward(+ self,+ speech: torch.Tensor,+ speech_lengths: torch.Tensor,+ text: torch.Tensor,+ text_lengths: torch.Tensor,+ input_ids: torch.Tensor,+ attention_mask: torch.Tensor,+ labels_ids: torch.Tensor,+ label_mask: torch.Tensor,+ audio_mask: torch.Tensor,+ **kwargs,+ ) -> Tuple[torch.Tensor, Dict[str, torch.Tensor], torch.Tensor]:+ """Encoder + Decoder + Calc loss+ Args:+ speech: (Batch, Length, ...)+ speech_lengths: (Batch, )+ text: (Batch, Length)+ text_lengths: (Batch,)+ """+ if len(text_lengths.size()) > 1:+ text_lengths = text_lengths[:, 0]+ if len(speech_lengths.size()) > 1:+ speech_lengths = speech_lengths[:, 0]++ batch_size = speech.shape[0]++ # audio encoder+ encoder_out, encoder_out_lens = self.encode(speech, speech_lengths, audio_mask=audio_mask)++</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/llm_asr_nar/model.py#L182" target="_blank" class="src-more">View full source →</a></details></div><div class="mblk"><code>.encode(speech, speech_lengths, **kwargs)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/llm_asr_nar/model.py#L253" target="_blank" class="gh-link">L253</a><div class="mdoc"><p>Encode.</p>+<h4>Args:</h4>+<ul>+<li><code>speech</code> — Speech audio tensor, shape (batch, time).</li>+<li><code>speech_lengths</code> — Length of each speech sample.</li>+<li><code>**kwargs</code> — Additional keyword arguments.</li>+</ul>+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def encode(+ self,+ speech: torch.Tensor,+ speech_lengths: torch.Tensor,+ **kwargs,+ ):++ """Encode.++ Args:+ speech: Speech audio tensor, shape (batch, time).+ speech_lengths: Length of each speech sample.+ **kwargs: Additional keyword arguments.+ """+ audio_mask = kwargs.get("audio_mask", None)+ audio_token_lengths = audio_mask.sum(-1) if audio_mask is not None else None+ text_token_int = kwargs.get("text_token_int", None)+ if audio_token_lengths is None:+ audio_token_lengths = torch.tensor([len(text_token_int)], dtype=torch.int64)++ batch = {"speech": speech, "speech_lengths": speech_lengths}+ enc, enc_lens = self.audio_encoder.encode(**batch)+ with autocast(False):+ enc_mask = sequence_mask(enc_lens, enc.size(1), device=enc.device)[:, None, :]+ pre_acoustic_embeds, pre_token_length, _, _ = self.audio_encoder.predictor(+ enc,+ mask=enc_mask,+ target_label_length=audio_token_lengths,+ )++</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/llm_asr_nar/model.py#L253" target="_blank" class="src-more">View full source →</a></details></div><div class="mblk"><code>.inference(data_in, data_lengths, key, tokenizer, frontend, **kwargs)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/llm_asr_nar/model.py#L285" target="_blank" class="gh-link">L285</a><div class="mdoc"><p>Run inference on input data.</p>+<h4>Args:</h4>+<ul>+<li><code>data_in</code> — Input data (audio samples, file paths, or text).</li>+<li><code>data_lengths</code> — Lengths of each input sample in the batch.</li>+<li><code>key</code> — Sample identifiers.</li>+<li><code>tokenizer</code> — Tokenizer instance for text encoding/decoding.</li>+<li><code>frontend</code> — Audio frontend for feature extraction.</li>+<li><code>**kwargs</code> — Additional keyword arguments.</li>+</ul>+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def inference(+ self,+ data_in,+ data_lengths=None,+ key: list = None,+ tokenizer=None,+ frontend=None,+ **kwargs,+ ):++ """Run inference on input data.++ Args:+ data_in: Input data (audio samples, file paths, or text).+ data_lengths: Lengths of each input sample in the batch.+ key: Sample identifiers.+ tokenizer: Tokenizer instance for text encoding/decoding.+ frontend: Audio frontend for feature extraction.+ **kwargs: Additional keyword arguments.+ """+ prompt = kwargs.get("prompt", "Transcribe speech to text.")++ if kwargs.get("batch_size", 1) > 1:+ raise NotImplementedError("batch decoding is not implemented")++ meta_data = {}+ if (+ isinstance(data_in, torch.Tensor) and kwargs.get("data_type", "sound") == "fbank"+ ): # fbank+ speech, speech_lengths = data_in, data_lengths+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/llm_asr_nar/model.py#L285" target="_blank" class="src-more">View full source →</a></details></div></div>+<div class="api-detail" id="e145"><span class="dtag badge-m">method</span><h2><code>LLMASRNAR.forward(speech, speech_lengths, text, text_lengths, input_ids, attention_mask, labels_ids, label_mask, audio_mask, **kwargs)</code></h2><div class="dmod">funasr.models.llm_asr_nar.LLMASRNAR · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/llm_asr_nar/model.py#L182" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Encoder + Decoder + Calc loss</p>+<h4>Args:</h4>+<ul>+<li><code>speech</code> — (Batch, Length, ...)</li>+<li><code>speech_lengths</code> — (Batch, )</li>+<li><code>text</code> — (Batch, Length)</li>+<li><code>text_lengths</code> — (Batch,)</li>+</ul>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def forward(+ self,+ speech: torch.Tensor,+ speech_lengths: torch.Tensor,+ text: torch.Tensor,+ text_lengths: torch.Tensor,+ input_ids: torch.Tensor,+ attention_mask: torch.Tensor,+ labels_ids: torch.Tensor,+ label_mask: torch.Tensor,+ audio_mask: torch.Tensor,+ **kwargs,+ ) -> Tuple[torch.Tensor, Dict[str, torch.Tensor], torch.Tensor]:+ """Encoder + Decoder + Calc loss+ Args:+ speech: (Batch, Length, ...)+ speech_lengths: (Batch, )+ text: (Batch, Length)+ text_lengths: (Batch,)+ """+ if len(text_lengths.size()) > 1:+ text_lengths = text_lengths[:, 0]+ if len(speech_lengths.size()) > 1:+ speech_lengths = speech_lengths[:, 0]++ batch_size = speech.shape[0]++ # audio encoder+ encoder_out, encoder_out_lens = self.encode(speech, speech_lengths, audio_mask=audio_mask)++</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/llm_asr_nar/model.py#L182" target="_blank" class="src-more">View full source on GitHub →</a></details></div>+<div class="api-detail" id="e146"><span class="dtag badge-m">method</span><h2><code>LLMASRNAR.encode(speech, speech_lengths, **kwargs)</code></h2><div class="dmod">funasr.models.llm_asr_nar.LLMASRNAR · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/llm_asr_nar/model.py#L253" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Encode.</p>+<h4>Args:</h4>+<ul>+<li><code>speech</code> — Speech audio tensor, shape (batch, time).</li>+<li><code>speech_lengths</code> — Length of each speech sample.</li>+<li><code>**kwargs</code> — Additional keyword arguments.</li>+</ul>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def encode(+ self,+ speech: torch.Tensor,+ speech_lengths: torch.Tensor,+ **kwargs,+ ):++ """Encode.++ Args:+ speech: Speech audio tensor, shape (batch, time).+ speech_lengths: Length of each speech sample.+ **kwargs: Additional keyword arguments.+ """+ audio_mask = kwargs.get("audio_mask", None)+ audio_token_lengths = audio_mask.sum(-1) if audio_mask is not None else None+ text_token_int = kwargs.get("text_token_int", None)+ if audio_token_lengths is None:+ audio_token_lengths = torch.tensor([len(text_token_int)], dtype=torch.int64)++ batch = {"speech": speech, "speech_lengths": speech_lengths}+ enc, enc_lens = self.audio_encoder.encode(**batch)+ with autocast(False):+ enc_mask = sequence_mask(enc_lens, enc.size(1), device=enc.device)[:, None, :]+ pre_acoustic_embeds, pre_token_length, _, _ = self.audio_encoder.predictor(+ enc,+ mask=enc_mask,+ target_label_length=audio_token_lengths,+ )++</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/llm_asr_nar/model.py#L253" target="_blank" class="src-more">View full source on GitHub →</a></details></div>+<div class="api-detail" id="e147"><span class="dtag badge-m">method</span><h2><code>LLMASRNAR.inference(data_in, data_lengths, key, tokenizer, frontend, **kwargs)</code></h2><div class="dmod">funasr.models.llm_asr_nar.LLMASRNAR · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/llm_asr_nar/model.py#L285" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Run inference on input data.</p>+<h4>Args:</h4>+<ul>+<li><code>data_in</code> — Input data (audio samples, file paths, or text).</li>+<li><code>data_lengths</code> — Lengths of each input sample in the batch.</li>+<li><code>key</code> — Sample identifiers.</li>+<li><code>tokenizer</code> — Tokenizer instance for text encoding/decoding.</li>+<li><code>frontend</code> — Audio frontend for feature extraction.</li>+<li><code>**kwargs</code> — Additional keyword arguments.</li>+</ul>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def inference(+ self,+ data_in,+ data_lengths=None,+ key: list = None,+ tokenizer=None,+ frontend=None,+ **kwargs,+ ):++ """Run inference on input data.++ Args:+ data_in: Input data (audio samples, file paths, or text).+ data_lengths: Lengths of each input sample in the batch.+ key: Sample identifiers.+ tokenizer: Tokenizer instance for text encoding/decoding.+ frontend: Audio frontend for feature extraction.+ **kwargs: Additional keyword arguments.+ """+ prompt = kwargs.get("prompt", "Transcribe speech to text.")++ if kwargs.get("batch_size", 1) > 1:+ raise NotImplementedError("batch decoding is not implemented")++ meta_data = {}+ if (+ isinstance(data_in, torch.Tensor) and kwargs.get("data_type", "sound") == "fbank"+ ): # fbank+ speech, speech_lengths = data_in, data_lengths+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/llm_asr_nar/model.py#L285" target="_blank" class="src-more">View full source on GitHub →</a></details></div>+<div class="api-detail" id="e148"><span class="dtag badge-c">class</span><h2>LLMASRNARPrompt</h2><div class="dmod">funasr.models.llm_asr_nar · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/llm_asr_nar/model.py#L421" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p class="muted">No documentation yet.</p></div><details class="src-block"><summary>📄 Source code</summary><pre><code>class LLMASRNARPrompt(nn.Module):+ """ """++ def __init__(+ self,+ specaug: str = None,+ specaug_conf: dict = None,+ normalize: str = None,+ normalize_conf: dict = None,+ encoder: str = None,+ encoder_conf: dict = None,+ decoder: str = None,+ decoder_conf: dict = None,+ ctc: str = None,+ ctc_conf: dict = None,+ ctc_weight: float = 0.0,+ llm: str = None,+ llm_conf: dict = None,+ adaptor: str = None,+ adaptor_conf: dict = None,+ input_size: int = 80,+ vocab_size: int = -1,+ ignore_id: int = -1,+ blank_id: int = 0,+ sos: int = 1,+ eos: int = 2,+ lsm_weight: float = 0.0,+ length_normalized_loss: bool = False,+ predictor_weight: int = 1.0,+ report_cer: bool = True,+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/llm_asr_nar/model.py#L421" target="_blank" class="src-more">View full source on GitHub →</a></details><h3 class="mt">Methods</h3><div class="mblk"><code>.forward(speech, speech_lengths, text, text_lengths, input_ids, attention_mask, labels_ids, label_mask, audio_mask, **kwargs)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/llm_asr_nar/model.py#L588" target="_blank" class="gh-link">L588</a><div class="mdoc"><p>Encoder + Decoder + Calc loss</p>+<h4>Args:</h4>+<ul>+<li><code>speech</code> — (Batch, Length, ...)</li>+<li><code>speech_lengths</code> — (Batch, )</li>+<li><code>text</code> — (Batch, Length)</li>+<li><code>text_lengths</code> — (Batch,)</li>+</ul>+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def forward(+ self,+ speech: torch.Tensor,+ speech_lengths: torch.Tensor,+ text: torch.Tensor,+ text_lengths: torch.Tensor,+ input_ids: torch.Tensor,+ attention_mask: torch.Tensor,+ labels_ids: torch.Tensor,+ label_mask: torch.Tensor,+ audio_mask: torch.Tensor,+ **kwargs,+ ) -> Tuple[torch.Tensor, Dict[str, torch.Tensor], torch.Tensor]:+ """Encoder + Decoder + Calc loss+ Args:+ speech: (Batch, Length, ...)+ speech_lengths: (Batch, )+ text: (Batch, Length)+ text_lengths: (Batch,)+ """+ # import pdb;+ # pdb.set_trace()+ if len(text_lengths.size()) > 1:+ text_lengths = text_lengths[:, 0]+ if len(speech_lengths.size()) > 1:+ speech_lengths = speech_lengths[:, 0]++ batch_size = speech.shape[0]++ stats = {}+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/llm_asr_nar/model.py#L588" target="_blank" class="src-more">View full source →</a></details></div><div class="mblk"><code>.encode(speech, speech_lengths, **kwargs)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/llm_asr_nar/model.py#L691" target="_blank" class="gh-link">L691</a><div class="mdoc"><p>Encode.</p>+<h4>Args:</h4>+<ul>+<li><code>speech</code> — Speech audio tensor, shape (batch, time).</li>+<li><code>speech_lengths</code> — Length of each speech sample.</li>+<li><code>**kwargs</code> — Additional keyword arguments.</li>+</ul>+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def encode(+ self,+ speech: torch.Tensor,+ speech_lengths: torch.Tensor,+ **kwargs,+ ):++ """Encode.++ Args:+ speech: Speech audio tensor, shape (batch, time).+ speech_lengths: Length of each speech sample.+ **kwargs: Additional keyword arguments.+ """+ audio_mask = kwargs.get("audio_mask", None)+ audio_token_lengths = audio_mask.sum(-1) if audio_mask is not None else None+ text_token_int = kwargs.get("text_token_int", None)+ if audio_token_lengths is None and text_token_int is not None:+ audio_token_lengths = torch.tensor([len(text_token_int)], dtype=torch.int64)++ batch = {"speech": speech, "speech_lengths": speech_lengths}+ enc, enc_lens = self.audio_encoder.encode(**batch)+ with autocast(False):+ enc_mask = sequence_mask(enc_lens, enc.size(1), device=enc.device)[:, None, :]+ pre_acoustic_embeds, pre_token_length, _, _ = self.audio_encoder.predictor(+ enc,+ mask=enc_mask,+ target_label_length=audio_token_lengths,+ )+ loss_pre = 0.0+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/llm_asr_nar/model.py#L691" target="_blank" class="src-more">View full source →</a></details></div><div class="mblk"><code>.inference(data_in, data_lengths, key, tokenizer, frontend, **kwargs)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/llm_asr_nar/model.py#L753" target="_blank" class="gh-link">L753</a><div class="mdoc"><p>Run inference on input data.</p>+<h4>Args:</h4>+<ul>+<li><code>data_in</code> — Input data (audio samples, file paths, or text).</li>+<li><code>data_lengths</code> — Lengths of each input sample in the batch.</li>+<li><code>key</code> — Sample identifiers.</li>+<li><code>tokenizer</code> — Tokenizer instance for text encoding/decoding.</li>+<li><code>frontend</code> — Audio frontend for feature extraction.</li>+<li><code>**kwargs</code> — Additional keyword arguments.</li>+</ul>+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def inference(+ self,+ data_in,+ data_lengths=None,+ key: list = None,+ tokenizer=None,+ frontend=None,+ **kwargs,+ ):++ """Run inference on input data.++ Args:+ data_in: Input data (audio samples, file paths, or text).+ data_lengths: Lengths of each input sample in the batch.+ key: Sample identifiers.+ tokenizer: Tokenizer instance for text encoding/decoding.+ frontend: Audio frontend for feature extraction.+ **kwargs: Additional keyword arguments.+ """+ prompt = kwargs.get("prompt", "Transcribe speech to text.")++ if kwargs.get("batch_size", 1) > 1:+ raise NotImplementedError("batch decoding is not implemented")++ meta_data = {}+ if (+ isinstance(data_in, torch.Tensor) and kwargs.get("data_type", "sound") == "fbank"+ ): # fbank+ speech, speech_lengths = data_in, data_lengths+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/llm_asr_nar/model.py#L753" target="_blank" class="src-more">View full source →</a></details></div></div>+<div class="api-detail" id="e149"><span class="dtag badge-m">method</span><h2><code>LLMASRNARPrompt.forward(speech, speech_lengths, text, text_lengths, input_ids, attention_mask, labels_ids, label_mask, audio_mask, **kwargs)</code></h2><div class="dmod">funasr.models.llm_asr_nar.LLMASRNARPrompt · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/llm_asr_nar/model.py#L588" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Encoder + Decoder + Calc loss</p>+<h4>Args:</h4>+<ul>+<li><code>speech</code> — (Batch, Length, ...)</li>+<li><code>speech_lengths</code> — (Batch, )</li>+<li><code>text</code> — (Batch, Length)</li>+<li><code>text_lengths</code> — (Batch,)</li>+</ul>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def forward(+ self,+ speech: torch.Tensor,+ speech_lengths: torch.Tensor,+ text: torch.Tensor,+ text_lengths: torch.Tensor,+ input_ids: torch.Tensor,+ attention_mask: torch.Tensor,+ labels_ids: torch.Tensor,+ label_mask: torch.Tensor,+ audio_mask: torch.Tensor,+ **kwargs,+ ) -> Tuple[torch.Tensor, Dict[str, torch.Tensor], torch.Tensor]:+ """Encoder + Decoder + Calc loss+ Args:+ speech: (Batch, Length, ...)+ speech_lengths: (Batch, )+ text: (Batch, Length)+ text_lengths: (Batch,)+ """+ # import pdb;+ # pdb.set_trace()+ if len(text_lengths.size()) > 1:+ text_lengths = text_lengths[:, 0]+ if len(speech_lengths.size()) > 1:+ speech_lengths = speech_lengths[:, 0]++ batch_size = speech.shape[0]++ stats = {}+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/llm_asr_nar/model.py#L588" target="_blank" class="src-more">View full source on GitHub →</a></details></div>+<div class="api-detail" id="e150"><span class="dtag badge-m">method</span><h2><code>LLMASRNARPrompt.encode(speech, speech_lengths, **kwargs)</code></h2><div class="dmod">funasr.models.llm_asr_nar.LLMASRNARPrompt · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/llm_asr_nar/model.py#L691" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Encode.</p>+<h4>Args:</h4>+<ul>+<li><code>speech</code> — Speech audio tensor, shape (batch, time).</li>+<li><code>speech_lengths</code> — Length of each speech sample.</li>+<li><code>**kwargs</code> — Additional keyword arguments.</li>+</ul>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def encode(+ self,+ speech: torch.Tensor,+ speech_lengths: torch.Tensor,+ **kwargs,+ ):++ """Encode.++ Args:+ speech: Speech audio tensor, shape (batch, time).+ speech_lengths: Length of each speech sample.+ **kwargs: Additional keyword arguments.+ """+ audio_mask = kwargs.get("audio_mask", None)+ audio_token_lengths = audio_mask.sum(-1) if audio_mask is not None else None+ text_token_int = kwargs.get("text_token_int", None)+ if audio_token_lengths is None and text_token_int is not None:+ audio_token_lengths = torch.tensor([len(text_token_int)], dtype=torch.int64)++ batch = {"speech": speech, "speech_lengths": speech_lengths}+ enc, enc_lens = self.audio_encoder.encode(**batch)+ with autocast(False):+ enc_mask = sequence_mask(enc_lens, enc.size(1), device=enc.device)[:, None, :]+ pre_acoustic_embeds, pre_token_length, _, _ = self.audio_encoder.predictor(+ enc,+ mask=enc_mask,+ target_label_length=audio_token_lengths,+ )+ loss_pre = 0.0+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/llm_asr_nar/model.py#L691" target="_blank" class="src-more">View full source on GitHub →</a></details></div>+<div class="api-detail" id="e151"><span class="dtag badge-m">method</span><h2><code>LLMASRNARPrompt.inference(data_in, data_lengths, key, tokenizer, frontend, **kwargs)</code></h2><div class="dmod">funasr.models.llm_asr_nar.LLMASRNARPrompt · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/llm_asr_nar/model.py#L753" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Run inference on input data.</p>+<h4>Args:</h4>+<ul>+<li><code>data_in</code> — Input data (audio samples, file paths, or text).</li>+<li><code>data_lengths</code> — Lengths of each input sample in the batch.</li>+<li><code>key</code> — Sample identifiers.</li>+<li><code>tokenizer</code> — Tokenizer instance for text encoding/decoding.</li>+<li><code>frontend</code> — Audio frontend for feature extraction.</li>+<li><code>**kwargs</code> — Additional keyword arguments.</li>+</ul>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def inference(+ self,+ data_in,+ data_lengths=None,+ key: list = None,+ tokenizer=None,+ frontend=None,+ **kwargs,+ ):++ """Run inference on input data.++ Args:+ data_in: Input data (audio samples, file paths, or text).+ data_lengths: Lengths of each input sample in the batch.+ key: Sample identifiers.+ tokenizer: Tokenizer instance for text encoding/decoding.+ frontend: Audio frontend for feature extraction.+ **kwargs: Additional keyword arguments.+ """+ prompt = kwargs.get("prompt", "Transcribe speech to text.")++ if kwargs.get("batch_size", 1) > 1:+ raise NotImplementedError("batch decoding is not implemented")++ meta_data = {}+ if (+ isinstance(data_in, torch.Tensor) and kwargs.get("data_type", "sound") == "fbank"+ ): # fbank+ speech, speech_lengths = data_in, data_lengths+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/llm_asr_nar/model.py#L753" target="_blank" class="src-more">View full source on GitHub →</a></details></div>+<div class="api-detail" id="e152"><span class="dtag badge-c">class</span><h2>MonotonicAligner</h2><div class="dmod">funasr.models.monotonic_aligner · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/monotonic_aligner/model.py#L25" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><ul>+<li><code>MonotonicAligner</code> — Forced alignment model for timestamp prediction.</li>+</ul>+<p>Given audio and text, computes character-level time alignments</p>+<p>using monotonic attention constraint.</p>+<ul>+<li><code>Usage</code> — model.generate(input=(audio_path, text_path), data_type=("sound", "text"))</li>+<li><code>Output</code> — {"key": str, "timestamp": [[start_ms, end_ms], ...], "text": str}</li>+</ul>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code>class MonotonicAligner(torch.nn.Module):+ """MonotonicAligner: Forced alignment model for timestamp prediction.++ Given audio and text, computes character-level time alignments+ using monotonic attention constraint.++ Usage: model.generate(input=(audio_path, text_path), data_type=("sound", "text"))+ Output: {"key": str, "timestamp": [[start_ms, end_ms], ...], "text": str}+ """++ def __init__(+ self,+ input_size: int = 80,+ specaug: Optional[str] = None,+ specaug_conf: Optional[Dict] = None,+ normalize: str = None,+ normalize_conf: Optional[Dict] = None,+ encoder: str = None,+ encoder_conf: Optional[Dict] = None,+ predictor: str = None,+ predictor_conf: Optional[Dict] = None,+ predictor_bias: int = 0,+ length_normalized_loss: bool = False,+ **kwargs,+ ):+ """Initialize MonotonicAligner.++ Args:+ input_size: Size/dimension parameter.+ specaug: TODO.+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/monotonic_aligner/model.py#L25" target="_blank" class="src-more">View full source on GitHub →</a></details><h3 class="mt">Methods</h3><div class="mblk"><code>.forward(speech, speech_lengths, text, text_lengths)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/monotonic_aligner/model.py#L86" target="_blank" class="gh-link">L86</a><div class="mdoc"><p>Frontend + Encoder + Decoder + Calc loss</p>+<h4>Args:</h4>+<ul>+<li><code>speech</code> — (Batch, Length, ...)</li>+<li><code>speech_lengths</code> — (Batch, )</li>+<li><code>text</code> — (Batch, Length)</li>+<li><code>text_lengths</code> — (Batch,)</li>+</ul>+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def forward(+ self,+ speech: torch.Tensor,+ speech_lengths: torch.Tensor,+ text: torch.Tensor,+ text_lengths: torch.Tensor,+ ) -> Tuple[torch.Tensor, Dict[str, torch.Tensor], torch.Tensor]:+ """Frontend + Encoder + Decoder + Calc loss+ Args:+ speech: (Batch, Length, ...)+ speech_lengths: (Batch, )+ text: (Batch, Length)+ text_lengths: (Batch,)+ """+ assert text_lengths.dim() == 1, text_lengths.shape+ # Check that batch_size is unified+ assert (+ speech.shape[0] == speech_lengths.shape[0] == text.shape[0] == text_lengths.shape[0]+ ), (speech.shape, speech_lengths.shape, text.shape, text_lengths.shape)+ batch_size = speech.shape[0]+ # for data-parallel+ text = text[:, : text_lengths.max()]+ speech = speech[:, : speech_lengths.max()]++ # 1. Encoder+ encoder_out, encoder_out_lens = self.encode(speech, speech_lengths)++ encoder_out_mask = (+ ~make_pad_mask(encoder_out_lens, maxlen=encoder_out.size(1))[:, None, :]+ ).to(encoder_out.device)+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/monotonic_aligner/model.py#L86" target="_blank" class="src-more">View full source →</a></details></div><div class="mblk"><code>.calc_predictor_timestamp(encoder_out, encoder_out_lens, token_num)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/monotonic_aligner/model.py#L137" target="_blank" class="gh-link">L137</a><div class="mdoc"><p>Calc predictor timestamp.</p>+<h4>Args:</h4>+<ul>+<li><code>encoder_out</code> — Encoder output tensor.</li>+<li><code>encoder_out_lens</code> — Encoder output lengths.</li>+<li><code>token_num</code> — TODO.</li>+</ul>+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def calc_predictor_timestamp(self, encoder_out, encoder_out_lens, token_num):+ """Calc predictor timestamp.++ Args:+ encoder_out: Encoder output tensor.+ encoder_out_lens: Encoder output lengths.+ token_num: TODO.+ """+ encoder_out_mask = (+ ~make_pad_mask(encoder_out_lens, maxlen=encoder_out.size(1))[:, None, :]+ ).to(encoder_out.device)+ ds_alphas, ds_cif_peak, us_alphas, us_peaks = self.predictor.get_upsample_timestamp(+ encoder_out, encoder_out_mask, token_num+ )+ return ds_alphas, ds_cif_peak, us_alphas, us_peaks+</code></pre></details></div><div class="mblk"><code>.encode(speech, speech_lengths, **kwargs)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/monotonic_aligner/model.py#L153" target="_blank" class="gh-link">L153</a><div class="mdoc"><p>Encoder. Note that this method is used by asr_inference.py</p>+<h4>Args:</h4>+<ul>+<li><code>speech</code> — (Batch, Length, ...)</li>+<li><code>speech_lengths</code> — (Batch, )</li>+<li><code>ind</code> — int</li>+</ul>+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def encode(+ self,+ speech: torch.Tensor,+ speech_lengths: torch.Tensor,+ **kwargs,+ ) -> Tuple[torch.Tensor, torch.Tensor]:+ """Encoder. Note that this method is used by asr_inference.py+ Args:+ speech: (Batch, Length, ...)+ speech_lengths: (Batch, )+ ind: int+ """+ with autocast(False):++ # Data augmentation+ if self.specaug is not None and self.training:+ speech, speech_lengths = self.specaug(speech, speech_lengths)++ # Normalization for feature: e.g. Global-CMVN, Utterance-CMVN+ if self.normalize is not None:+ speech, speech_lengths = self.normalize(speech, speech_lengths)++ # Forward encoder+ encoder_out, encoder_out_lens, _ = self.encoder(speech, speech_lengths)+ if isinstance(encoder_out, tuple):+ encoder_out = encoder_out[0]++ return encoder_out, encoder_out_lens+</code></pre></details></div><div class="mblk"><code>.inference(data_in, data_lengths, key, tokenizer, frontend, **kwargs)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/monotonic_aligner/model.py#L182" target="_blank" class="gh-link">L182</a><div class="mdoc"><p>Run inference on input data.</p>+<h4>Args:</h4>+<ul>+<li><code>data_in</code> — Input data (audio samples, file paths, or text).</li>+<li><code>data_lengths</code> — Lengths of each input sample in the batch.</li>+<li><code>key</code> — Sample identifiers.</li>+<li><code>tokenizer</code> — Tokenizer instance for text encoding/decoding.</li>+<li><code>frontend</code> — Audio frontend for feature extraction.</li>+<li><code>**kwargs</code> — Additional keyword arguments.</li>+</ul>+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def inference(+ self,+ data_in,+ data_lengths=None,+ key: list = None,+ tokenizer=None,+ frontend=None,+ **kwargs,+ ):+ """Run inference on input data.++ Args:+ data_in: Input data (audio samples, file paths, or text).+ data_lengths: Lengths of each input sample in the batch.+ key: Sample identifiers.+ tokenizer: Tokenizer instance for text encoding/decoding.+ frontend: Audio frontend for feature extraction.+ **kwargs: Additional keyword arguments.+ """+ meta_data = {}+ # extract fbank feats+ time1 = time.perf_counter()+ audio_list, text_token_int_list = load_audio_text_image_video(+ data_in,+ fs=frontend.fs,+ audio_fs=kwargs.get("fs", 16000),+ data_type=kwargs.get("data_type", "sound"),+ tokenizer=tokenizer,+ )+ time2 = time.perf_counter()+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/monotonic_aligner/model.py#L182" target="_blank" class="src-more">View full source →</a></details></div></div>+<div class="api-detail" id="e153"><span class="dtag badge-m">method</span><h2><code>MonotonicAligner.forward(speech, speech_lengths, text, text_lengths)</code></h2><div class="dmod">funasr.models.monotonic_aligner.MonotonicAligner · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/monotonic_aligner/model.py#L86" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Frontend + Encoder + Decoder + Calc loss</p>+<h4>Args:</h4>+<ul>+<li><code>speech</code> — (Batch, Length, ...)</li>+<li><code>speech_lengths</code> — (Batch, )</li>+<li><code>text</code> — (Batch, Length)</li>+<li><code>text_lengths</code> — (Batch,)</li>+</ul>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def forward(+ self,+ speech: torch.Tensor,+ speech_lengths: torch.Tensor,+ text: torch.Tensor,+ text_lengths: torch.Tensor,+ ) -> Tuple[torch.Tensor, Dict[str, torch.Tensor], torch.Tensor]:+ """Frontend + Encoder + Decoder + Calc loss+ Args:+ speech: (Batch, Length, ...)+ speech_lengths: (Batch, )+ text: (Batch, Length)+ text_lengths: (Batch,)+ """+ assert text_lengths.dim() == 1, text_lengths.shape+ # Check that batch_size is unified+ assert (+ speech.shape[0] == speech_lengths.shape[0] == text.shape[0] == text_lengths.shape[0]+ ), (speech.shape, speech_lengths.shape, text.shape, text_lengths.shape)+ batch_size = speech.shape[0]+ # for data-parallel+ text = text[:, : text_lengths.max()]+ speech = speech[:, : speech_lengths.max()]++ # 1. Encoder+ encoder_out, encoder_out_lens = self.encode(speech, speech_lengths)++ encoder_out_mask = (+ ~make_pad_mask(encoder_out_lens, maxlen=encoder_out.size(1))[:, None, :]+ ).to(encoder_out.device)+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/monotonic_aligner/model.py#L86" target="_blank" class="src-more">View full source on GitHub →</a></details></div>+<div class="api-detail" id="e154"><span class="dtag badge-m">method</span><h2><code>MonotonicAligner.calc_predictor_timestamp(encoder_out, encoder_out_lens, token_num)</code></h2><div class="dmod">funasr.models.monotonic_aligner.MonotonicAligner · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/monotonic_aligner/model.py#L137" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Calc predictor timestamp.</p>+<h4>Args:</h4>+<ul>+<li><code>encoder_out</code> — Encoder output tensor.</li>+<li><code>encoder_out_lens</code> — Encoder output lengths.</li>+<li><code>token_num</code> — TODO.</li>+</ul>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def calc_predictor_timestamp(self, encoder_out, encoder_out_lens, token_num):+ """Calc predictor timestamp.++ Args:+ encoder_out: Encoder output tensor.+ encoder_out_lens: Encoder output lengths.+ token_num: TODO.+ """+ encoder_out_mask = (+ ~make_pad_mask(encoder_out_lens, maxlen=encoder_out.size(1))[:, None, :]+ ).to(encoder_out.device)+ ds_alphas, ds_cif_peak, us_alphas, us_peaks = self.predictor.get_upsample_timestamp(+ encoder_out, encoder_out_mask, token_num+ )+ return ds_alphas, ds_cif_peak, us_alphas, us_peaks+</code></pre></details></div>+<div class="api-detail" id="e155"><span class="dtag badge-m">method</span><h2><code>MonotonicAligner.encode(speech, speech_lengths, **kwargs)</code></h2><div class="dmod">funasr.models.monotonic_aligner.MonotonicAligner · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/monotonic_aligner/model.py#L153" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Encoder. Note that this method is used by asr_inference.py</p>+<h4>Args:</h4>+<ul>+<li><code>speech</code> — (Batch, Length, ...)</li>+<li><code>speech_lengths</code> — (Batch, )</li>+<li><code>ind</code> — int</li>+</ul>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def encode(+ self,+ speech: torch.Tensor,+ speech_lengths: torch.Tensor,+ **kwargs,+ ) -> Tuple[torch.Tensor, torch.Tensor]:+ """Encoder. Note that this method is used by asr_inference.py+ Args:+ speech: (Batch, Length, ...)+ speech_lengths: (Batch, )+ ind: int+ """+ with autocast(False):++ # Data augmentation+ if self.specaug is not None and self.training:+ speech, speech_lengths = self.specaug(speech, speech_lengths)++ # Normalization for feature: e.g. Global-CMVN, Utterance-CMVN+ if self.normalize is not None:+ speech, speech_lengths = self.normalize(speech, speech_lengths)++ # Forward encoder+ encoder_out, encoder_out_lens, _ = self.encoder(speech, speech_lengths)+ if isinstance(encoder_out, tuple):+ encoder_out = encoder_out[0]++ return encoder_out, encoder_out_lens+</code></pre></details></div>+<div class="api-detail" id="e156"><span class="dtag badge-m">method</span><h2><code>MonotonicAligner.inference(data_in, data_lengths, key, tokenizer, frontend, **kwargs)</code></h2><div class="dmod">funasr.models.monotonic_aligner.MonotonicAligner · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/monotonic_aligner/model.py#L182" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Run inference on input data.</p>+<h4>Args:</h4>+<ul>+<li><code>data_in</code> — Input data (audio samples, file paths, or text).</li>+<li><code>data_lengths</code> — Lengths of each input sample in the batch.</li>+<li><code>key</code> — Sample identifiers.</li>+<li><code>tokenizer</code> — Tokenizer instance for text encoding/decoding.</li>+<li><code>frontend</code> — Audio frontend for feature extraction.</li>+<li><code>**kwargs</code> — Additional keyword arguments.</li>+</ul>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def inference(+ self,+ data_in,+ data_lengths=None,+ key: list = None,+ tokenizer=None,+ frontend=None,+ **kwargs,+ ):+ """Run inference on input data.++ Args:+ data_in: Input data (audio samples, file paths, or text).+ data_lengths: Lengths of each input sample in the batch.+ key: Sample identifiers.+ tokenizer: Tokenizer instance for text encoding/decoding.+ frontend: Audio frontend for feature extraction.+ **kwargs: Additional keyword arguments.+ """+ meta_data = {}+ # extract fbank feats+ time1 = time.perf_counter()+ audio_list, text_token_int_list = load_audio_text_image_video(+ data_in,+ fs=frontend.fs,+ audio_fs=kwargs.get("fs", 16000),+ data_type=kwargs.get("data_type", "sound"),+ tokenizer=tokenizer,+ )+ time2 = time.perf_counter()+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/monotonic_aligner/model.py#L182" target="_blank" class="src-more">View full source on GitHub →</a></details></div>+<div class="api-detail" id="e157"><span class="dtag badge-c">class</span><h2>Paraformer</h2><div class="dmod">funasr.models.paraformer · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/paraformer/model.py#L30" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><ul>+<li><code>Paraformer</code> — Non-autoregressive End-to-End ASR Model.</li>+<li><code>High</code> — accuracy speech recognition for Chinese/English. The production workhorse.</li>+</ul>+<h4>Features:</h4>+<ul>+<li>Non-autoregressive (parallel decoding, fast inference)</li>+<li>Character-level timestamps via CIF predictor</li>+<li>Streaming and offline modes</li>+<li>Hotword customization</li>+<li>Speaker diarization (with spk_model)</li>+<li>ONNX export support</li>+<li><code>Output</code> — {"key": "...", "text": "recognized text", "timestamp": [[start_ms, end_ms], ...]}</li>+<li><code>Note</code> — Requires punc_model="ct-punc" for punctuation (unlike Fun-ASR-Nano/SenseVoice</li>+</ul>+<p>which output punctuation natively).</p>+<ul>+<li><code>Author</code> — Speech Lab of DAMO Academy, Alibaba Group</li>+<li><code>Paraformer</code> — Fast and Accurate Parallel Transformer for Non-autoregressive End-to-End Speech Recognition</li>+</ul>+<p>https://arxiv.org/abs/2206.08317</p>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code>class Paraformer(torch.nn.Module):+ """Paraformer: Non-autoregressive End-to-End ASR Model.++ High-accuracy speech recognition for Chinese/English. The production workhorse.++ Features:+ - Non-autoregressive (parallel decoding, fast inference)+ - Character-level timestamps via CIF predictor+ - Streaming and offline modes+ - Hotword customization+ - Speaker diarization (with spk_model)+ - ONNX export support++ Output: {"key": "...", "text": "recognized text", "timestamp": [[start_ms, end_ms], ...]}++ Note: Requires punc_model="ct-punc" for punctuation (unlike Fun-ASR-Nano/SenseVoice+ which output punctuation natively).++ Author: Speech Lab of DAMO Academy, Alibaba Group+ Paraformer: Fast and Accurate Parallel Transformer for Non-autoregressive End-to-End Speech Recognition+ https://arxiv.org/abs/2206.08317+ """++ def __init__(+ self,+ specaug: Optional[str] = None,+ specaug_conf: Optional[Dict] = None,+ normalize: str = None,+ normalize_conf: Optional[Dict] = None,+ encoder: str = None,+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/paraformer/model.py#L30" target="_blank" class="src-more">View full source on GitHub →</a></details><h3 class="mt">Methods</h3><div class="mblk"><code>.forward(speech, speech_lengths, text, text_lengths, **kwargs)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/paraformer/model.py#L215" target="_blank" class="gh-link">L215</a><div class="mdoc"><p>Encoder + Decoder + Calc loss</p>+<h4>Args:</h4>+<ul>+<li><code>speech</code> — (Batch, Length, ...)</li>+<li><code>speech_lengths</code> — (Batch, )</li>+<li><code>text</code> — (Batch, Length)</li>+<li><code>text_lengths</code> — (Batch,)</li>+</ul>+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def forward(+ self,+ speech: torch.Tensor,+ speech_lengths: torch.Tensor,+ text: torch.Tensor,+ text_lengths: torch.Tensor,+ **kwargs,+ ) -> Tuple[torch.Tensor, Dict[str, torch.Tensor], torch.Tensor]:+ """Encoder + Decoder + Calc loss+ Args:+ speech: (Batch, Length, ...)+ speech_lengths: (Batch, )+ text: (Batch, Length)+ text_lengths: (Batch,)+ """+ if len(text_lengths.size()) > 1:+ text_lengths = text_lengths[:, 0]+ if len(speech_lengths.size()) > 1:+ speech_lengths = speech_lengths[:, 0]++ batch_size = speech.shape[0]++ # Encoder+ encoder_out, encoder_out_lens = self.encode(speech, speech_lengths)++ loss_ctc, cer_ctc = None, None+ loss_pre = None+ stats = dict()++ # decoder: CTC branch+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/paraformer/model.py#L215" target="_blank" class="src-more">View full source →</a></details></div><div class="mblk"><code>.encode(speech, speech_lengths, **kwargs)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/paraformer/model.py#L286" target="_blank" class="gh-link">L286</a><div class="mdoc"><p>Encoder. Note that this method is used by asr_inference.py</p>+<h4>Args:</h4>+<ul>+<li><code>speech</code> — (Batch, Length, ...)</li>+<li><code>speech_lengths</code> — (Batch, )</li>+<li><code>ind</code> — int</li>+</ul>+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def encode(+ self,+ speech: torch.Tensor,+ speech_lengths: torch.Tensor,+ **kwargs,+ ) -> Tuple[torch.Tensor, torch.Tensor]:+ """Encoder. Note that this method is used by asr_inference.py+ Args:+ speech: (Batch, Length, ...)+ speech_lengths: (Batch, )+ ind: int+ """+ with autocast(False):++ # Data augmentation+ if self.specaug is not None and self.training:+ speech, speech_lengths = self.specaug(speech, speech_lengths)++ # Normalization for feature: e.g. Global-CMVN, Utterance-CMVN+ if self.normalize is not None:+ speech, speech_lengths = self.normalize(speech, speech_lengths)++ # Forward encoder+ encoder_out, encoder_out_lens, _ = self.encoder(speech, speech_lengths)+ if isinstance(encoder_out, tuple):+ encoder_out = encoder_out[0]++ return encoder_out, encoder_out_lens+</code></pre></details></div><div class="mblk"><code>.calc_predictor(encoder_out, encoder_out_lens)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/paraformer/model.py#L315" target="_blank" class="gh-link">L315</a><div class="mdoc"><p>Calc predictor.</p>+<h4>Args:</h4>+<ul>+<li><code>encoder_out</code> — Encoder output tensor.</li>+<li><code>encoder_out_lens</code> — Encoder output lengths.</li>+</ul>+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def calc_predictor(self, encoder_out, encoder_out_lens):++ """Calc predictor.++ Args:+ encoder_out: Encoder output tensor.+ encoder_out_lens: Encoder output lengths.+ """+ encoder_out_mask = (+ ~make_pad_mask(encoder_out_lens, maxlen=encoder_out.size(1))[:, None, :]+ ).to(encoder_out.device)+ pre_acoustic_embeds, pre_token_length, alphas, pre_peak_index = self.predictor(+ encoder_out, None, encoder_out_mask, ignore_id=self.ignore_id+ )+ return pre_acoustic_embeds, pre_token_length, alphas, pre_peak_index+</code></pre></details></div><div class="mblk"><code>.cal_decoder_with_predictor(encoder_out, encoder_out_lens, sematic_embeds, ys_pad_lens)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/paraformer/model.py#L331" target="_blank" class="gh-link">L331</a><div class="mdoc"><p>Cal decoder with predictor.</p>+<h4>Args:</h4>+<ul>+<li><code>encoder_out</code> — Encoder output tensor.</li>+<li><code>encoder_out_lens</code> — Encoder output lengths.</li>+<li><code>sematic_embeds</code> — TODO.</li>+<li><code>ys_pad_lens</code> — Lengths of ys_pad.</li>+</ul>+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def cal_decoder_with_predictor(+ self, encoder_out, encoder_out_lens, sematic_embeds, ys_pad_lens+ ):++ """Cal decoder with predictor.++ Args:+ encoder_out: Encoder output tensor.+ encoder_out_lens: Encoder output lengths.+ sematic_embeds: TODO.+ ys_pad_lens: Lengths of ys_pad.+ """+ decoder_outs = self.decoder(encoder_out, encoder_out_lens, sematic_embeds, ys_pad_lens)+ decoder_out = decoder_outs[0]+ decoder_out = torch.log_softmax(decoder_out, dim=-1)+ return decoder_out, ys_pad_lens+</code></pre></details></div><div class="mblk"><code>.sampler(encoder_out, encoder_out_lens, ys_pad, ys_pad_lens, pre_acoustic_embeds)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/paraformer/model.py#L408" target="_blank" class="gh-link">L408</a><div class="mdoc"><p>Sampler.</p>+<h4>Args:</h4>+<ul>+<li><code>encoder_out</code> — Encoder output tensor.</li>+<li><code>encoder_out_lens</code> — Encoder output lengths.</li>+<li><code>ys_pad</code> — TODO.</li>+<li><code>ys_pad_lens</code> — Lengths of ys_pad.</li>+<li><code>pre_acoustic_embeds</code> — TODO.</li>+</ul>+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def sampler(self, encoder_out, encoder_out_lens, ys_pad, ys_pad_lens, pre_acoustic_embeds):++ """Sampler.++ Args:+ encoder_out: Encoder output tensor.+ encoder_out_lens: Encoder output lengths.+ ys_pad: TODO.+ ys_pad_lens: Lengths of ys_pad.+ pre_acoustic_embeds: TODO.+ """+ tgt_mask = (~make_pad_mask(ys_pad_lens, maxlen=ys_pad_lens.max())[:, :, None]).to(+ ys_pad.device+ )+ ys_pad_masked = ys_pad * tgt_mask[:, :, 0]+ if self.share_embedding:+ ys_pad_embed = self.decoder.output_layer.weight[ys_pad_masked]+ else:+ ys_pad_embed = self.decoder.embed(ys_pad_masked)+ with torch.no_grad():+ decoder_outs = self.decoder(+ encoder_out, encoder_out_lens, pre_acoustic_embeds, ys_pad_lens+ )+ decoder_out, _ = decoder_outs[0], decoder_outs[1]+ pred_tokens = decoder_out.argmax(-1)+ nonpad_positions = ys_pad.ne(self.ignore_id)+ seq_lens = (nonpad_positions).sum(1)+ same_num = ((pred_tokens == ys_pad) & nonpad_positions).sum(1)+ input_mask = torch.ones_like(nonpad_positions)+ bsz, seq_len = ys_pad.size()+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/paraformer/model.py#L408" target="_blank" class="src-more">View full source →</a></details></div><div class="mblk"><code>.init_beam_search(**kwargs)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/paraformer/model.py#L482" target="_blank" class="gh-link">L482</a><div class="mdoc"><p>Init beam search.</p>+<h4>Args:</h4>+<ul>+<li><code>**kwargs</code> — Additional keyword arguments.</li>+</ul>+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def init_beam_search(+ self,+ **kwargs,+ ):+ """Init beam search.++ Args:+ **kwargs: Additional keyword arguments.+ """+ from funasr.models.paraformer.search import BeamSearchPara+ from funasr.models.transformer.scorers.ctc import CTCPrefixScorer+ from funasr.models.transformer.scorers.length_bonus import LengthBonus++ # 1. Build ASR model+ scorers = {}++ if self.ctc != None:+ ctc = CTCPrefixScorer(ctc=self.ctc, eos=self.eos)+ scorers.update(ctc=ctc)+ token_list = kwargs.get("token_list")+ scorers.update(+ length_bonus=LengthBonus(len(token_list)),+ )++ # 3. Build ngram model+ # ngram is not supported now+ ngram = None+ scorers["ngram"] = ngram++ weights = dict(+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/paraformer/model.py#L482" target="_blank" class="src-more">View full source →</a></details></div><div class="mblk"><code>.inference(data_in, data_lengths, key, tokenizer, frontend, **kwargs)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/paraformer/model.py#L534" target="_blank" class="gh-link">L534</a><div class="mdoc"><p>Run inference on input data.</p>+<h4>Args:</h4>+<ul>+<li><code>data_in</code> — Input data (audio samples, file paths, or text).</li>+<li><code>data_lengths</code> — Lengths of each input sample in the batch.</li>+<li><code>key</code> — Sample identifiers.</li>+<li><code>tokenizer</code> — Tokenizer instance for text encoding/decoding.</li>+<li><code>frontend</code> — Audio frontend for feature extraction.</li>+<li><code>**kwargs</code> — Additional keyword arguments.</li>+</ul>+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def inference(+ self,+ data_in,+ data_lengths=None,+ key: list = None,+ tokenizer=None,+ frontend=None,+ **kwargs,+ ):+ # init beamsearch+ """Run inference on input data.++ Args:+ data_in: Input data (audio samples, file paths, or text).+ data_lengths: Lengths of each input sample in the batch.+ key: Sample identifiers.+ tokenizer: Tokenizer instance for text encoding/decoding.+ frontend: Audio frontend for feature extraction.+ **kwargs: Additional keyword arguments.+ """+ is_use_ctc = kwargs.get("decoding_ctc_weight", 0.0) > 0.00001 and self.ctc != None+ is_use_lm = (+ kwargs.get("lm_weight", 0.0) > 0.00001 and kwargs.get("lm_file", None) is not None+ )+ pred_timestamp = kwargs.get("pred_timestamp", False)+ if self.beam_search is None and (is_use_lm or is_use_ctc):+ logging.info("enable beam_search")+ self.init_beam_search(**kwargs)+ self.nbest = kwargs.get("nbest", 1)++</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/paraformer/model.py#L534" target="_blank" class="src-more">View full source →</a></details></div><div class="mblk"><code>.export(**kwargs)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/paraformer/model.py#L699" target="_blank" class="gh-link">L699</a><div class="mdoc"><p>Export.</p>+<h4>Args:</h4>+<ul>+<li><code>**kwargs</code> — Additional keyword arguments.</li>+</ul>+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def export(self, **kwargs):+ """Export.++ Args:+ **kwargs: Additional keyword arguments.+ """+ from .export_meta import export_rebuild_model++ if "max_seq_len" not in kwargs:+ kwargs["max_seq_len"] = 512+ models = export_rebuild_model(model=self, **kwargs)+ return models+</code></pre></details></div></div>+<div class="api-detail" id="e158"><span class="dtag badge-m">method</span><h2><code>Paraformer.forward(speech, speech_lengths, text, text_lengths, **kwargs)</code></h2><div class="dmod">funasr.models.paraformer.Paraformer · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/paraformer/model.py#L215" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Encoder + Decoder + Calc loss</p>+<h4>Args:</h4>+<ul>+<li><code>speech</code> — (Batch, Length, ...)</li>+<li><code>speech_lengths</code> — (Batch, )</li>+<li><code>text</code> — (Batch, Length)</li>+<li><code>text_lengths</code> — (Batch,)</li>+</ul>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def forward(+ self,+ speech: torch.Tensor,+ speech_lengths: torch.Tensor,+ text: torch.Tensor,+ text_lengths: torch.Tensor,+ **kwargs,+ ) -> Tuple[torch.Tensor, Dict[str, torch.Tensor], torch.Tensor]:+ """Encoder + Decoder + Calc loss+ Args:+ speech: (Batch, Length, ...)+ speech_lengths: (Batch, )+ text: (Batch, Length)+ text_lengths: (Batch,)+ """+ if len(text_lengths.size()) > 1:+ text_lengths = text_lengths[:, 0]+ if len(speech_lengths.size()) > 1:+ speech_lengths = speech_lengths[:, 0]++ batch_size = speech.shape[0]++ # Encoder+ encoder_out, encoder_out_lens = self.encode(speech, speech_lengths)++ loss_ctc, cer_ctc = None, None+ loss_pre = None+ stats = dict()++ # decoder: CTC branch+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/paraformer/model.py#L215" target="_blank" class="src-more">View full source on GitHub →</a></details></div>+<div class="api-detail" id="e159"><span class="dtag badge-m">method</span><h2><code>Paraformer.encode(speech, speech_lengths, **kwargs)</code></h2><div class="dmod">funasr.models.paraformer.Paraformer · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/paraformer/model.py#L286" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Encoder. Note that this method is used by asr_inference.py</p>+<h4>Args:</h4>+<ul>+<li><code>speech</code> — (Batch, Length, ...)</li>+<li><code>speech_lengths</code> — (Batch, )</li>+<li><code>ind</code> — int</li>+</ul>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def encode(+ self,+ speech: torch.Tensor,+ speech_lengths: torch.Tensor,+ **kwargs,+ ) -> Tuple[torch.Tensor, torch.Tensor]:+ """Encoder. Note that this method is used by asr_inference.py+ Args:+ speech: (Batch, Length, ...)+ speech_lengths: (Batch, )+ ind: int+ """+ with autocast(False):++ # Data augmentation+ if self.specaug is not None and self.training:+ speech, speech_lengths = self.specaug(speech, speech_lengths)++ # Normalization for feature: e.g. Global-CMVN, Utterance-CMVN+ if self.normalize is not None:+ speech, speech_lengths = self.normalize(speech, speech_lengths)++ # Forward encoder+ encoder_out, encoder_out_lens, _ = self.encoder(speech, speech_lengths)+ if isinstance(encoder_out, tuple):+ encoder_out = encoder_out[0]++ return encoder_out, encoder_out_lens+</code></pre></details></div>+<div class="api-detail" id="e160"><span class="dtag badge-m">method</span><h2><code>Paraformer.calc_predictor(encoder_out, encoder_out_lens)</code></h2><div class="dmod">funasr.models.paraformer.Paraformer · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/paraformer/model.py#L315" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Calc predictor.</p>+<h4>Args:</h4>+<ul>+<li><code>encoder_out</code> — Encoder output tensor.</li>+<li><code>encoder_out_lens</code> — Encoder output lengths.</li>+</ul>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def calc_predictor(self, encoder_out, encoder_out_lens):++ """Calc predictor.++ Args:+ encoder_out: Encoder output tensor.+ encoder_out_lens: Encoder output lengths.+ """+ encoder_out_mask = (+ ~make_pad_mask(encoder_out_lens, maxlen=encoder_out.size(1))[:, None, :]+ ).to(encoder_out.device)+ pre_acoustic_embeds, pre_token_length, alphas, pre_peak_index = self.predictor(+ encoder_out, None, encoder_out_mask, ignore_id=self.ignore_id+ )+ return pre_acoustic_embeds, pre_token_length, alphas, pre_peak_index+</code></pre></details></div>+<div class="api-detail" id="e161"><span class="dtag badge-m">method</span><h2><code>Paraformer.cal_decoder_with_predictor(encoder_out, encoder_out_lens, sematic_embeds, ys_pad_lens)</code></h2><div class="dmod">funasr.models.paraformer.Paraformer · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/paraformer/model.py#L331" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Cal decoder with predictor.</p>+<h4>Args:</h4>+<ul>+<li><code>encoder_out</code> — Encoder output tensor.</li>+<li><code>encoder_out_lens</code> — Encoder output lengths.</li>+<li><code>sematic_embeds</code> — TODO.</li>+<li><code>ys_pad_lens</code> — Lengths of ys_pad.</li>+</ul>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def cal_decoder_with_predictor(+ self, encoder_out, encoder_out_lens, sematic_embeds, ys_pad_lens+ ):++ """Cal decoder with predictor.++ Args:+ encoder_out: Encoder output tensor.+ encoder_out_lens: Encoder output lengths.+ sematic_embeds: TODO.+ ys_pad_lens: Lengths of ys_pad.+ """+ decoder_outs = self.decoder(encoder_out, encoder_out_lens, sematic_embeds, ys_pad_lens)+ decoder_out = decoder_outs[0]+ decoder_out = torch.log_softmax(decoder_out, dim=-1)+ return decoder_out, ys_pad_lens+</code></pre></details></div>+<div class="api-detail" id="e162"><span class="dtag badge-m">method</span><h2><code>Paraformer.sampler(encoder_out, encoder_out_lens, ys_pad, ys_pad_lens, pre_acoustic_embeds)</code></h2><div class="dmod">funasr.models.paraformer.Paraformer · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/paraformer/model.py#L408" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Sampler.</p>+<h4>Args:</h4>+<ul>+<li><code>encoder_out</code> — Encoder output tensor.</li>+<li><code>encoder_out_lens</code> — Encoder output lengths.</li>+<li><code>ys_pad</code> — TODO.</li>+<li><code>ys_pad_lens</code> — Lengths of ys_pad.</li>+<li><code>pre_acoustic_embeds</code> — TODO.</li>+</ul>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def sampler(self, encoder_out, encoder_out_lens, ys_pad, ys_pad_lens, pre_acoustic_embeds):++ """Sampler.++ Args:+ encoder_out: Encoder output tensor.+ encoder_out_lens: Encoder output lengths.+ ys_pad: TODO.+ ys_pad_lens: Lengths of ys_pad.+ pre_acoustic_embeds: TODO.+ """+ tgt_mask = (~make_pad_mask(ys_pad_lens, maxlen=ys_pad_lens.max())[:, :, None]).to(+ ys_pad.device+ )+ ys_pad_masked = ys_pad * tgt_mask[:, :, 0]+ if self.share_embedding:+ ys_pad_embed = self.decoder.output_layer.weight[ys_pad_masked]+ else:+ ys_pad_embed = self.decoder.embed(ys_pad_masked)+ with torch.no_grad():+ decoder_outs = self.decoder(+ encoder_out, encoder_out_lens, pre_acoustic_embeds, ys_pad_lens+ )+ decoder_out, _ = decoder_outs[0], decoder_outs[1]+ pred_tokens = decoder_out.argmax(-1)+ nonpad_positions = ys_pad.ne(self.ignore_id)+ seq_lens = (nonpad_positions).sum(1)+ same_num = ((pred_tokens == ys_pad) & nonpad_positions).sum(1)+ input_mask = torch.ones_like(nonpad_positions)+ bsz, seq_len = ys_pad.size()+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/paraformer/model.py#L408" target="_blank" class="src-more">View full source on GitHub →</a></details></div>+<div class="api-detail" id="e163"><span class="dtag badge-m">method</span><h2><code>Paraformer.init_beam_search(**kwargs)</code></h2><div class="dmod">funasr.models.paraformer.Paraformer · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/paraformer/model.py#L482" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Init beam search.</p>+<h4>Args:</h4>+<ul>+<li><code>**kwargs</code> — Additional keyword arguments.</li>+</ul>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def init_beam_search(+ self,+ **kwargs,+ ):+ """Init beam search.++ Args:+ **kwargs: Additional keyword arguments.+ """+ from funasr.models.paraformer.search import BeamSearchPara+ from funasr.models.transformer.scorers.ctc import CTCPrefixScorer+ from funasr.models.transformer.scorers.length_bonus import LengthBonus++ # 1. Build ASR model+ scorers = {}++ if self.ctc != None:+ ctc = CTCPrefixScorer(ctc=self.ctc, eos=self.eos)+ scorers.update(ctc=ctc)+ token_list = kwargs.get("token_list")+ scorers.update(+ length_bonus=LengthBonus(len(token_list)),+ )++ # 3. Build ngram model+ # ngram is not supported now+ ngram = None+ scorers["ngram"] = ngram++ weights = dict(+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/paraformer/model.py#L482" target="_blank" class="src-more">View full source on GitHub →</a></details></div>+<div class="api-detail" id="e164"><span class="dtag badge-m">method</span><h2><code>Paraformer.inference(data_in, data_lengths, key, tokenizer, frontend, **kwargs)</code></h2><div class="dmod">funasr.models.paraformer.Paraformer · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/paraformer/model.py#L534" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Run inference on input data.</p>+<h4>Args:</h4>+<ul>+<li><code>data_in</code> — Input data (audio samples, file paths, or text).</li>+<li><code>data_lengths</code> — Lengths of each input sample in the batch.</li>+<li><code>key</code> — Sample identifiers.</li>+<li><code>tokenizer</code> — Tokenizer instance for text encoding/decoding.</li>+<li><code>frontend</code> — Audio frontend for feature extraction.</li>+<li><code>**kwargs</code> — Additional keyword arguments.</li>+</ul>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def inference(+ self,+ data_in,+ data_lengths=None,+ key: list = None,+ tokenizer=None,+ frontend=None,+ **kwargs,+ ):+ # init beamsearch+ """Run inference on input data.++ Args:+ data_in: Input data (audio samples, file paths, or text).+ data_lengths: Lengths of each input sample in the batch.+ key: Sample identifiers.+ tokenizer: Tokenizer instance for text encoding/decoding.+ frontend: Audio frontend for feature extraction.+ **kwargs: Additional keyword arguments.+ """+ is_use_ctc = kwargs.get("decoding_ctc_weight", 0.0) > 0.00001 and self.ctc != None+ is_use_lm = (+ kwargs.get("lm_weight", 0.0) > 0.00001 and kwargs.get("lm_file", None) is not None+ )+ pred_timestamp = kwargs.get("pred_timestamp", False)+ if self.beam_search is None and (is_use_lm or is_use_ctc):+ logging.info("enable beam_search")+ self.init_beam_search(**kwargs)+ self.nbest = kwargs.get("nbest", 1)++</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/paraformer/model.py#L534" target="_blank" class="src-more">View full source on GitHub →</a></details></div>+<div class="api-detail" id="e165"><span class="dtag badge-m">method</span><h2><code>Paraformer.export(**kwargs)</code></h2><div class="dmod">funasr.models.paraformer.Paraformer · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/paraformer/model.py#L699" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Export.</p>+<h4>Args:</h4>+<ul>+<li><code>**kwargs</code> — Additional keyword arguments.</li>+</ul>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def export(self, **kwargs):+ """Export.++ Args:+ **kwargs: Additional keyword arguments.+ """+ from .export_meta import export_rebuild_model++ if "max_seq_len" not in kwargs:+ kwargs["max_seq_len"] = 512+ models = export_rebuild_model(model=self, **kwargs)+ return models+</code></pre></details></div>+<div class="api-detail" id="e166"><span class="dtag badge-c">class</span><h2>ParaformerStreaming</h2><div class="dmod">funasr.models.paraformer_streaming · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/paraformer_streaming/model.py#L43" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><ul>+<li><code>ParaformerStreaming</code> — Streaming (online) version of Paraformer.</li>+</ul>+<p>Processes audio chunk-by-chunk with encoder lookback for real-time</p>+<p>transcription. Uses cache mechanism to maintain state across chunks.</p>+<ul>+<li><code>Usage</code> — generate(input=chunk, cache=cache, is_final=bool, chunk_size=[0,10,5])</li>+<li><code>Output</code> — {"key": str, "text": str} (partial results per chunk)</li>+</ul>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code>class ParaformerStreaming(Paraformer):+ """ParaformerStreaming: Streaming (online) version of Paraformer.++ Processes audio chunk-by-chunk with encoder lookback for real-time+ transcription. Uses cache mechanism to maintain state across chunks.++ Usage: generate(input=chunk, cache=cache, is_final=bool, chunk_size=[0,10,5])+ Output: {"key": str, "text": str} (partial results per chunk)+ """++ def __init__(+ self,+ *args,+ **kwargs,+ ):++ """Initialize ParaformerStreaming.++ Args:+ *args: Variable positional arguments.+ **kwargs: Additional keyword arguments.+ """+ super().__init__(*args, **kwargs)++ self.sampling_ratio = kwargs.get("sampling_ratio", 0.2)++ self.scama_mask = None+ if (+ hasattr(self.encoder, "overlap_chunk_cls")+ and self.encoder.overlap_chunk_cls is not None+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/paraformer_streaming/model.py#L43" target="_blank" class="src-more">View full source on GitHub →</a></details><h3 class="mt">Methods</h3><div class="mblk"><code>.forward(speech, speech_lengths, text, text_lengths, **kwargs)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/paraformer_streaming/model.py#L83" target="_blank" class="gh-link">L83</a><div class="mdoc"><p>Encoder + Decoder + Calc loss</p>+<h4>Args:</h4>+<ul>+<li><code>speech</code> — (Batch, Length, ...)</li>+<li><code>speech_lengths</code> — (Batch, )</li>+<li><code>text</code> — (Batch, Length)</li>+<li><code>text_lengths</code> — (Batch,)</li>+</ul>+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def forward(+ self,+ speech: torch.Tensor,+ speech_lengths: torch.Tensor,+ text: torch.Tensor,+ text_lengths: torch.Tensor,+ **kwargs,+ ) -> Tuple[torch.Tensor, Dict[str, torch.Tensor], torch.Tensor]:+ """Encoder + Decoder + Calc loss+ Args:+ speech: (Batch, Length, ...)+ speech_lengths: (Batch, )+ text: (Batch, Length)+ text_lengths: (Batch,)+ """+ decoding_ind = kwargs.get("decoding_ind")+ if len(text_lengths.size()) > 1:+ text_lengths = text_lengths[:, 0]+ if len(speech_lengths.size()) > 1:+ speech_lengths = speech_lengths[:, 0]++ batch_size = speech.shape[0]++ # Encoder+ if hasattr(self.encoder, "overlap_chunk_cls"):+ ind = self.encoder.overlap_chunk_cls.random_choice(self.training, decoding_ind)+ encoder_out, encoder_out_lens = self.encode(speech, speech_lengths, ind=ind)+ else:+ encoder_out, encoder_out_lens = self.encode(speech, speech_lengths)++</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/paraformer_streaming/model.py#L83" target="_blank" class="src-more">View full source →</a></details></div><div class="mblk"><code>.encode_chunk(speech, speech_lengths, cache, **kwargs)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/paraformer_streaming/model.py#L165" target="_blank" class="gh-link">L165</a><div class="mdoc"><p>Encode chunk.</p>+<h4>Args:</h4>+<ul>+<li><code>speech</code> — Speech audio tensor, shape (batch, time).</li>+<li><code>speech_lengths</code> — Length of each speech sample.</li>+<li><code>cache</code> — State cache dict for streaming inference.</li>+<li><code>**kwargs</code> — Additional keyword arguments.</li>+</ul>+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def encode_chunk(+ self,+ speech: torch.Tensor,+ speech_lengths: torch.Tensor,+ cache: dict = None,+ **kwargs,+ ) -> Tuple[torch.Tensor, torch.Tensor]:+ """Encode chunk.++ Args:+ speech: Speech audio tensor, shape (batch, time).+ speech_lengths: Length of each speech sample.+ cache: State cache dict for streaming inference.+ **kwargs: Additional keyword arguments.+ """+ if cache is None:+ cache = {}+ """Frontend + Encoder. Note that this method is used by asr_inference.py+ Args:+ speech: (Batch, Length, ...)+ speech_lengths: (Batch, )+ ind: int+ """+ with autocast(False):++ # Data augmentation+ if self.specaug is not None and self.training:+ speech, speech_lengths = self.specaug(speech, speech_lengths)++ # Normalization for feature: e.g. Global-CMVN, Utterance-CMVN+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/paraformer_streaming/model.py#L165" target="_blank" class="src-more">View full source →</a></details></div><div class="mblk"><code>.sampler(encoder_out, encoder_out_lens, ys_pad, ys_pad_lens, pre_acoustic_embeds, chunk_mask)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/paraformer_streaming/model.py#L336" target="_blank" class="gh-link">L336</a><div class="mdoc"><p>Sampler.</p>+<h4>Args:</h4>+<ul>+<li><code>encoder_out</code> — Encoder output tensor.</li>+<li><code>encoder_out_lens</code> — Encoder output lengths.</li>+<li><code>ys_pad</code> — TODO.</li>+<li><code>ys_pad_lens</code> — Lengths of ys_pad.</li>+<li><code>pre_acoustic_embeds</code> — TODO.</li>+<li><code>chunk_mask</code> — TODO.</li>+</ul>+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def sampler(+ self,+ encoder_out,+ encoder_out_lens,+ ys_pad,+ ys_pad_lens,+ pre_acoustic_embeds,+ chunk_mask=None,+ ):++ """Sampler.++ Args:+ encoder_out: Encoder output tensor.+ encoder_out_lens: Encoder output lengths.+ ys_pad: TODO.+ ys_pad_lens: Lengths of ys_pad.+ pre_acoustic_embeds: TODO.+ chunk_mask: TODO.+ """+ tgt_mask = (~make_pad_mask(ys_pad_lens, maxlen=ys_pad_lens.max())[:, :, None]).to(+ ys_pad.device+ )+ ys_pad_masked = ys_pad * tgt_mask[:, :, 0]+ if self.share_embedding:+ ys_pad_embed = self.decoder.output_layer.weight[ys_pad_masked]+ else:+ ys_pad_embed = self.decoder.embed(ys_pad_masked)+ with torch.no_grad():+ decoder_outs = self.decoder(+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/paraformer_streaming/model.py#L336" target="_blank" class="src-more">View full source →</a></details></div><div class="mblk"><code>.calc_predictor(encoder_out, encoder_out_lens)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/paraformer_streaming/model.py#L392" target="_blank" class="gh-link">L392</a><div class="mdoc"><p>Calc predictor.</p>+<h4>Args:</h4>+<ul>+<li><code>encoder_out</code> — Encoder output tensor.</li>+<li><code>encoder_out_lens</code> — Encoder output lengths.</li>+</ul>+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def calc_predictor(self, encoder_out, encoder_out_lens):++ """Calc predictor.++ Args:+ encoder_out: Encoder output tensor.+ encoder_out_lens: Encoder output lengths.+ """+ encoder_out_mask = (+ ~make_pad_mask(encoder_out_lens, maxlen=encoder_out.size(1))[:, None, :]+ ).to(encoder_out.device)+ mask_chunk_predictor = None+ if self.encoder.overlap_chunk_cls is not None:+ mask_chunk_predictor = self.encoder.overlap_chunk_cls.get_mask_chunk_predictor(+ None, device=encoder_out.device, batch_size=encoder_out.size(0)+ )+ mask_shfit_chunk = self.encoder.overlap_chunk_cls.get_mask_shfit_chunk(+ None, device=encoder_out.device, batch_size=encoder_out.size(0)+ )+ encoder_out = encoder_out * mask_shfit_chunk+ pre_acoustic_embeds, pre_token_length, pre_alphas, pre_peak_index = self.predictor(+ encoder_out,+ None,+ encoder_out_mask,+ ignore_id=self.ignore_id,+ mask_chunk_predictor=mask_chunk_predictor,+ target_label_length=None,+ )+ predictor_alignments, predictor_alignments_len = self.predictor.gen_frame_alignments(+ pre_alphas,+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/paraformer_streaming/model.py#L392" target="_blank" class="src-more">View full source →</a></details></div><div class="mblk"><code>.calc_predictor_chunk(encoder_out, encoder_out_lens, cache, **kwargs)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/paraformer_streaming/model.py#L460" target="_blank" class="gh-link">L460</a><div class="mdoc"><p>Calc predictor chunk.</p>+<h4>Args:</h4>+<ul>+<li><code>encoder_out</code> — Encoder output tensor.</li>+<li><code>encoder_out_lens</code> — Encoder output lengths.</li>+<li><code>cache</code> — State cache dict for streaming inference.</li>+<li><code>**kwargs</code> — Additional keyword arguments.</li>+</ul>+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def calc_predictor_chunk(self, encoder_out, encoder_out_lens, cache=None, **kwargs):+ """Calc predictor chunk.++ Args:+ encoder_out: Encoder output tensor.+ encoder_out_lens: Encoder output lengths.+ cache: State cache dict for streaming inference.+ **kwargs: Additional keyword arguments.+ """+ is_final = kwargs.get("is_final", False)++ return self.predictor.forward_chunk(encoder_out, cache["encoder"], is_final=is_final)+</code></pre></details></div><div class="mblk"><code>.cal_decoder_with_predictor(encoder_out, encoder_out_lens, sematic_embeds, ys_pad_lens)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/paraformer_streaming/model.py#L473" target="_blank" class="gh-link">L473</a><div class="mdoc"><p>Cal decoder with predictor.</p>+<h4>Args:</h4>+<ul>+<li><code>encoder_out</code> — Encoder output tensor.</li>+<li><code>encoder_out_lens</code> — Encoder output lengths.</li>+<li><code>sematic_embeds</code> — TODO.</li>+<li><code>ys_pad_lens</code> — Lengths of ys_pad.</li>+</ul>+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def cal_decoder_with_predictor(+ self, encoder_out, encoder_out_lens, sematic_embeds, ys_pad_lens+ ):+ """Cal decoder with predictor.++ Args:+ encoder_out: Encoder output tensor.+ encoder_out_lens: Encoder output lengths.+ sematic_embeds: TODO.+ ys_pad_lens: Lengths of ys_pad.+ """+ decoder_outs = self.decoder(+ encoder_out, encoder_out_lens, sematic_embeds, ys_pad_lens, self.scama_mask+ )+ decoder_out = decoder_outs[0]+ decoder_out = torch.log_softmax(decoder_out, dim=-1)+ return decoder_out, ys_pad_lens+</code></pre></details></div><div class="mblk"><code>.cal_decoder_with_predictor_chunk(encoder_out, encoder_out_lens, sematic_embeds, ys_pad_lens, cache)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/paraformer_streaming/model.py#L491" target="_blank" class="gh-link">L491</a><div class="mdoc"><p>Cal decoder with predictor chunk.</p>+<h4>Args:</h4>+<ul>+<li><code>encoder_out</code> — Encoder output tensor.</li>+<li><code>encoder_out_lens</code> — Encoder output lengths.</li>+<li><code>sematic_embeds</code> — TODO.</li>+<li><code>ys_pad_lens</code> — Lengths of ys_pad.</li>+<li><code>cache</code> — State cache dict for streaming inference.</li>+</ul>+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def cal_decoder_with_predictor_chunk(+ self, encoder_out, encoder_out_lens, sematic_embeds, ys_pad_lens, cache=None+ ):+ """Cal decoder with predictor chunk.++ Args:+ encoder_out: Encoder output tensor.+ encoder_out_lens: Encoder output lengths.+ sematic_embeds: TODO.+ ys_pad_lens: Lengths of ys_pad.+ cache: State cache dict for streaming inference.+ """+ decoder_outs = self.decoder.forward_chunk(encoder_out, sematic_embeds, cache["decoder"])+ decoder_out = decoder_outs+ decoder_out = torch.log_softmax(decoder_out, dim=-1)+ return decoder_out, ys_pad_lens+</code></pre></details></div><div class="mblk"><code>.init_cache(cache, **kwargs)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/paraformer_streaming/model.py#L508" target="_blank" class="gh-link">L508</a><div class="mdoc"><p>Init cache.</p>+<h4>Args:</h4>+<ul>+<li><code>cache</code> — State cache dict for streaming inference.</li>+<li><code>**kwargs</code> — Additional keyword arguments.</li>+</ul>+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def init_cache(self, cache: dict = None, **kwargs):+ """Init cache.++ Args:+ cache: State cache dict for streaming inference.+ **kwargs: Additional keyword arguments.+ """+ if cache is None:+ cache = {}+ chunk_size = kwargs.get("chunk_size", [0, 10, 5])+ encoder_chunk_look_back = kwargs.get("encoder_chunk_look_back", 0)+ decoder_chunk_look_back = kwargs.get("decoder_chunk_look_back", 0)+ batch_size = 1++ enc_output_size = kwargs["encoder_conf"]["output_size"]+ feats_dims = kwargs["frontend_conf"]["n_mels"] * kwargs["frontend_conf"]["lfr_m"]+ cache_encoder = {+ "start_idx": 0,+ "cif_hidden": torch.zeros((batch_size, 1, enc_output_size)),+ "cif_alphas": torch.zeros((batch_size, 1)),+ "chunk_size": chunk_size,+ "encoder_chunk_look_back": encoder_chunk_look_back,+ "last_chunk": False,+ "opt": None,+ "feats": torch.zeros((batch_size, chunk_size[0] + chunk_size[2], feats_dims)),+ "tail_chunk": False,+ }+ cache["encoder"] = cache_encoder++ cache_decoder = {+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/paraformer_streaming/model.py#L508" target="_blank" class="src-more">View full source →</a></details></div><div class="mblk"><code>.generate_chunk(speech, speech_lengths, key, tokenizer, frontend, **kwargs)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/paraformer_streaming/model.py#L549" target="_blank" class="gh-link">L549</a><div class="mdoc"><p>Generate chunk.</p>+<h4>Args:</h4>+<ul>+<li><code>speech</code> — Speech audio tensor, shape (batch, time).</li>+<li><code>speech_lengths</code> — Length of each speech sample.</li>+<li><code>key</code> — Sample identifiers.</li>+<li><code>tokenizer</code> — Tokenizer instance for text encoding/decoding.</li>+<li><code>frontend</code> — Audio frontend for feature extraction.</li>+<li><code>**kwargs</code> — Additional keyword arguments.</li>+</ul>+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def generate_chunk(+ self,+ speech,+ speech_lengths=None,+ key: list = None,+ tokenizer=None,+ frontend=None,+ **kwargs,+ ):+ """Generate chunk.++ Args:+ speech: Speech audio tensor, shape (batch, time).+ speech_lengths: Length of each speech sample.+ key: Sample identifiers.+ tokenizer: Tokenizer instance for text encoding/decoding.+ frontend: Audio frontend for feature extraction.+ **kwargs: Additional keyword arguments.+ """+ cache = kwargs.get("cache", {})+ speech = speech.to(device=kwargs["device"])+ speech_lengths = speech_lengths.to(device=kwargs["device"])++ # Encoder+ encoder_out, encoder_out_lens = self.encode_chunk(+ speech, speech_lengths, cache=cache, is_final=kwargs.get("is_final", False)+ )+ if isinstance(encoder_out, tuple):+ encoder_out = encoder_out[0]++</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/paraformer_streaming/model.py#L549" target="_blank" class="src-more">View full source →</a></details></div><div class="mblk"><code>.inference(data_in, data_lengths, key, tokenizer, frontend, cache, **kwargs)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/paraformer_streaming/model.py#L647" target="_blank" class="gh-link">L647</a><div class="mdoc"><p>Run inference on input data.</p>+<h4>Args:</h4>+<ul>+<li><code>data_in</code> — Input data (audio samples, file paths, or text).</li>+<li><code>data_lengths</code> — Lengths of each input sample in the batch.</li>+<li><code>key</code> — Sample identifiers.</li>+<li><code>tokenizer</code> — Tokenizer instance for text encoding/decoding.</li>+<li><code>frontend</code> — Audio frontend for feature extraction.</li>+<li><code>cache</code> — State cache dict for streaming inference.</li>+<li><code>**kwargs</code> — Additional keyword arguments.</li>+</ul>+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def inference(+ self,+ data_in,+ data_lengths=None,+ key: list = None,+ tokenizer=None,+ frontend=None,+ cache: dict = None,+ **kwargs,+ ):+ """Run inference on input data.++ Args:+ data_in: Input data (audio samples, file paths, or text).+ data_lengths: Lengths of each input sample in the batch.+ key: Sample identifiers.+ tokenizer: Tokenizer instance for text encoding/decoding.+ frontend: Audio frontend for feature extraction.+ cache: State cache dict for streaming inference.+ **kwargs: Additional keyword arguments.+ """+ if cache is None:+ cache = {}++ # init beamsearch+ is_use_ctc = kwargs.get("decoding_ctc_weight", 0.0) > 0.00001 and self.ctc != None+ is_use_lm = (+ kwargs.get("lm_weight", 0.0) > 0.00001 and kwargs.get("lm_file", None) is not None+ )+ if self.beam_search is None and (is_use_lm or is_use_ctc):+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/paraformer_streaming/model.py#L647" target="_blank" class="src-more">View full source →</a></details></div><div class="mblk"><code>.export(**kwargs)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/paraformer_streaming/model.py#L762" target="_blank" class="gh-link">L762</a><div class="mdoc"><p>Export.</p>+<h4>Args:</h4>+<ul>+<li><code>**kwargs</code> — Additional keyword arguments.</li>+</ul>+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def export(self, **kwargs):+ """Export.++ Args:+ **kwargs: Additional keyword arguments.+ """+ from .export_meta import export_rebuild_model++ models = export_rebuild_model(model=self, **kwargs)+ return models+</code></pre></details></div></div>+<div class="api-detail" id="e167"><span class="dtag badge-m">method</span><h2><code>ParaformerStreaming.forward(speech, speech_lengths, text, text_lengths, **kwargs)</code></h2><div class="dmod">funasr.models.paraformer_streaming.ParaformerStreaming · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/paraformer_streaming/model.py#L83" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Encoder + Decoder + Calc loss</p>+<h4>Args:</h4>+<ul>+<li><code>speech</code> — (Batch, Length, ...)</li>+<li><code>speech_lengths</code> — (Batch, )</li>+<li><code>text</code> — (Batch, Length)</li>+<li><code>text_lengths</code> — (Batch,)</li>+</ul>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def forward(+ self,+ speech: torch.Tensor,+ speech_lengths: torch.Tensor,+ text: torch.Tensor,+ text_lengths: torch.Tensor,+ **kwargs,+ ) -> Tuple[torch.Tensor, Dict[str, torch.Tensor], torch.Tensor]:+ """Encoder + Decoder + Calc loss+ Args:+ speech: (Batch, Length, ...)+ speech_lengths: (Batch, )+ text: (Batch, Length)+ text_lengths: (Batch,)+ """+ decoding_ind = kwargs.get("decoding_ind")+ if len(text_lengths.size()) > 1:+ text_lengths = text_lengths[:, 0]+ if len(speech_lengths.size()) > 1:+ speech_lengths = speech_lengths[:, 0]++ batch_size = speech.shape[0]++ # Encoder+ if hasattr(self.encoder, "overlap_chunk_cls"):+ ind = self.encoder.overlap_chunk_cls.random_choice(self.training, decoding_ind)+ encoder_out, encoder_out_lens = self.encode(speech, speech_lengths, ind=ind)+ else:+ encoder_out, encoder_out_lens = self.encode(speech, speech_lengths)++</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/paraformer_streaming/model.py#L83" target="_blank" class="src-more">View full source on GitHub →</a></details></div>+<div class="api-detail" id="e168"><span class="dtag badge-m">method</span><h2><code>ParaformerStreaming.encode_chunk(speech, speech_lengths, cache, **kwargs)</code></h2><div class="dmod">funasr.models.paraformer_streaming.ParaformerStreaming · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/paraformer_streaming/model.py#L165" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Encode chunk.</p>+<h4>Args:</h4>+<ul>+<li><code>speech</code> — Speech audio tensor, shape (batch, time).</li>+<li><code>speech_lengths</code> — Length of each speech sample.</li>+<li><code>cache</code> — State cache dict for streaming inference.</li>+<li><code>**kwargs</code> — Additional keyword arguments.</li>+</ul>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def encode_chunk(+ self,+ speech: torch.Tensor,+ speech_lengths: torch.Tensor,+ cache: dict = None,+ **kwargs,+ ) -> Tuple[torch.Tensor, torch.Tensor]:+ """Encode chunk.++ Args:+ speech: Speech audio tensor, shape (batch, time).+ speech_lengths: Length of each speech sample.+ cache: State cache dict for streaming inference.+ **kwargs: Additional keyword arguments.+ """+ if cache is None:+ cache = {}+ """Frontend + Encoder. Note that this method is used by asr_inference.py+ Args:+ speech: (Batch, Length, ...)+ speech_lengths: (Batch, )+ ind: int+ """+ with autocast(False):++ # Data augmentation+ if self.specaug is not None and self.training:+ speech, speech_lengths = self.specaug(speech, speech_lengths)++ # Normalization for feature: e.g. Global-CMVN, Utterance-CMVN+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/paraformer_streaming/model.py#L165" target="_blank" class="src-more">View full source on GitHub →</a></details></div>+<div class="api-detail" id="e169"><span class="dtag badge-m">method</span><h2><code>ParaformerStreaming.sampler(encoder_out, encoder_out_lens, ys_pad, ys_pad_lens, pre_acoustic_embeds, chunk_mask)</code></h2><div class="dmod">funasr.models.paraformer_streaming.ParaformerStreaming · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/paraformer_streaming/model.py#L336" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Sampler.</p>+<h4>Args:</h4>+<ul>+<li><code>encoder_out</code> — Encoder output tensor.</li>+<li><code>encoder_out_lens</code> — Encoder output lengths.</li>+<li><code>ys_pad</code> — TODO.</li>+<li><code>ys_pad_lens</code> — Lengths of ys_pad.</li>+<li><code>pre_acoustic_embeds</code> — TODO.</li>+<li><code>chunk_mask</code> — TODO.</li>+</ul>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def sampler(+ self,+ encoder_out,+ encoder_out_lens,+ ys_pad,+ ys_pad_lens,+ pre_acoustic_embeds,+ chunk_mask=None,+ ):++ """Sampler.++ Args:+ encoder_out: Encoder output tensor.+ encoder_out_lens: Encoder output lengths.+ ys_pad: TODO.+ ys_pad_lens: Lengths of ys_pad.+ pre_acoustic_embeds: TODO.+ chunk_mask: TODO.+ """+ tgt_mask = (~make_pad_mask(ys_pad_lens, maxlen=ys_pad_lens.max())[:, :, None]).to(+ ys_pad.device+ )+ ys_pad_masked = ys_pad * tgt_mask[:, :, 0]+ if self.share_embedding:+ ys_pad_embed = self.decoder.output_layer.weight[ys_pad_masked]+ else:+ ys_pad_embed = self.decoder.embed(ys_pad_masked)+ with torch.no_grad():+ decoder_outs = self.decoder(+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/paraformer_streaming/model.py#L336" target="_blank" class="src-more">View full source on GitHub →</a></details></div>+<div class="api-detail" id="e170"><span class="dtag badge-m">method</span><h2><code>ParaformerStreaming.calc_predictor(encoder_out, encoder_out_lens)</code></h2><div class="dmod">funasr.models.paraformer_streaming.ParaformerStreaming · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/paraformer_streaming/model.py#L392" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Calc predictor.</p>+<h4>Args:</h4>+<ul>+<li><code>encoder_out</code> — Encoder output tensor.</li>+<li><code>encoder_out_lens</code> — Encoder output lengths.</li>+</ul>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def calc_predictor(self, encoder_out, encoder_out_lens):++ """Calc predictor.++ Args:+ encoder_out: Encoder output tensor.+ encoder_out_lens: Encoder output lengths.+ """+ encoder_out_mask = (+ ~make_pad_mask(encoder_out_lens, maxlen=encoder_out.size(1))[:, None, :]+ ).to(encoder_out.device)+ mask_chunk_predictor = None+ if self.encoder.overlap_chunk_cls is not None:+ mask_chunk_predictor = self.encoder.overlap_chunk_cls.get_mask_chunk_predictor(+ None, device=encoder_out.device, batch_size=encoder_out.size(0)+ )+ mask_shfit_chunk = self.encoder.overlap_chunk_cls.get_mask_shfit_chunk(+ None, device=encoder_out.device, batch_size=encoder_out.size(0)+ )+ encoder_out = encoder_out * mask_shfit_chunk+ pre_acoustic_embeds, pre_token_length, pre_alphas, pre_peak_index = self.predictor(+ encoder_out,+ None,+ encoder_out_mask,+ ignore_id=self.ignore_id,+ mask_chunk_predictor=mask_chunk_predictor,+ target_label_length=None,+ )+ predictor_alignments, predictor_alignments_len = self.predictor.gen_frame_alignments(+ pre_alphas,+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/paraformer_streaming/model.py#L392" target="_blank" class="src-more">View full source on GitHub →</a></details></div>+<div class="api-detail" id="e171"><span class="dtag badge-m">method</span><h2><code>ParaformerStreaming.calc_predictor_chunk(encoder_out, encoder_out_lens, cache, **kwargs)</code></h2><div class="dmod">funasr.models.paraformer_streaming.ParaformerStreaming · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/paraformer_streaming/model.py#L460" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Calc predictor chunk.</p>+<h4>Args:</h4>+<ul>+<li><code>encoder_out</code> — Encoder output tensor.</li>+<li><code>encoder_out_lens</code> — Encoder output lengths.</li>+<li><code>cache</code> — State cache dict for streaming inference.</li>+<li><code>**kwargs</code> — Additional keyword arguments.</li>+</ul>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def calc_predictor_chunk(self, encoder_out, encoder_out_lens, cache=None, **kwargs):+ """Calc predictor chunk.++ Args:+ encoder_out: Encoder output tensor.+ encoder_out_lens: Encoder output lengths.+ cache: State cache dict for streaming inference.+ **kwargs: Additional keyword arguments.+ """+ is_final = kwargs.get("is_final", False)++ return self.predictor.forward_chunk(encoder_out, cache["encoder"], is_final=is_final)+</code></pre></details></div>+<div class="api-detail" id="e172"><span class="dtag badge-m">method</span><h2><code>ParaformerStreaming.cal_decoder_with_predictor(encoder_out, encoder_out_lens, sematic_embeds, ys_pad_lens)</code></h2><div class="dmod">funasr.models.paraformer_streaming.ParaformerStreaming · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/paraformer_streaming/model.py#L473" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Cal decoder with predictor.</p>+<h4>Args:</h4>+<ul>+<li><code>encoder_out</code> — Encoder output tensor.</li>+<li><code>encoder_out_lens</code> — Encoder output lengths.</li>+<li><code>sematic_embeds</code> — TODO.</li>+<li><code>ys_pad_lens</code> — Lengths of ys_pad.</li>+</ul>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def cal_decoder_with_predictor(+ self, encoder_out, encoder_out_lens, sematic_embeds, ys_pad_lens+ ):+ """Cal decoder with predictor.++ Args:+ encoder_out: Encoder output tensor.+ encoder_out_lens: Encoder output lengths.+ sematic_embeds: TODO.+ ys_pad_lens: Lengths of ys_pad.+ """+ decoder_outs = self.decoder(+ encoder_out, encoder_out_lens, sematic_embeds, ys_pad_lens, self.scama_mask+ )+ decoder_out = decoder_outs[0]+ decoder_out = torch.log_softmax(decoder_out, dim=-1)+ return decoder_out, ys_pad_lens+</code></pre></details></div>+<div class="api-detail" id="e173"><span class="dtag badge-m">method</span><h2><code>ParaformerStreaming.cal_decoder_with_predictor_chunk(encoder_out, encoder_out_lens, sematic_embeds, ys_pad_lens, cache)</code></h2><div class="dmod">funasr.models.paraformer_streaming.ParaformerStreaming · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/paraformer_streaming/model.py#L491" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Cal decoder with predictor chunk.</p>+<h4>Args:</h4>+<ul>+<li><code>encoder_out</code> — Encoder output tensor.</li>+<li><code>encoder_out_lens</code> — Encoder output lengths.</li>+<li><code>sematic_embeds</code> — TODO.</li>+<li><code>ys_pad_lens</code> — Lengths of ys_pad.</li>+<li><code>cache</code> — State cache dict for streaming inference.</li>+</ul>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def cal_decoder_with_predictor_chunk(+ self, encoder_out, encoder_out_lens, sematic_embeds, ys_pad_lens, cache=None+ ):+ """Cal decoder with predictor chunk.++ Args:+ encoder_out: Encoder output tensor.+ encoder_out_lens: Encoder output lengths.+ sematic_embeds: TODO.+ ys_pad_lens: Lengths of ys_pad.+ cache: State cache dict for streaming inference.+ """+ decoder_outs = self.decoder.forward_chunk(encoder_out, sematic_embeds, cache["decoder"])+ decoder_out = decoder_outs+ decoder_out = torch.log_softmax(decoder_out, dim=-1)+ return decoder_out, ys_pad_lens+</code></pre></details></div>+<div class="api-detail" id="e174"><span class="dtag badge-m">method</span><h2><code>ParaformerStreaming.init_cache(cache, **kwargs)</code></h2><div class="dmod">funasr.models.paraformer_streaming.ParaformerStreaming · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/paraformer_streaming/model.py#L508" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Init cache.</p>+<h4>Args:</h4>+<ul>+<li><code>cache</code> — State cache dict for streaming inference.</li>+<li><code>**kwargs</code> — Additional keyword arguments.</li>+</ul>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def init_cache(self, cache: dict = None, **kwargs):+ """Init cache.++ Args:+ cache: State cache dict for streaming inference.+ **kwargs: Additional keyword arguments.+ """+ if cache is None:+ cache = {}+ chunk_size = kwargs.get("chunk_size", [0, 10, 5])+ encoder_chunk_look_back = kwargs.get("encoder_chunk_look_back", 0)+ decoder_chunk_look_back = kwargs.get("decoder_chunk_look_back", 0)+ batch_size = 1++ enc_output_size = kwargs["encoder_conf"]["output_size"]+ feats_dims = kwargs["frontend_conf"]["n_mels"] * kwargs["frontend_conf"]["lfr_m"]+ cache_encoder = {+ "start_idx": 0,+ "cif_hidden": torch.zeros((batch_size, 1, enc_output_size)),+ "cif_alphas": torch.zeros((batch_size, 1)),+ "chunk_size": chunk_size,+ "encoder_chunk_look_back": encoder_chunk_look_back,+ "last_chunk": False,+ "opt": None,+ "feats": torch.zeros((batch_size, chunk_size[0] + chunk_size[2], feats_dims)),+ "tail_chunk": False,+ }+ cache["encoder"] = cache_encoder++ cache_decoder = {+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/paraformer_streaming/model.py#L508" target="_blank" class="src-more">View full source on GitHub →</a></details></div>+<div class="api-detail" id="e175"><span class="dtag badge-m">method</span><h2><code>ParaformerStreaming.generate_chunk(speech, speech_lengths, key, tokenizer, frontend, **kwargs)</code></h2><div class="dmod">funasr.models.paraformer_streaming.ParaformerStreaming · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/paraformer_streaming/model.py#L549" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Generate chunk.</p>+<h4>Args:</h4>+<ul>+<li><code>speech</code> — Speech audio tensor, shape (batch, time).</li>+<li><code>speech_lengths</code> — Length of each speech sample.</li>+<li><code>key</code> — Sample identifiers.</li>+<li><code>tokenizer</code> — Tokenizer instance for text encoding/decoding.</li>+<li><code>frontend</code> — Audio frontend for feature extraction.</li>+<li><code>**kwargs</code> — Additional keyword arguments.</li>+</ul>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def generate_chunk(+ self,+ speech,+ speech_lengths=None,+ key: list = None,+ tokenizer=None,+ frontend=None,+ **kwargs,+ ):+ """Generate chunk.++ Args:+ speech: Speech audio tensor, shape (batch, time).+ speech_lengths: Length of each speech sample.+ key: Sample identifiers.+ tokenizer: Tokenizer instance for text encoding/decoding.+ frontend: Audio frontend for feature extraction.+ **kwargs: Additional keyword arguments.+ """+ cache = kwargs.get("cache", {})+ speech = speech.to(device=kwargs["device"])+ speech_lengths = speech_lengths.to(device=kwargs["device"])++ # Encoder+ encoder_out, encoder_out_lens = self.encode_chunk(+ speech, speech_lengths, cache=cache, is_final=kwargs.get("is_final", False)+ )+ if isinstance(encoder_out, tuple):+ encoder_out = encoder_out[0]++</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/paraformer_streaming/model.py#L549" target="_blank" class="src-more">View full source on GitHub →</a></details></div>+<div class="api-detail" id="e176"><span class="dtag badge-m">method</span><h2><code>ParaformerStreaming.inference(data_in, data_lengths, key, tokenizer, frontend, cache, **kwargs)</code></h2><div class="dmod">funasr.models.paraformer_streaming.ParaformerStreaming · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/paraformer_streaming/model.py#L647" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Run inference on input data.</p>+<h4>Args:</h4>+<ul>+<li><code>data_in</code> — Input data (audio samples, file paths, or text).</li>+<li><code>data_lengths</code> — Lengths of each input sample in the batch.</li>+<li><code>key</code> — Sample identifiers.</li>+<li><code>tokenizer</code> — Tokenizer instance for text encoding/decoding.</li>+<li><code>frontend</code> — Audio frontend for feature extraction.</li>+<li><code>cache</code> — State cache dict for streaming inference.</li>+<li><code>**kwargs</code> — Additional keyword arguments.</li>+</ul>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def inference(+ self,+ data_in,+ data_lengths=None,+ key: list = None,+ tokenizer=None,+ frontend=None,+ cache: dict = None,+ **kwargs,+ ):+ """Run inference on input data.++ Args:+ data_in: Input data (audio samples, file paths, or text).+ data_lengths: Lengths of each input sample in the batch.+ key: Sample identifiers.+ tokenizer: Tokenizer instance for text encoding/decoding.+ frontend: Audio frontend for feature extraction.+ cache: State cache dict for streaming inference.+ **kwargs: Additional keyword arguments.+ """+ if cache is None:+ cache = {}++ # init beamsearch+ is_use_ctc = kwargs.get("decoding_ctc_weight", 0.0) > 0.00001 and self.ctc != None+ is_use_lm = (+ kwargs.get("lm_weight", 0.0) > 0.00001 and kwargs.get("lm_file", None) is not None+ )+ if self.beam_search is None and (is_use_lm or is_use_ctc):+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/paraformer_streaming/model.py#L647" target="_blank" class="src-more">View full source on GitHub →</a></details></div>+<div class="api-detail" id="e177"><span class="dtag badge-m">method</span><h2><code>ParaformerStreaming.export(**kwargs)</code></h2><div class="dmod">funasr.models.paraformer_streaming.ParaformerStreaming · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/paraformer_streaming/model.py#L762" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Export.</p>+<h4>Args:</h4>+<ul>+<li><code>**kwargs</code> — Additional keyword arguments.</li>+</ul>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def export(self, **kwargs):+ """Export.++ Args:+ **kwargs: Additional keyword arguments.+ """+ from .export_meta import export_rebuild_model++ models = export_rebuild_model(model=self, **kwargs)+ return models+</code></pre></details></div>+<div class="api-detail" id="e178"><span class="dtag badge-c">class</span><h2>Paraformer</h2><div class="dmod">funasr.models.paraformer_v2_community · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/paraformer_v2_community/model.py#L31" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><ul>+<li><code>Author</code> — Speech Lab of DAMO Academy, Alibaba Group</li>+<li><code>Paraformer</code> — Fast and Accurate Parallel Transformer for Non-autoregressive End-to-End Speech Recognition</li>+</ul>+<p>https://arxiv.org/abs/2206.08317</p>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code>class Paraformer(torch.nn.Module):+ """+ Author: Speech Lab of DAMO Academy, Alibaba Group+ Paraformer: Fast and Accurate Parallel Transformer for Non-autoregressive End-to-End Speech Recognition+ https://arxiv.org/abs/2206.08317+ """++ def __init__(+ self,+ specaug: Optional[str] = None,+ specaug_conf: Optional[Dict] = None,+ normalize: str = None,+ normalize_conf: Optional[Dict] = None,+ encoder: str = None,+ encoder_conf: Optional[Dict] = None,+ decoder: str = None,+ decoder_conf: Optional[Dict] = None,+ ctc: str = None,+ ctc_conf: Optional[Dict] = None,+ ctc_weight: float = 0.5,+ input_size: int = 80,+ vocab_size: int = -1,+ ignore_id: int = -1,+ blank_id: int = 0,+ sos: int = 1,+ eos: int = 2,+ lsm_weight: float = 0.0,+ length_normalized_loss: bool = False,+ # report_cer: bool = True,+ # report_wer: bool = True,+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/paraformer_v2_community/model.py#L31" target="_blank" class="src-more">View full source on GitHub →</a></details><h3 class="mt">Methods</h3><div class="mblk"><code>.forward(speech, speech_lengths, text, text_lengths, **kwargs)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/paraformer_v2_community/model.py#L182" target="_blank" class="gh-link">L182</a><div class="mdoc"><p>Encoder + Decoder + Calc loss</p>+<h4>Args:</h4>+<ul>+<li><code>speech</code> — (Batch, Length, ...)</li>+<li><code>speech_lengths</code> — (Batch, )</li>+<li><code>text</code> — (Batch, Length)</li>+<li><code>text_lengths</code> — (Batch,)</li>+</ul>+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def forward(+ self,+ speech: torch.Tensor,+ speech_lengths: torch.Tensor,+ text: torch.Tensor,+ text_lengths: torch.Tensor,+ **kwargs,+ ) -> Tuple[torch.Tensor, Dict[str, torch.Tensor], torch.Tensor]:+ """Encoder + Decoder + Calc loss+ Args:+ speech: (Batch, Length, ...)+ speech_lengths: (Batch, )+ text: (Batch, Length)+ text_lengths: (Batch,)+ """+ if len(text_lengths.size()) > 1:+ text_lengths = text_lengths[:, 0]+ if len(speech_lengths.size()) > 1:+ speech_lengths = speech_lengths[:, 0]++ batch_size = speech.shape[0]++ # Encoder+ encoder_out, encoder_out_lens = self.encode(speech, speech_lengths)++ loss_ctc, cer_ctc = None, None+ loss_pre = None+ stats = dict()++ # decoder: CTC branch+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/paraformer_v2_community/model.py#L182" target="_blank" class="src-more">View full source →</a></details></div><div class="mblk"><code>.encode(speech, speech_lengths, **kwargs)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/paraformer_v2_community/model.py#L250" target="_blank" class="gh-link">L250</a><div class="mdoc"><p>Encoder. Note that this method is used by asr_inference.py</p>+<h4>Args:</h4>+<ul>+<li><code>speech</code> — (Batch, Length, ...)</li>+<li><code>speech_lengths</code> — (Batch, )</li>+<li><code>ind</code> — int</li>+</ul>+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def encode(+ self,+ speech: torch.Tensor,+ speech_lengths: torch.Tensor,+ **kwargs,+ ) -> Tuple[torch.Tensor, torch.Tensor]:+ """Encoder. Note that this method is used by asr_inference.py+ Args:+ speech: (Batch, Length, ...)+ speech_lengths: (Batch, )+ ind: int+ """+ with autocast(False):++ # Data augmentation+ if self.specaug is not None and self.training:+ speech, speech_lengths = self.specaug(speech, speech_lengths)++ # Normalization for feature: e.g. Global-CMVN, Utterance-CMVN+ if self.normalize is not None:+ speech, speech_lengths = self.normalize(speech, speech_lengths)++ # Forward encoder+ encoder_out, encoder_out_lens, _ = self.encoder(speech, speech_lengths)+ if isinstance(encoder_out, tuple):+ encoder_out = encoder_out[0]++ return encoder_out, encoder_out_lens+</code></pre></details></div><div class="mblk"><code>.map_alignment_to_target_index(align_path, blank_id)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/paraformer_v2_community/model.py#L366" target="_blank" class="gh-link">L366</a><div class="mdoc"><p>Robustly map CTC alignment path (Token IDs) to Target Indices.</p>+<p>Logic:</p>+<p>Detect boundaries where a new token segment begins.</p>+<p>A segment starts if the current frame is a Token AND it is different from the previous frame</p>+<p>(considering CTC topology where repeats are separated by blanks or are distinct tokens).</p>+<p>Example:</p>+<ul>+<li><code>Text</code> — [A, B]</li>+</ul>+<p>Align Path: [A, A, _, B, B]</p>+<ul>+<li><code>Output</code> — [0, 0, -1, 1, 1]</li>+</ul>+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def map_alignment_to_target_index(self, align_path, blank_id):+ """+ Robustly map CTC alignment path (Token IDs) to Target Indices.++ Logic:+ Detect boundaries where a new token segment begins.+ A segment starts if the current frame is a Token AND it is different from the previous frame+ (considering CTC topology where repeats are separated by blanks or are distinct tokens).++ Example:+ Text: [A, B]+ Align Path: [A, A, _, B, B]+ Output: [0, 0, -1, 1, 1]+ """+ # 1. Identify where the path is NOT blank+ is_token = align_path != blank_id++ # 2. Identify transitions+ prev_path = torch.roll(align_path, 1)+ # Handle the very first frame: if it's a token, it must be the start of segment 0.+ prev_path[0] = blank_id # force mismatch for the first element++ # A new segment starts if: It's a token AND (it differs from prev OR prev was blank)+ # Note: If align_path[i] == align_path[i-1] (and not blank), it's the same segment.+ new_segment_start = is_token & (align_path != prev_path)++ # 3. Cumulative sum to assign indices (1..U)+ segment_ids = torch.cumsum(new_segment_start.long(), dim=0) - 1++ # 4. Mask out blank positions with -1+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/paraformer_v2_community/model.py#L366" target="_blank" class="src-more">View full source →</a></details></div><div class="mblk"><code>.force_align(ctc_probs, y, blank_id)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/paraformer_v2_community/model.py#L399" target="_blank" class="gh-link">L399</a><div class="mdoc"><p>ctc forced alignment.</p>+<h4>Args:</h4>+<p>torch.Tensor ctc_probs: hidden state sequence, 2d tensor (T, D)</p>+<p>torch.Tensor y: id sequence tensor 1d tensor (L)</p>+<p>int blank_id: blank symbol index</p>+<h4>Returns:</h4>+<ul>+<li><code>torch.Tensor</code> — alignment result</li>+</ul>+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def force_align(self, ctc_probs: torch.Tensor, y: torch.Tensor, blank_id=0) -> list:+ """ctc forced alignment.++ Args:+ torch.Tensor ctc_probs: hidden state sequence, 2d tensor (T, D)+ torch.Tensor y: id sequence tensor 1d tensor (L)+ int blank_id: blank symbol index+ Returns:+ torch.Tensor: alignment result+ """+ ctc_probs = ctc_probs[None].cpu()+ y = y[None].cpu()+ alignments, _ = torchaudio.functional.forced_align(ctc_probs, y, blank=blank_id)+ return alignments[0]+</code></pre></details></div><div class="mblk"><code>.average_repeats_training(ctc_probs, target_idx_path, target_len)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/paraformer_v2_community/model.py#L414" target="_blank" class="gh-link">L414</a><div class="mdoc"><p>Aggregates frames belonging to the same target index using scatter_add.</p>+<h4>Args:</h4>+<ul>+<li><code>ctc_probs</code> — [T, V]</li>+<li><code>target_idx_path</code> — [T], values in [-1, 0, ... U-1]</li>+<li><code>target_len</code> — U</li>+</ul>+<h4>Returns:</h4>+<ul>+<li><code>compressed</code> — [U, V]</li>+</ul>+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def average_repeats_training(self, ctc_probs, target_idx_path, target_len):+ """+ Aggregates frames belonging to the same target index using scatter_add.++ Args:+ ctc_probs: [T, V]+ target_idx_path: [T], values in [-1, 0, ... U-1]+ target_len: U+ Returns:+ compressed: [U, V]+ """+ U = target_len+ V = ctc_probs.size(1)++ compressed = torch.zeros((U, V), device=ctc_probs.device, dtype=ctc_probs.dtype)+ counts = torch.zeros((U, 1), device=ctc_probs.device, dtype=ctc_probs.dtype)++ # Filter valid frames (non-blank)+ mask = target_idx_path != -1+ valid_indices = target_idx_path[mask] # [T_valid]+ valid_probs = ctc_probs[mask] # [T_valid, V]++ if valid_indices.numel() == 0:+ return compressed++ # Scatter Add Probs+ index_expanded = valid_indices.unsqueeze(1).repeat(1, V)+ compressed.scatter_add_(0, index_expanded, valid_probs)++ # Scatter Add Counts+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/paraformer_v2_community/model.py#L414" target="_blank" class="src-more">View full source →</a></details></div><div class="mblk"><code>.average_repeats_inference(ctc_probs, greedy_path)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/paraformer_v2_community/model.py#L451" target="_blank" class="gh-link">L451</a><div class="mdoc"><h4>Returns:</h4>+<ul>+<li><code>merged_probs</code> — [U', V]</li>+<li><code>timestamps</code> — List[Tuple[int, int]] -> [(start_frame, end_frame), ...]</li>+</ul>+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def average_repeats_inference(self, ctc_probs, greedy_path):+ """+ Returns:+ merged_probs: [U', V]+ timestamps: List[Tuple[int, int]] -> [(start_frame, end_frame), ...]+ """+ if greedy_path.numel() == 0:+ return torch.zeros((0, ctc_probs.size(1)), device=ctc_probs.device)++ # Find consecutive segments in the greedy path+ unique_tokens, counts = torch.unique_consecutive(greedy_path, return_counts=True)++ # Compute start and end indices for each segment+ end_indices = torch.cumsum(counts, dim=0)+ start_indices = torch.cat([torch.tensor([0], device=counts.device), end_indices[:-1]])++ merged_probs = []++ for i, token in enumerate(unique_tokens):+ if token != self.blank_id:+ start = start_indices[i].item()+ end = end_indices[i].item()++ # Extract and average probabilities for the decoder+ avg_prob = ctc_probs[start:end].mean(dim=0)+ merged_probs.append(avg_prob)+++ if not merged_probs:+ return torch.zeros((0, ctc_probs.size(1)), device=ctc_probs.device)+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/paraformer_v2_community/model.py#L451" target="_blank" class="src-more">View full source →</a></details></div><div class="mblk"><code>.inference(data_in, data_lengths, key, tokenizer, frontend, **kwargs)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/paraformer_v2_community/model.py#L484" target="_blank" class="gh-link">L484</a><div class="mdoc"><p>Run inference on input data.</p>+<h4>Args:</h4>+<ul>+<li><code>data_in</code> — Input data (audio samples, file paths, or text).</li>+<li><code>data_lengths</code> — Lengths of each input sample in the batch.</li>+<li><code>key</code> — Sample identifiers.</li>+<li><code>tokenizer</code> — Tokenizer instance for text encoding/decoding.</li>+<li><code>frontend</code> — Audio frontend for feature extraction.</li>+<li><code>**kwargs</code> — Additional keyword arguments.</li>+</ul>+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def inference(+ self,+ data_in,+ data_lengths=None,+ key: list = None,+ tokenizer=None,+ frontend=None,+ **kwargs,+ ):++ """Run inference on input data.++ Args:+ data_in: Input data (audio samples, file paths, or text).+ data_lengths: Lengths of each input sample in the batch.+ key: Sample identifiers.+ tokenizer: Tokenizer instance for text encoding/decoding.+ frontend: Audio frontend for feature extraction.+ **kwargs: Additional keyword arguments.+ """+ meta_data = {}+ if (+ isinstance(data_in, torch.Tensor) and kwargs.get("data_type", "sound") == "fbank"+ ): # fbank+ speech, speech_lengths = data_in, data_lengths+ if len(speech.shape) < 3:+ speech = speech[None, :, :]+ if speech_lengths is not None:+ speech_lengths = speech_lengths.squeeze(-1)+ else:+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/paraformer_v2_community/model.py#L484" target="_blank" class="src-more">View full source →</a></details></div><div class="mblk"><code>.export(**kwargs)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/paraformer_v2_community/model.py#L592" target="_blank" class="gh-link">L592</a><div class="mdoc"><p>Export.</p>+<h4>Args:</h4>+<ul>+<li><code>**kwargs</code> — Additional keyword arguments.</li>+</ul>+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def export(self, **kwargs):+ """Export.++ Args:+ **kwargs: Additional keyword arguments.+ """+ from .export_meta import export_rebuild_model++ if "max_seq_len" not in kwargs:+ kwargs["max_seq_len"] = 512+ models = export_rebuild_model(model=self, **kwargs)+ return models+</code></pre></details></div></div>+<div class="api-detail" id="e179"><span class="dtag badge-m">method</span><h2><code>Paraformer.forward(speech, speech_lengths, text, text_lengths, **kwargs)</code></h2><div class="dmod">funasr.models.paraformer_v2_community.Paraformer · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/paraformer_v2_community/model.py#L182" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Encoder + Decoder + Calc loss</p>+<h4>Args:</h4>+<ul>+<li><code>speech</code> — (Batch, Length, ...)</li>+<li><code>speech_lengths</code> — (Batch, )</li>+<li><code>text</code> — (Batch, Length)</li>+<li><code>text_lengths</code> — (Batch,)</li>+</ul>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def forward(+ self,+ speech: torch.Tensor,+ speech_lengths: torch.Tensor,+ text: torch.Tensor,+ text_lengths: torch.Tensor,+ **kwargs,+ ) -> Tuple[torch.Tensor, Dict[str, torch.Tensor], torch.Tensor]:+ """Encoder + Decoder + Calc loss+ Args:+ speech: (Batch, Length, ...)+ speech_lengths: (Batch, )+ text: (Batch, Length)+ text_lengths: (Batch,)+ """+ if len(text_lengths.size()) > 1:+ text_lengths = text_lengths[:, 0]+ if len(speech_lengths.size()) > 1:+ speech_lengths = speech_lengths[:, 0]++ batch_size = speech.shape[0]++ # Encoder+ encoder_out, encoder_out_lens = self.encode(speech, speech_lengths)++ loss_ctc, cer_ctc = None, None+ loss_pre = None+ stats = dict()++ # decoder: CTC branch+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/paraformer_v2_community/model.py#L182" target="_blank" class="src-more">View full source on GitHub →</a></details></div>+<div class="api-detail" id="e180"><span class="dtag badge-m">method</span><h2><code>Paraformer.encode(speech, speech_lengths, **kwargs)</code></h2><div class="dmod">funasr.models.paraformer_v2_community.Paraformer · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/paraformer_v2_community/model.py#L250" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Encoder. Note that this method is used by asr_inference.py</p>+<h4>Args:</h4>+<ul>+<li><code>speech</code> — (Batch, Length, ...)</li>+<li><code>speech_lengths</code> — (Batch, )</li>+<li><code>ind</code> — int</li>+</ul>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def encode(+ self,+ speech: torch.Tensor,+ speech_lengths: torch.Tensor,+ **kwargs,+ ) -> Tuple[torch.Tensor, torch.Tensor]:+ """Encoder. Note that this method is used by asr_inference.py+ Args:+ speech: (Batch, Length, ...)+ speech_lengths: (Batch, )+ ind: int+ """+ with autocast(False):++ # Data augmentation+ if self.specaug is not None and self.training:+ speech, speech_lengths = self.specaug(speech, speech_lengths)++ # Normalization for feature: e.g. Global-CMVN, Utterance-CMVN+ if self.normalize is not None:+ speech, speech_lengths = self.normalize(speech, speech_lengths)++ # Forward encoder+ encoder_out, encoder_out_lens, _ = self.encoder(speech, speech_lengths)+ if isinstance(encoder_out, tuple):+ encoder_out = encoder_out[0]++ return encoder_out, encoder_out_lens+</code></pre></details></div>+<div class="api-detail" id="e181"><span class="dtag badge-m">method</span><h2><code>Paraformer.map_alignment_to_target_index(align_path, blank_id)</code></h2><div class="dmod">funasr.models.paraformer_v2_community.Paraformer · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/paraformer_v2_community/model.py#L366" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Robustly map CTC alignment path (Token IDs) to Target Indices.</p>+<p>Logic:</p>+<p>Detect boundaries where a new token segment begins.</p>+<p>A segment starts if the current frame is a Token AND it is different from the previous frame</p>+<p>(considering CTC topology where repeats are separated by blanks or are distinct tokens).</p>+<p>Example:</p>+<ul>+<li><code>Text</code> — [A, B]</li>+</ul>+<p>Align Path: [A, A, _, B, B]</p>+<ul>+<li><code>Output</code> — [0, 0, -1, 1, 1]</li>+</ul>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def map_alignment_to_target_index(self, align_path, blank_id):+ """+ Robustly map CTC alignment path (Token IDs) to Target Indices.++ Logic:+ Detect boundaries where a new token segment begins.+ A segment starts if the current frame is a Token AND it is different from the previous frame+ (considering CTC topology where repeats are separated by blanks or are distinct tokens).++ Example:+ Text: [A, B]+ Align Path: [A, A, _, B, B]+ Output: [0, 0, -1, 1, 1]+ """+ # 1. Identify where the path is NOT blank+ is_token = align_path != blank_id++ # 2. Identify transitions+ prev_path = torch.roll(align_path, 1)+ # Handle the very first frame: if it's a token, it must be the start of segment 0.+ prev_path[0] = blank_id # force mismatch for the first element++ # A new segment starts if: It's a token AND (it differs from prev OR prev was blank)+ # Note: If align_path[i] == align_path[i-1] (and not blank), it's the same segment.+ new_segment_start = is_token & (align_path != prev_path)++ # 3. Cumulative sum to assign indices (1..U)+ segment_ids = torch.cumsum(new_segment_start.long(), dim=0) - 1++ # 4. Mask out blank positions with -1+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/paraformer_v2_community/model.py#L366" target="_blank" class="src-more">View full source on GitHub →</a></details></div>+<div class="api-detail" id="e182"><span class="dtag badge-m">method</span><h2><code>Paraformer.force_align(ctc_probs, y, blank_id)</code></h2><div class="dmod">funasr.models.paraformer_v2_community.Paraformer · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/paraformer_v2_community/model.py#L399" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>ctc forced alignment.</p>+<h4>Args:</h4>+<p>torch.Tensor ctc_probs: hidden state sequence, 2d tensor (T, D)</p>+<p>torch.Tensor y: id sequence tensor 1d tensor (L)</p>+<p>int blank_id: blank symbol index</p>+<h4>Returns:</h4>+<ul>+<li><code>torch.Tensor</code> — alignment result</li>+</ul>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def force_align(self, ctc_probs: torch.Tensor, y: torch.Tensor, blank_id=0) -> list:+ """ctc forced alignment.++ Args:+ torch.Tensor ctc_probs: hidden state sequence, 2d tensor (T, D)+ torch.Tensor y: id sequence tensor 1d tensor (L)+ int blank_id: blank symbol index+ Returns:+ torch.Tensor: alignment result+ """+ ctc_probs = ctc_probs[None].cpu()+ y = y[None].cpu()+ alignments, _ = torchaudio.functional.forced_align(ctc_probs, y, blank=blank_id)+ return alignments[0]+</code></pre></details></div>+<div class="api-detail" id="e183"><span class="dtag badge-m">method</span><h2><code>Paraformer.average_repeats_training(ctc_probs, target_idx_path, target_len)</code></h2><div class="dmod">funasr.models.paraformer_v2_community.Paraformer · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/paraformer_v2_community/model.py#L414" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Aggregates frames belonging to the same target index using scatter_add.</p>+<h4>Args:</h4>+<ul>+<li><code>ctc_probs</code> — [T, V]</li>+<li><code>target_idx_path</code> — [T], values in [-1, 0, ... U-1]</li>+<li><code>target_len</code> — U</li>+</ul>+<h4>Returns:</h4>+<ul>+<li><code>compressed</code> — [U, V]</li>+</ul>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def average_repeats_training(self, ctc_probs, target_idx_path, target_len):+ """+ Aggregates frames belonging to the same target index using scatter_add.++ Args:+ ctc_probs: [T, V]+ target_idx_path: [T], values in [-1, 0, ... U-1]+ target_len: U+ Returns:+ compressed: [U, V]+ """+ U = target_len+ V = ctc_probs.size(1)++ compressed = torch.zeros((U, V), device=ctc_probs.device, dtype=ctc_probs.dtype)+ counts = torch.zeros((U, 1), device=ctc_probs.device, dtype=ctc_probs.dtype)++ # Filter valid frames (non-blank)+ mask = target_idx_path != -1+ valid_indices = target_idx_path[mask] # [T_valid]+ valid_probs = ctc_probs[mask] # [T_valid, V]++ if valid_indices.numel() == 0:+ return compressed++ # Scatter Add Probs+ index_expanded = valid_indices.unsqueeze(1).repeat(1, V)+ compressed.scatter_add_(0, index_expanded, valid_probs)++ # Scatter Add Counts+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/paraformer_v2_community/model.py#L414" target="_blank" class="src-more">View full source on GitHub →</a></details></div>+<div class="api-detail" id="e184"><span class="dtag badge-m">method</span><h2><code>Paraformer.average_repeats_inference(ctc_probs, greedy_path)</code></h2><div class="dmod">funasr.models.paraformer_v2_community.Paraformer · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/paraformer_v2_community/model.py#L451" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><h4>Returns:</h4>+<ul>+<li><code>merged_probs</code> — [U', V]</li>+<li><code>timestamps</code> — List[Tuple[int, int]] -> [(start_frame, end_frame), ...]</li>+</ul>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def average_repeats_inference(self, ctc_probs, greedy_path):+ """+ Returns:+ merged_probs: [U', V]+ timestamps: List[Tuple[int, int]] -> [(start_frame, end_frame), ...]+ """+ if greedy_path.numel() == 0:+ return torch.zeros((0, ctc_probs.size(1)), device=ctc_probs.device)++ # Find consecutive segments in the greedy path+ unique_tokens, counts = torch.unique_consecutive(greedy_path, return_counts=True)++ # Compute start and end indices for each segment+ end_indices = torch.cumsum(counts, dim=0)+ start_indices = torch.cat([torch.tensor([0], device=counts.device), end_indices[:-1]])++ merged_probs = []++ for i, token in enumerate(unique_tokens):+ if token != self.blank_id:+ start = start_indices[i].item()+ end = end_indices[i].item()++ # Extract and average probabilities for the decoder+ avg_prob = ctc_probs[start:end].mean(dim=0)+ merged_probs.append(avg_prob)+++ if not merged_probs:+ return torch.zeros((0, ctc_probs.size(1)), device=ctc_probs.device)+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/paraformer_v2_community/model.py#L451" target="_blank" class="src-more">View full source on GitHub →</a></details></div>+<div class="api-detail" id="e185"><span class="dtag badge-m">method</span><h2><code>Paraformer.inference(data_in, data_lengths, key, tokenizer, frontend, **kwargs)</code></h2><div class="dmod">funasr.models.paraformer_v2_community.Paraformer · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/paraformer_v2_community/model.py#L484" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Run inference on input data.</p>+<h4>Args:</h4>+<ul>+<li><code>data_in</code> — Input data (audio samples, file paths, or text).</li>+<li><code>data_lengths</code> — Lengths of each input sample in the batch.</li>+<li><code>key</code> — Sample identifiers.</li>+<li><code>tokenizer</code> — Tokenizer instance for text encoding/decoding.</li>+<li><code>frontend</code> — Audio frontend for feature extraction.</li>+<li><code>**kwargs</code> — Additional keyword arguments.</li>+</ul>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def inference(+ self,+ data_in,+ data_lengths=None,+ key: list = None,+ tokenizer=None,+ frontend=None,+ **kwargs,+ ):++ """Run inference on input data.++ Args:+ data_in: Input data (audio samples, file paths, or text).+ data_lengths: Lengths of each input sample in the batch.+ key: Sample identifiers.+ tokenizer: Tokenizer instance for text encoding/decoding.+ frontend: Audio frontend for feature extraction.+ **kwargs: Additional keyword arguments.+ """+ meta_data = {}+ if (+ isinstance(data_in, torch.Tensor) and kwargs.get("data_type", "sound") == "fbank"+ ): # fbank+ speech, speech_lengths = data_in, data_lengths+ if len(speech.shape) < 3:+ speech = speech[None, :, :]+ if speech_lengths is not None:+ speech_lengths = speech_lengths.squeeze(-1)+ else:+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/paraformer_v2_community/model.py#L484" target="_blank" class="src-more">View full source on GitHub →</a></details></div>+<div class="api-detail" id="e186"><span class="dtag badge-m">method</span><h2><code>Paraformer.export(**kwargs)</code></h2><div class="dmod">funasr.models.paraformer_v2_community.Paraformer · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/paraformer_v2_community/model.py#L592" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Export.</p>+<h4>Args:</h4>+<ul>+<li><code>**kwargs</code> — Additional keyword arguments.</li>+</ul>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def export(self, **kwargs):+ """Export.++ Args:+ **kwargs: Additional keyword arguments.+ """+ from .export_meta import export_rebuild_model++ if "max_seq_len" not in kwargs:+ kwargs["max_seq_len"] = 512+ models = export_rebuild_model(model=self, **kwargs)+ return models+</code></pre></details></div>+<div class="api-detail" id="e187"><span class="dtag badge-c">class</span><h2>Qwen3ASR</h2><div class="dmod">funasr.models.qwen3_asr · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/qwen3_asr/model.py#L16" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><ul>+<li><code>Qwen3</code> — ASR: Large Language Model based ASR supporting 52 languages.</li>+</ul>+<p>Wraps the qwen-asr package's Qwen3ASRModel for use within FunASR's AutoModel interface.</p>+<p>Supports auto language detection, contextual recognition, and optional forced alignment</p>+<p>for character-level timestamps.</p>+<h4>Requirements:</h4>+<p>pip install qwen-asr</p>+<h4>Models:</h4>+<ul>+<li>Qwen/Qwen3-ASR-0.6B (lighter, ~4GB GPU memory)</li>+<li>Qwen/Qwen3-ASR-1.7B (more accurate, ~8GB GPU memory)</li>+</ul>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code>class Qwen3ASR(nn.Module):+ """Qwen3-ASR: Large Language Model based ASR supporting 52 languages.++ Wraps the qwen-asr package's Qwen3ASRModel for use within FunASR's AutoModel interface.+ Supports auto language detection, contextual recognition, and optional forced alignment+ for character-level timestamps.++ Requirements:+ pip install qwen-asr++ Models:+ - Qwen/Qwen3-ASR-0.6B (lighter, ~4GB GPU memory)+ - Qwen/Qwen3-ASR-1.7B (more accurate, ~8GB GPU memory)+ """++ def __init__(self, **kwargs):+ """Initialize Qwen3ASR.++ Args:+ **kwargs: Additional keyword arguments.+ """+ super().__init__()+ model_path = kwargs.get("model_path", kwargs.get("model", "Qwen/Qwen3-ASR-1.7B"))+ device = kwargs.get("device", "cuda:0")+ dtype = kwargs.get("dtype", "bf16")+ hub = kwargs.get("hub", "ms")+ max_new_tokens = kwargs.get("max_new_tokens", 512)+ max_inference_batch_size = kwargs.get("max_inference_batch_size", 32)+ forced_aligner = kwargs.get("forced_aligner", None)+ forced_aligner_kwargs = kwargs.get("forced_aligner_kwargs", None)+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/qwen3_asr/model.py#L16" target="_blank" class="src-more">View full source on GitHub →</a></details><h3 class="mt">Methods</h3><div class="mblk"><code>.forward(**kwargs)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/qwen3_asr/model.py#L105" target="_blank" class="gh-link">L105</a><div class="mdoc"><p>Forward pass for training.</p>+<h4>Args:</h4>+<ul>+<li><code>**kwargs</code> — Additional keyword arguments.</li>+</ul>+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def forward(self, **kwargs):+ """Forward pass for training.++ Args:+ **kwargs: Additional keyword arguments.+ """+ raise NotImplementedError("Qwen3ASR only supports inference mode")+</code></pre></details></div><div class="mblk"><code>.inference(data_in, data_lengths, key, tokenizer, frontend, **kwargs)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/qwen3_asr/model.py#L113" target="_blank" class="gh-link">L113</a><div class="mdoc"><p>Run Qwen3-ASR speech recognition.</p>+<h4>Args:</h4>+<ul>+<li><code>data_in</code> — Audio input. Accepts:</li>+<li>list of file paths/URLs</li>+<li>list of (numpy_array, sample_rate) tuples</li>+<li>single numpy array or torch Tensor</li>+<li><code>data_lengths</code> — Not used.</li>+<li><code>key (list)</code> — Sample identifiers.</li>+<li><code>tokenizer</code> — Not used (Qwen3-ASR has internal tokenizer).</li>+<li><code>frontend</code> — Not used (Qwen3-ASR has internal audio processing).</li>+<li><code>**kwargs</code> — Runtime parameters:</li>+<li>language (str): Language hint (e.g. "Chinese", "English") or None for auto-detect.</li>+<li>return_time_stamps (bool): Return character-level timestamps (requires forced_aligner).</li>+<li>output_timestamp (bool): Same as return_time_stamps (for pipeline compatibility).</li>+<li>context (str): Context prompt for contextual recognition.</li>+</ul>+<h4>Returns:</h4>+<ul>+<li><code>tuple</code> — (results, meta_data) where results is list of dicts:</li>+<li>"key" (str): Sample ID</li>+<li>"text" (str): Recognized text (with punctuation)</li>+<li>"language" (str): Detected language (if available)</li>+<li>"timestamp" (list): [[start_ms, end_ms], ...] (if timestamps enabled)</li>+</ul>+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def inference(+ self,+ data_in,+ data_lengths=None,+ key: list = None,+ tokenizer=None,+ frontend=None,+ **kwargs,+ ):+ """Run Qwen3-ASR speech recognition.++ Args:+ data_in: Audio input. Accepts:+ - list of file paths/URLs+ - list of (numpy_array, sample_rate) tuples+ - single numpy array or torch Tensor+ data_lengths: Not used.+ key (list): Sample identifiers.+ tokenizer: Not used (Qwen3-ASR has internal tokenizer).+ frontend: Not used (Qwen3-ASR has internal audio processing).+ **kwargs: Runtime parameters:+ - language (str): Language hint (e.g. "Chinese", "English") or None for auto-detect.+ - return_time_stamps (bool): Return character-level timestamps (requires forced_aligner).+ - output_timestamp (bool): Same as return_time_stamps (for pipeline compatibility).+ - context (str): Context prompt for contextual recognition.++ Returns:+ tuple: (results, meta_data) where results is list of dicts:+ - "key" (str): Sample ID+ - "text" (str): Recognized text (with punctuation)+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/qwen3_asr/model.py#L113" target="_blank" class="src-more">View full source →</a></details></div></div>+<div class="api-detail" id="e188"><span class="dtag badge-m">method</span><h2><code>Qwen3ASR.forward(**kwargs)</code></h2><div class="dmod">funasr.models.qwen3_asr.Qwen3ASR · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/qwen3_asr/model.py#L105" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Forward pass for training.</p>+<h4>Args:</h4>+<ul>+<li><code>**kwargs</code> — Additional keyword arguments.</li>+</ul>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def forward(self, **kwargs):+ """Forward pass for training.++ Args:+ **kwargs: Additional keyword arguments.+ """+ raise NotImplementedError("Qwen3ASR only supports inference mode")+</code></pre></details></div>+<div class="api-detail" id="e189"><span class="dtag badge-m">method</span><h2><code>Qwen3ASR.inference(data_in, data_lengths, key, tokenizer, frontend, **kwargs)</code></h2><div class="dmod">funasr.models.qwen3_asr.Qwen3ASR · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/qwen3_asr/model.py#L113" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Run Qwen3-ASR speech recognition.</p>+<h4>Args:</h4>+<ul>+<li><code>data_in</code> — Audio input. Accepts:</li>+<li>list of file paths/URLs</li>+<li>list of (numpy_array, sample_rate) tuples</li>+<li>single numpy array or torch Tensor</li>+<li><code>data_lengths</code> — Not used.</li>+<li><code>key (list)</code> — Sample identifiers.</li>+<li><code>tokenizer</code> — Not used (Qwen3-ASR has internal tokenizer).</li>+<li><code>frontend</code> — Not used (Qwen3-ASR has internal audio processing).</li>+<li><code>**kwargs</code> — Runtime parameters:</li>+<li>language (str): Language hint (e.g. "Chinese", "English") or None for auto-detect.</li>+<li>return_time_stamps (bool): Return character-level timestamps (requires forced_aligner).</li>+<li>output_timestamp (bool): Same as return_time_stamps (for pipeline compatibility).</li>+<li>context (str): Context prompt for contextual recognition.</li>+</ul>+<h4>Returns:</h4>+<ul>+<li><code>tuple</code> — (results, meta_data) where results is list of dicts:</li>+<li>"key" (str): Sample ID</li>+<li>"text" (str): Recognized text (with punctuation)</li>+<li>"language" (str): Detected language (if available)</li>+<li>"timestamp" (list): [[start_ms, end_ms], ...] (if timestamps enabled)</li>+</ul>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def inference(+ self,+ data_in,+ data_lengths=None,+ key: list = None,+ tokenizer=None,+ frontend=None,+ **kwargs,+ ):+ """Run Qwen3-ASR speech recognition.++ Args:+ data_in: Audio input. Accepts:+ - list of file paths/URLs+ - list of (numpy_array, sample_rate) tuples+ - single numpy array or torch Tensor+ data_lengths: Not used.+ key (list): Sample identifiers.+ tokenizer: Not used (Qwen3-ASR has internal tokenizer).+ frontend: Not used (Qwen3-ASR has internal audio processing).+ **kwargs: Runtime parameters:+ - language (str): Language hint (e.g. "Chinese", "English") or None for auto-detect.+ - return_time_stamps (bool): Return character-level timestamps (requires forced_aligner).+ - output_timestamp (bool): Same as return_time_stamps (for pipeline compatibility).+ - context (str): Context prompt for contextual recognition.++ Returns:+ tuple: (results, meta_data) where results is list of dicts:+ - "key" (str): Sample ID+ - "text" (str): Recognized text (with punctuation)+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/qwen3_asr/model.py#L113" target="_blank" class="src-more">View full source on GitHub →</a></details></div>+<div class="api-detail" id="e190"><span class="dtag badge-c">class</span><h2>QwenAudioWarp</h2><div class="dmod">funasr.models.qwen_audio · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/qwen_audio/model.py#L22" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><ul>+<li><code>Qwen</code> — Audio: Advancing Universal Audio Understanding via Unified Large-Scale Audio-Language Models</li>+</ul>+<p>https://arxiv.org/abs/2311.07919</p>+<p>Modified from https://github.com/QwenLM/Qwen-Audio</p>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code>class QwenAudioWarp(nn.Module):+ """+ Qwen-Audio: Advancing Universal Audio Understanding via Unified Large-Scale Audio-Language Models+ https://arxiv.org/abs/2311.07919+ Modified from https://github.com/QwenLM/Qwen-Audio+ """++ def __init__(self, *args, **kwargs):+ """Initialize QwenAudioWarp.++ Args:+ *args: Variable positional arguments.+ **kwargs: Additional keyword arguments.+ """+ super().__init__()+ from transformers import AutoModelForCausalLM, AutoTokenizer+ from transformers.generation import GenerationConfig++ model_or_path = kwargs.get("model_path", "QwenAudio")+ model = AutoModelForCausalLM.from_pretrained(+ model_or_path, device_map="cpu", trust_remote_code=True+ )+ tokenizer = AutoTokenizer.from_pretrained(model_or_path, trust_remote_code=True)++ self.model = model+ self.tokenizer = tokenizer++ def forward(+ self,+ ):+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/qwen_audio/model.py#L22" target="_blank" class="src-more">View full source on GitHub →</a></details><h3 class="mt">Methods</h3><div class="mblk"><code>.forward()</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/qwen_audio/model.py#L49" target="_blank" class="gh-link">L49</a><div class="mdoc"><p>Forward pass for training.</p>+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def forward(+ self,+ ):+ """Forward pass for training."""+ pass+</code></pre></details></div><div class="mblk"><code>.inference(data_in, data_lengths, key, tokenizer, frontend, **kwargs)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/qwen_audio/model.py#L55" target="_blank" class="gh-link">L55</a><div class="mdoc"><p>Run inference on input data.</p>+<h4>Args:</h4>+<ul>+<li><code>data_in</code> — Input data (audio samples, file paths, or text).</li>+<li><code>data_lengths</code> — Lengths of each input sample in the batch.</li>+<li><code>key</code> — Sample identifiers.</li>+<li><code>tokenizer</code> — Tokenizer instance for text encoding/decoding.</li>+<li><code>frontend</code> — Audio frontend for feature extraction.</li>+<li><code>**kwargs</code> — Additional keyword arguments.</li>+</ul>+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def inference(+ self,+ data_in,+ data_lengths=None,+ key: list = None,+ tokenizer=None,+ frontend=None,+ **kwargs,+ ):+ """Run inference on input data.++ Args:+ data_in: Input data (audio samples, file paths, or text).+ data_lengths: Lengths of each input sample in the batch.+ key: Sample identifiers.+ tokenizer: Tokenizer instance for text encoding/decoding.+ frontend: Audio frontend for feature extraction.+ **kwargs: Additional keyword arguments.+ """+ if kwargs.get("batch_size", 1) > 1:+ raise NotImplementedError("batch decoding is not implemented")++ meta_data = {}+ # meta_data["batch_data_time"] = -1+ prompt = kwargs.get(+ "prompt", "<|startoftranscription|><|en|><|transcribe|><|en|><|notimestamps|><|wo_itn|>"+ )+ query = f"<audio>{data_in[0]}</audio>{prompt}"+ audio_info = self.tokenizer.process_audio(query)+ inputs = self.tokenizer(query, return_tensors="pt", audio_info=audio_info)+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/qwen_audio/model.py#L55" target="_blank" class="src-more">View full source →</a></details></div></div>+<div class="api-detail" id="e191"><span class="dtag badge-m">method</span><h2><code>QwenAudioWarp.forward()</code></h2><div class="dmod">funasr.models.qwen_audio.QwenAudioWarp · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/qwen_audio/model.py#L49" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Forward pass for training.</p>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def forward(+ self,+ ):+ """Forward pass for training."""+ pass+</code></pre></details></div>+<div class="api-detail" id="e192"><span class="dtag badge-m">method</span><h2><code>QwenAudioWarp.inference(data_in, data_lengths, key, tokenizer, frontend, **kwargs)</code></h2><div class="dmod">funasr.models.qwen_audio.QwenAudioWarp · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/qwen_audio/model.py#L55" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Run inference on input data.</p>+<h4>Args:</h4>+<ul>+<li><code>data_in</code> — Input data (audio samples, file paths, or text).</li>+<li><code>data_lengths</code> — Lengths of each input sample in the batch.</li>+<li><code>key</code> — Sample identifiers.</li>+<li><code>tokenizer</code> — Tokenizer instance for text encoding/decoding.</li>+<li><code>frontend</code> — Audio frontend for feature extraction.</li>+<li><code>**kwargs</code> — Additional keyword arguments.</li>+</ul>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def inference(+ self,+ data_in,+ data_lengths=None,+ key: list = None,+ tokenizer=None,+ frontend=None,+ **kwargs,+ ):+ """Run inference on input data.++ Args:+ data_in: Input data (audio samples, file paths, or text).+ data_lengths: Lengths of each input sample in the batch.+ key: Sample identifiers.+ tokenizer: Tokenizer instance for text encoding/decoding.+ frontend: Audio frontend for feature extraction.+ **kwargs: Additional keyword arguments.+ """+ if kwargs.get("batch_size", 1) > 1:+ raise NotImplementedError("batch decoding is not implemented")++ meta_data = {}+ # meta_data["batch_data_time"] = -1+ prompt = kwargs.get(+ "prompt", "<|startoftranscription|><|en|><|transcribe|><|en|><|notimestamps|><|wo_itn|>"+ )+ query = f"<audio>{data_in[0]}</audio>{prompt}"+ audio_info = self.tokenizer.process_audio(query)+ inputs = self.tokenizer(query, return_tensors="pt", audio_info=audio_info)+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/qwen_audio/model.py#L55" target="_blank" class="src-more">View full source on GitHub →</a></details></div>+<div class="api-detail" id="e193"><span class="dtag badge-c">class</span><h2>QwenAudioChatWarp</h2><div class="dmod">funasr.models.qwen_audio · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/qwen_audio/model.py#L104" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><ul>+<li><code>QwenAudioChat</code> — Qwen Audio Chat model wrapper.</li>+</ul>+<p>Interactive audio chat using the Qwen-Audio-Chat model.</p>+<p>Supports multi-turn conversation about audio content.</p>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code>class QwenAudioChatWarp(nn.Module):+ """QwenAudioChat: Qwen Audio Chat model wrapper.++ Interactive audio chat using the Qwen-Audio-Chat model.+ Supports multi-turn conversation about audio content.+ """++ def __init__(self, *args, **kwargs):+ """+ Qwen-Audio: Advancing Universal Audio Understanding via Unified Large-Scale Audio-Language Models+ https://arxiv.org/abs/2311.07919+ Modified from https://github.com/QwenLM/Qwen-Audio+ """+ super().__init__()+ from transformers import AutoModelForCausalLM, AutoTokenizer+ from transformers.generation import GenerationConfig++ model_or_path = kwargs.get("model_path", "QwenAudio")+ bf16 = kwargs.get("bf16", False)+ fp16 = kwargs.get("fp16", False)+ model = AutoModelForCausalLM.from_pretrained(+ model_or_path, device_map="cpu", bf16=bf16, fp16=fp16, trust_remote_code=True+ )+ tokenizer = AutoTokenizer.from_pretrained(model_or_path, trust_remote_code=True)++ self.model = model+ self.tokenizer = tokenizer++ def forward(+ self,+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/qwen_audio/model.py#L104" target="_blank" class="src-more">View full source on GitHub →</a></details><h3 class="mt">Methods</h3><div class="mblk"><code>.forward()</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/qwen_audio/model.py#L132" target="_blank" class="gh-link">L132</a><div class="mdoc"><p>Forward pass for training.</p>+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def forward(+ self,+ ):+ """Forward pass for training."""+ pass+</code></pre></details></div><div class="mblk"><code>.inference(data_in, data_lengths, key, tokenizer, frontend, **kwargs)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/qwen_audio/model.py#L138" target="_blank" class="gh-link">L138</a><div class="mdoc"><p>Run inference on input data.</p>+<h4>Args:</h4>+<ul>+<li><code>data_in</code> — Input data (audio samples, file paths, or text).</li>+<li><code>data_lengths</code> — Lengths of each input sample in the batch.</li>+<li><code>key</code> — Sample identifiers.</li>+<li><code>tokenizer</code> — Tokenizer instance for text encoding/decoding.</li>+<li><code>frontend</code> — Audio frontend for feature extraction.</li>+<li><code>**kwargs</code> — Additional keyword arguments.</li>+</ul>+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def inference(+ self,+ data_in,+ data_lengths=None,+ key: list = None,+ tokenizer=None,+ frontend=None,+ **kwargs,+ ):+ """Run inference on input data.++ Args:+ data_in: Input data (audio samples, file paths, or text).+ data_lengths: Lengths of each input sample in the batch.+ key: Sample identifiers.+ tokenizer: Tokenizer instance for text encoding/decoding.+ frontend: Audio frontend for feature extraction.+ **kwargs: Additional keyword arguments.+ """+ if kwargs.get("batch_size", 1) > 1:+ raise NotImplementedError("batch decoding is not implemented")++ meta_data = {}++ prompt = kwargs.get("prompt", "what does the person say?")+ cache = kwargs.get("cache", {})+ history = cache.get("history", None)+ if data_in[0] is not None:+ # 1st dialogue turn+ query = self.tokenizer.from_list_format(+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/qwen_audio/model.py#L138" target="_blank" class="src-more">View full source →</a></details></div></div>+<div class="api-detail" id="e194"><span class="dtag badge-m">method</span><h2><code>QwenAudioChatWarp.forward()</code></h2><div class="dmod">funasr.models.qwen_audio.QwenAudioChatWarp · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/qwen_audio/model.py#L132" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Forward pass for training.</p>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def forward(+ self,+ ):+ """Forward pass for training."""+ pass+</code></pre></details></div>+<div class="api-detail" id="e195"><span class="dtag badge-m">method</span><h2><code>QwenAudioChatWarp.inference(data_in, data_lengths, key, tokenizer, frontend, **kwargs)</code></h2><div class="dmod">funasr.models.qwen_audio.QwenAudioChatWarp · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/qwen_audio/model.py#L138" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Run inference on input data.</p>+<h4>Args:</h4>+<ul>+<li><code>data_in</code> — Input data (audio samples, file paths, or text).</li>+<li><code>data_lengths</code> — Lengths of each input sample in the batch.</li>+<li><code>key</code> — Sample identifiers.</li>+<li><code>tokenizer</code> — Tokenizer instance for text encoding/decoding.</li>+<li><code>frontend</code> — Audio frontend for feature extraction.</li>+<li><code>**kwargs</code> — Additional keyword arguments.</li>+</ul>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def inference(+ self,+ data_in,+ data_lengths=None,+ key: list = None,+ tokenizer=None,+ frontend=None,+ **kwargs,+ ):+ """Run inference on input data.++ Args:+ data_in: Input data (audio samples, file paths, or text).+ data_lengths: Lengths of each input sample in the batch.+ key: Sample identifiers.+ tokenizer: Tokenizer instance for text encoding/decoding.+ frontend: Audio frontend for feature extraction.+ **kwargs: Additional keyword arguments.+ """+ if kwargs.get("batch_size", 1) > 1:+ raise NotImplementedError("batch decoding is not implemented")++ meta_data = {}++ prompt = kwargs.get("prompt", "what does the person say?")+ cache = kwargs.get("cache", {})+ history = cache.get("history", None)+ if data_in[0] is not None:+ # 1st dialogue turn+ query = self.tokenizer.from_list_format(+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/qwen_audio/model.py#L138" target="_blank" class="src-more">View full source on GitHub →</a></details></div>+<div class="api-detail" id="e196"><span class="dtag badge-c">class</span><h2>SANM</h2><div class="dmod">funasr.models.sanm · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/sanm/model.py#L15" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><ul>+<li><code>Author</code> — Zhifu Gao, Shiliang Zhang, Ming Lei, Ian McLoughlin</li>+<li><code>San</code> — m: Memory equipped self-attention for end-to-end speech recognition</li>+</ul>+<p>https://arxiv.org/abs/2006.01713</p>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code>class SANM(Transformer):+ """+ Author: Zhifu Gao, Shiliang Zhang, Ming Lei, Ian McLoughlin+ San-m: Memory equipped self-attention for end-to-end speech recognition+ https://arxiv.org/abs/2006.01713+ """++ def __init__(+ self,+ *args,+ **kwargs,+ ):++ """Initialize SANM.++ Args:+ *args: Variable positional arguments.+ **kwargs: Additional keyword arguments.+ """+ super().__init__(*args, **kwargs)+</code></pre></details></div>+<div class="api-detail" id="e197"><span class="dtag badge-c">class</span><h2>SanmKWS</h2><div class="dmod">funasr.models.sanm_kws · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/sanm_kws/model.py#L28" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><ul>+<li><code>SANM</code> — KWS: Self-Attention Neural Memory based Keyword Spotting.</li>+</ul>+<p>Advanced keyword spotting using self-attention mechanism</p>+<p>for better context modeling of keyword patterns.</p>+<ul>+<li><code>Output</code> — {"key": str, "value": detected_keyword_info}</li>+</ul>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code>class SanmKWS(torch.nn.Module):+ """SANM-KWS: Self-Attention Neural Memory based Keyword Spotting.++ Advanced keyword spotting using self-attention mechanism+ for better context modeling of keyword patterns.++ Output: {"key": str, "value": detected_keyword_info}+ """++ def __init__(+ self,+ specaug: Optional[str] = None,+ specaug_conf: Optional[Dict] = None,+ normalize: str = None,+ normalize_conf: Optional[Dict] = None,+ encoder: str = None,+ encoder_conf: Optional[Dict] = None,+ ctc: str = None,+ ctc_conf: Optional[Dict] = None,+ ctc_weight: float = 1.0,+ input_size: int = 360,+ vocab_size: int = -1,+ ignore_id: int = -1,+ blank_id: int = 0,+ sos: int = 1,+ eos: int = 2,+ **kwargs,+ ):++ """Initialize SanmKWS.+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/sanm_kws/model.py#L28" target="_blank" class="src-more">View full source on GitHub →</a></details><h3 class="mt">Methods</h3><div class="mblk"><code>.forward(speech, speech_lengths, text, text_lengths, **kwargs)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/sanm_kws/model.py#L112" target="_blank" class="gh-link">L112</a><div class="mdoc"><p>Encoder + Decoder + Calc loss</p>+<h4>Args:</h4>+<ul>+<li><code>speech</code> — (Batch, Length, ...)</li>+<li><code>speech_lengths</code> — (Batch, )</li>+<li><code>text</code> — (Batch, Length)</li>+<li><code>text_lengths</code> — (Batch,)</li>+</ul>+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def forward(+ self,+ speech: torch.Tensor,+ speech_lengths: torch.Tensor,+ text: torch.Tensor,+ text_lengths: torch.Tensor,+ **kwargs,+ ) -> Tuple[torch.Tensor, Dict[str, torch.Tensor], torch.Tensor]:+ """Encoder + Decoder + Calc loss+ Args:+ speech: (Batch, Length, ...)+ speech_lengths: (Batch, )+ text: (Batch, Length)+ text_lengths: (Batch,)+ """+ if len(text_lengths.size()) > 1:+ text_lengths = text_lengths[:, 0]+ if len(speech_lengths.size()) > 1:+ speech_lengths = speech_lengths[:, 0]+ batch_size = speech.shape[0]++ # Encoder+ encoder_out, encoder_out_lens = self.encode(speech, speech_lengths)++ # decoder: CTC branch+ loss_ctc, cer_ctc = self._calc_ctc_loss(+ encoder_out, encoder_out_lens, text, text_lengths+ )++ # Collect CTC branch stats+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/sanm_kws/model.py#L112" target="_blank" class="src-more">View full source →</a></details></div><div class="mblk"><code>.encode(speech, speech_lengths, **kwargs)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/sanm_kws/model.py#L156" target="_blank" class="gh-link">L156</a><div class="mdoc"><p>Encoder. Note that this method is used by asr_inference.py</p>+<h4>Args:</h4>+<ul>+<li><code>speech</code> — (Batch, Length, ...)</li>+<li><code>speech_lengths</code> — (Batch, )</li>+<li><code>ind</code> — int</li>+</ul>+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def encode(+ self,+ speech: torch.Tensor,+ speech_lengths: torch.Tensor,+ **kwargs,+ ) -> Tuple[torch.Tensor, torch.Tensor]:+ """Encoder. Note that this method is used by asr_inference.py+ Args:+ speech: (Batch, Length, ...)+ speech_lengths: (Batch, )+ ind: int+ """+ with autocast(False):+ # Data augmentation+ if self.specaug is not None and self.training:+ speech, speech_lengths = self.specaug(speech, speech_lengths)++ # Normalization for feature: e.g. Global-CMVN, Utterance-CMVN+ if self.normalize is not None:+ speech, speech_lengths = self.normalize(speech, speech_lengths)++ # Forward encoder+ encoder_out, encoder_out_lens, _ = self.encoder(speech, speech_lengths)+ if isinstance(encoder_out, tuple):+ encoder_out = encoder_out[0]++ return encoder_out, encoder_out_lens+</code></pre></details></div><div class="mblk"><code>.inference(data_in, data_lengths, key, tokenizer, frontend, **kwargs)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/sanm_kws/model.py#L209" target="_blank" class="gh-link">L209</a><div class="mdoc"><p>Run inference on input data.</p>+<h4>Args:</h4>+<ul>+<li><code>data_in</code> — Input data (audio samples, file paths, or text).</li>+<li><code>data_lengths</code> — Lengths of each input sample in the batch.</li>+<li><code>key</code> — Sample identifiers.</li>+<li><code>tokenizer</code> — Tokenizer instance for text encoding/decoding.</li>+<li><code>frontend</code> — Audio frontend for feature extraction.</li>+<li><code>**kwargs</code> — Additional keyword arguments.</li>+</ul>+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def inference(+ self,+ data_in,+ data_lengths=None,+ key: list = None,+ tokenizer=None,+ frontend=None,+ **kwargs,+ ):+ """Run inference on input data.++ Args:+ data_in: Input data (audio samples, file paths, or text).+ data_lengths: Lengths of each input sample in the batch.+ key: Sample identifiers.+ tokenizer: Tokenizer instance for text encoding/decoding.+ frontend: Audio frontend for feature extraction.+ **kwargs: Additional keyword arguments.+ """+ keywords = kwargs.get("keywords")+ from funasr.utils.kws_utils import KwsCtcPrefixDecoder+ self.kws_decoder = KwsCtcPrefixDecoder(+ ctc=self.ctc,+ keywords=keywords,+ token_list=tokenizer.token_list,+ seg_dict=tokenizer.seg_dict,+ )++ meta_data = {}+ if (+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/sanm_kws/model.py#L209" target="_blank" class="src-more">View full source →</a></details></div><div class="mblk"><code>.export(**kwargs)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/sanm_kws/model.py#L300" target="_blank" class="gh-link">L300</a><div class="mdoc"><p>Export.</p>+<h4>Args:</h4>+<ul>+<li><code>**kwargs</code> — Additional keyword arguments.</li>+</ul>+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def export(self, **kwargs):+ """Export.++ Args:+ **kwargs: Additional keyword arguments.+ """+ from .export_meta import export_rebuild_model++ if "max_seq_len" not in kwargs:+ kwargs["max_seq_len"] = 512+ models = export_rebuild_model(model=self, **kwargs)+ return models+</code></pre></details></div></div>+<div class="api-detail" id="e198"><span class="dtag badge-m">method</span><h2><code>SanmKWS.forward(speech, speech_lengths, text, text_lengths, **kwargs)</code></h2><div class="dmod">funasr.models.sanm_kws.SanmKWS · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/sanm_kws/model.py#L112" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Encoder + Decoder + Calc loss</p>+<h4>Args:</h4>+<ul>+<li><code>speech</code> — (Batch, Length, ...)</li>+<li><code>speech_lengths</code> — (Batch, )</li>+<li><code>text</code> — (Batch, Length)</li>+<li><code>text_lengths</code> — (Batch,)</li>+</ul>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def forward(+ self,+ speech: torch.Tensor,+ speech_lengths: torch.Tensor,+ text: torch.Tensor,+ text_lengths: torch.Tensor,+ **kwargs,+ ) -> Tuple[torch.Tensor, Dict[str, torch.Tensor], torch.Tensor]:+ """Encoder + Decoder + Calc loss+ Args:+ speech: (Batch, Length, ...)+ speech_lengths: (Batch, )+ text: (Batch, Length)+ text_lengths: (Batch,)+ """+ if len(text_lengths.size()) > 1:+ text_lengths = text_lengths[:, 0]+ if len(speech_lengths.size()) > 1:+ speech_lengths = speech_lengths[:, 0]+ batch_size = speech.shape[0]++ # Encoder+ encoder_out, encoder_out_lens = self.encode(speech, speech_lengths)++ # decoder: CTC branch+ loss_ctc, cer_ctc = self._calc_ctc_loss(+ encoder_out, encoder_out_lens, text, text_lengths+ )++ # Collect CTC branch stats+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/sanm_kws/model.py#L112" target="_blank" class="src-more">View full source on GitHub →</a></details></div>+<div class="api-detail" id="e199"><span class="dtag badge-m">method</span><h2><code>SanmKWS.encode(speech, speech_lengths, **kwargs)</code></h2><div class="dmod">funasr.models.sanm_kws.SanmKWS · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/sanm_kws/model.py#L156" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Encoder. Note that this method is used by asr_inference.py</p>+<h4>Args:</h4>+<ul>+<li><code>speech</code> — (Batch, Length, ...)</li>+<li><code>speech_lengths</code> — (Batch, )</li>+<li><code>ind</code> — int</li>+</ul>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def encode(+ self,+ speech: torch.Tensor,+ speech_lengths: torch.Tensor,+ **kwargs,+ ) -> Tuple[torch.Tensor, torch.Tensor]:+ """Encoder. Note that this method is used by asr_inference.py+ Args:+ speech: (Batch, Length, ...)+ speech_lengths: (Batch, )+ ind: int+ """+ with autocast(False):+ # Data augmentation+ if self.specaug is not None and self.training:+ speech, speech_lengths = self.specaug(speech, speech_lengths)++ # Normalization for feature: e.g. Global-CMVN, Utterance-CMVN+ if self.normalize is not None:+ speech, speech_lengths = self.normalize(speech, speech_lengths)++ # Forward encoder+ encoder_out, encoder_out_lens, _ = self.encoder(speech, speech_lengths)+ if isinstance(encoder_out, tuple):+ encoder_out = encoder_out[0]++ return encoder_out, encoder_out_lens+</code></pre></details></div>+<div class="api-detail" id="e200"><span class="dtag badge-m">method</span><h2><code>SanmKWS.inference(data_in, data_lengths, key, tokenizer, frontend, **kwargs)</code></h2><div class="dmod">funasr.models.sanm_kws.SanmKWS · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/sanm_kws/model.py#L209" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Run inference on input data.</p>+<h4>Args:</h4>+<ul>+<li><code>data_in</code> — Input data (audio samples, file paths, or text).</li>+<li><code>data_lengths</code> — Lengths of each input sample in the batch.</li>+<li><code>key</code> — Sample identifiers.</li>+<li><code>tokenizer</code> — Tokenizer instance for text encoding/decoding.</li>+<li><code>frontend</code> — Audio frontend for feature extraction.</li>+<li><code>**kwargs</code> — Additional keyword arguments.</li>+</ul>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def inference(+ self,+ data_in,+ data_lengths=None,+ key: list = None,+ tokenizer=None,+ frontend=None,+ **kwargs,+ ):+ """Run inference on input data.++ Args:+ data_in: Input data (audio samples, file paths, or text).+ data_lengths: Lengths of each input sample in the batch.+ key: Sample identifiers.+ tokenizer: Tokenizer instance for text encoding/decoding.+ frontend: Audio frontend for feature extraction.+ **kwargs: Additional keyword arguments.+ """+ keywords = kwargs.get("keywords")+ from funasr.utils.kws_utils import KwsCtcPrefixDecoder+ self.kws_decoder = KwsCtcPrefixDecoder(+ ctc=self.ctc,+ keywords=keywords,+ token_list=tokenizer.token_list,+ seg_dict=tokenizer.seg_dict,+ )++ meta_data = {}+ if (+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/sanm_kws/model.py#L209" target="_blank" class="src-more">View full source on GitHub →</a></details></div>+<div class="api-detail" id="e201"><span class="dtag badge-m">method</span><h2><code>SanmKWS.export(**kwargs)</code></h2><div class="dmod">funasr.models.sanm_kws.SanmKWS · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/sanm_kws/model.py#L300" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Export.</p>+<h4>Args:</h4>+<ul>+<li><code>**kwargs</code> — Additional keyword arguments.</li>+</ul>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def export(self, **kwargs):+ """Export.++ Args:+ **kwargs: Additional keyword arguments.+ """+ from .export_meta import export_rebuild_model++ if "max_seq_len" not in kwargs:+ kwargs["max_seq_len"] = 512+ models = export_rebuild_model(model=self, **kwargs)+ return models+</code></pre></details></div>+<div class="api-detail" id="e202"><span class="dtag badge-c">class</span><h2>SanmKWSStreaming</h2><div class="dmod">funasr.models.sanm_kws_streaming · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/sanm_kws_streaming/model.py#L43" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><ul>+<li><code>Author</code> — Speech Lab of DAMO Academy, Alibaba Group</li>+<li><code>Paraformer</code> — Fast and Accurate Parallel Transformer for Non-autoregressive End-to-End Speech Recognition</li>+</ul>+<p>https://arxiv.org/abs/2206.08317</p>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code>class SanmKWSStreaming(SanmKWS):+ """+ Author: Speech Lab of DAMO Academy, Alibaba Group+ Paraformer: Fast and Accurate Parallel Transformer for Non-autoregressive End-to-End Speech Recognition+ https://arxiv.org/abs/2206.08317+ """++ def __init__(+ self,+ *args,+ **kwargs,+ ):+ """Initialize SanmKWSStreaming.++ Args:+ *args: Variable positional arguments.+ **kwargs: Additional keyword arguments.+ """+ super().__init__(*args, **kwargs)++ def forward(+ self,+ speech: torch.Tensor,+ speech_lengths: torch.Tensor,+ text: torch.Tensor,+ text_lengths: torch.Tensor,+ **kwargs,+ ) -> Tuple[torch.Tensor, Dict[str, torch.Tensor], torch.Tensor]:+ """Encoder + Decoder + Calc loss+ Args:+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/sanm_kws_streaming/model.py#L43" target="_blank" class="src-more">View full source on GitHub →</a></details><h3 class="mt">Methods</h3><div class="mblk"><code>.forward(speech, speech_lengths, text, text_lengths, **kwargs)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/sanm_kws_streaming/model.py#L63" target="_blank" class="gh-link">L63</a><div class="mdoc"><p>Encoder + Decoder + Calc loss</p>+<h4>Args:</h4>+<ul>+<li><code>speech</code> — (Batch, Length, ...)</li>+<li><code>speech_lengths</code> — (Batch, )</li>+<li><code>text</code> — (Batch, Length)</li>+<li><code>text_lengths</code> — (Batch,)</li>+</ul>+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def forward(+ self,+ speech: torch.Tensor,+ speech_lengths: torch.Tensor,+ text: torch.Tensor,+ text_lengths: torch.Tensor,+ **kwargs,+ ) -> Tuple[torch.Tensor, Dict[str, torch.Tensor], torch.Tensor]:+ """Encoder + Decoder + Calc loss+ Args:+ speech: (Batch, Length, ...)+ speech_lengths: (Batch, )+ text: (Batch, Length)+ text_lengths: (Batch,)+ """+ decoding_ind = kwargs.get("decoding_ind")+ if len(text_lengths.size()) > 1:+ text_lengths = text_lengths[:, 0]+ if len(speech_lengths.size()) > 1:+ speech_lengths = speech_lengths[:, 0]++ batch_size = speech.shape[0]++ # Encoder+ if hasattr(self.encoder, "overlap_chunk_cls"):+ ind = self.encoder.overlap_chunk_cls.random_choice(self.training, decoding_ind)+ encoder_out, encoder_out_lens = self.encode(speech, speech_lengths, ind=ind)+ else:+ encoder_out, encoder_out_lens = self.encode(speech, speech_lengths)++</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/sanm_kws_streaming/model.py#L63" target="_blank" class="src-more">View full source →</a></details></div><div class="mblk"><code>.encode_chunk(speech, speech_lengths, cache, **kwargs)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/sanm_kws_streaming/model.py#L119" target="_blank" class="gh-link">L119</a><div class="mdoc"><p>Encode chunk.</p>+<h4>Args:</h4>+<ul>+<li><code>speech</code> — Speech audio tensor, shape (batch, time).</li>+<li><code>speech_lengths</code> — Length of each speech sample.</li>+<li><code>cache</code> — State cache dict for streaming inference.</li>+<li><code>**kwargs</code> — Additional keyword arguments.</li>+</ul>+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def encode_chunk(+ self,+ speech: torch.Tensor,+ speech_lengths: torch.Tensor,+ cache: dict = None,+ **kwargs,+ ) -> Tuple[torch.Tensor, torch.Tensor]:+ """Encode chunk.++ Args:+ speech: Speech audio tensor, shape (batch, time).+ speech_lengths: Length of each speech sample.+ cache: State cache dict for streaming inference.+ **kwargs: Additional keyword arguments.+ """+ if cache is None:+ cache = {}+ """Frontend + Encoder. Note that this method is used by asr_inference.py+ Args:+ speech: (Batch, Length, ...)+ speech_lengths: (Batch, )+ ind: int+ """+ with autocast(False):+ # Data augmentation+ if self.specaug is not None and self.training:+ speech, speech_lengths = self.specaug(speech, speech_lengths)++ # Normalization for feature: e.g. Global-CMVN, Utterance-CMVN+ if self.normalize is not None:+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/sanm_kws_streaming/model.py#L119" target="_blank" class="src-more">View full source →</a></details></div><div class="mblk"><code>.init_cache(cache, **kwargs)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/sanm_kws_streaming/model.py#L161" target="_blank" class="gh-link">L161</a><div class="mdoc"><p>Init cache.</p>+<h4>Args:</h4>+<ul>+<li><code>cache</code> — State cache dict for streaming inference.</li>+<li><code>**kwargs</code> — Additional keyword arguments.</li>+</ul>+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def init_cache(self, cache: dict = None, **kwargs):+ """Init cache.++ Args:+ cache: State cache dict for streaming inference.+ **kwargs: Additional keyword arguments.+ """+ if cache is None:+ cache = {}+ chunk_size = kwargs.get("chunk_size", [0, 10, 5])+ encoder_chunk_look_back = kwargs.get("encoder_chunk_look_back", 0)+ decoder_chunk_look_back = kwargs.get("decoder_chunk_look_back", 0)+ batch_size = 1++ enc_output_size = kwargs["encoder_conf"]["output_size"]+ feats_dims = kwargs["frontend_conf"]["n_mels"] * kwargs["frontend_conf"]["lfr_m"]+ cache_encoder = {+ "start_idx": 0,+ "cif_hidden": torch.zeros((batch_size, 1, enc_output_size)),+ "cif_alphas": torch.zeros((batch_size, 1)),+ "encoder_out": None,+ "encoder_out_lens": None,+ "chunk_size": chunk_size,+ "encoder_chunk_look_back": encoder_chunk_look_back,+ "last_chunk": False,+ "opt": None,+ "feats": torch.zeros((batch_size, chunk_size[0] + chunk_size[2], feats_dims)),+ "tail_chunk": False,+ }+ cache["encoder"] = cache_encoder+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/sanm_kws_streaming/model.py#L161" target="_blank" class="src-more">View full source →</a></details></div><div class="mblk"><code>.generate_chunk(speech, speech_lengths, key, tokenizer, frontend, **kwargs)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/sanm_kws_streaming/model.py#L204" target="_blank" class="gh-link">L204</a><div class="mdoc"><p>Generate chunk.</p>+<h4>Args:</h4>+<ul>+<li><code>speech</code> — Speech audio tensor, shape (batch, time).</li>+<li><code>speech_lengths</code> — Length of each speech sample.</li>+<li><code>key</code> — Sample identifiers.</li>+<li><code>tokenizer</code> — Tokenizer instance for text encoding/decoding.</li>+<li><code>frontend</code> — Audio frontend for feature extraction.</li>+<li><code>**kwargs</code> — Additional keyword arguments.</li>+</ul>+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def generate_chunk(+ self,+ speech,+ speech_lengths=None,+ key: list = None,+ tokenizer=None,+ frontend=None,+ **kwargs,+ ):+ """Generate chunk.++ Args:+ speech: Speech audio tensor, shape (batch, time).+ speech_lengths: Length of each speech sample.+ key: Sample identifiers.+ tokenizer: Tokenizer instance for text encoding/decoding.+ frontend: Audio frontend for feature extraction.+ **kwargs: Additional keyword arguments.+ """+ cache = kwargs.get("cache", {})+ speech = speech.to(device=kwargs["device"])+ speech_lengths = speech_lengths.to(device=kwargs["device"])++ # Encoder+ is_final = kwargs.get("is_final", False)+ encoder_out, encoder_out_lens = self.encode_chunk(+ speech, speech_lengths, cache=cache, is_final=is_final+ )+ if isinstance(encoder_out, tuple):+ encoder_out = encoder_out[0]+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/sanm_kws_streaming/model.py#L204" target="_blank" class="src-more">View full source →</a></details></div><div class="mblk"><code>.inference(data_in, data_lengths, key, tokenizer, frontend, cache, **kwargs)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/sanm_kws_streaming/model.py#L287" target="_blank" class="gh-link">L287</a><div class="mdoc"><p>Run inference on input data.</p>+<h4>Args:</h4>+<ul>+<li><code>data_in</code> — Input data (audio samples, file paths, or text).</li>+<li><code>data_lengths</code> — Lengths of each input sample in the batch.</li>+<li><code>key</code> — Sample identifiers.</li>+<li><code>tokenizer</code> — Tokenizer instance for text encoding/decoding.</li>+<li><code>frontend</code> — Audio frontend for feature extraction.</li>+<li><code>cache</code> — State cache dict for streaming inference.</li>+<li><code>**kwargs</code> — Additional keyword arguments.</li>+</ul>+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def inference(+ self,+ data_in,+ data_lengths=None,+ key: list = None,+ tokenizer=None,+ frontend=None,+ cache: dict = None,+ **kwargs,+ ):+ """Run inference on input data.++ Args:+ data_in: Input data (audio samples, file paths, or text).+ data_lengths: Lengths of each input sample in the batch.+ key: Sample identifiers.+ tokenizer: Tokenizer instance for text encoding/decoding.+ frontend: Audio frontend for feature extraction.+ cache: State cache dict for streaming inference.+ **kwargs: Additional keyword arguments.+ """+ if cache is None:+ cache = {}+ keywords = kwargs.get("keywords")+ from funasr.utils.kws_utils import KwsCtcPrefixDecoder+ self.kws_decoder = KwsCtcPrefixDecoder(+ ctc=self.ctc,+ keywords=keywords,+ token_list=tokenizer.token_list,+ seg_dict=tokenizer.seg_dict,+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/sanm_kws_streaming/model.py#L287" target="_blank" class="src-more">View full source →</a></details></div><div class="mblk"><code>.export(**kwargs)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/sanm_kws_streaming/model.py#L490" target="_blank" class="gh-link">L490</a><div class="mdoc"><p>Export.</p>+<h4>Args:</h4>+<ul>+<li><code>**kwargs</code> — Additional keyword arguments.</li>+</ul>+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def export(self, **kwargs):+ """Export.++ Args:+ **kwargs: Additional keyword arguments.+ """+ from .export_meta import export_rebuild_model++ models = export_rebuild_model(model=self, **kwargs)+ return models+</code></pre></details></div></div>+<div class="api-detail" id="e203"><span class="dtag badge-m">method</span><h2><code>SanmKWSStreaming.forward(speech, speech_lengths, text, text_lengths, **kwargs)</code></h2><div class="dmod">funasr.models.sanm_kws_streaming.SanmKWSStreaming · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/sanm_kws_streaming/model.py#L63" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Encoder + Decoder + Calc loss</p>+<h4>Args:</h4>+<ul>+<li><code>speech</code> — (Batch, Length, ...)</li>+<li><code>speech_lengths</code> — (Batch, )</li>+<li><code>text</code> — (Batch, Length)</li>+<li><code>text_lengths</code> — (Batch,)</li>+</ul>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def forward(+ self,+ speech: torch.Tensor,+ speech_lengths: torch.Tensor,+ text: torch.Tensor,+ text_lengths: torch.Tensor,+ **kwargs,+ ) -> Tuple[torch.Tensor, Dict[str, torch.Tensor], torch.Tensor]:+ """Encoder + Decoder + Calc loss+ Args:+ speech: (Batch, Length, ...)+ speech_lengths: (Batch, )+ text: (Batch, Length)+ text_lengths: (Batch,)+ """+ decoding_ind = kwargs.get("decoding_ind")+ if len(text_lengths.size()) > 1:+ text_lengths = text_lengths[:, 0]+ if len(speech_lengths.size()) > 1:+ speech_lengths = speech_lengths[:, 0]++ batch_size = speech.shape[0]++ # Encoder+ if hasattr(self.encoder, "overlap_chunk_cls"):+ ind = self.encoder.overlap_chunk_cls.random_choice(self.training, decoding_ind)+ encoder_out, encoder_out_lens = self.encode(speech, speech_lengths, ind=ind)+ else:+ encoder_out, encoder_out_lens = self.encode(speech, speech_lengths)++</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/sanm_kws_streaming/model.py#L63" target="_blank" class="src-more">View full source on GitHub →</a></details></div>+<div class="api-detail" id="e204"><span class="dtag badge-m">method</span><h2><code>SanmKWSStreaming.encode_chunk(speech, speech_lengths, cache, **kwargs)</code></h2><div class="dmod">funasr.models.sanm_kws_streaming.SanmKWSStreaming · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/sanm_kws_streaming/model.py#L119" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Encode chunk.</p>+<h4>Args:</h4>+<ul>+<li><code>speech</code> — Speech audio tensor, shape (batch, time).</li>+<li><code>speech_lengths</code> — Length of each speech sample.</li>+<li><code>cache</code> — State cache dict for streaming inference.</li>+<li><code>**kwargs</code> — Additional keyword arguments.</li>+</ul>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def encode_chunk(+ self,+ speech: torch.Tensor,+ speech_lengths: torch.Tensor,+ cache: dict = None,+ **kwargs,+ ) -> Tuple[torch.Tensor, torch.Tensor]:+ """Encode chunk.++ Args:+ speech: Speech audio tensor, shape (batch, time).+ speech_lengths: Length of each speech sample.+ cache: State cache dict for streaming inference.+ **kwargs: Additional keyword arguments.+ """+ if cache is None:+ cache = {}+ """Frontend + Encoder. Note that this method is used by asr_inference.py+ Args:+ speech: (Batch, Length, ...)+ speech_lengths: (Batch, )+ ind: int+ """+ with autocast(False):+ # Data augmentation+ if self.specaug is not None and self.training:+ speech, speech_lengths = self.specaug(speech, speech_lengths)++ # Normalization for feature: e.g. Global-CMVN, Utterance-CMVN+ if self.normalize is not None:+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/sanm_kws_streaming/model.py#L119" target="_blank" class="src-more">View full source on GitHub →</a></details></div>+<div class="api-detail" id="e205"><span class="dtag badge-m">method</span><h2><code>SanmKWSStreaming.init_cache(cache, **kwargs)</code></h2><div class="dmod">funasr.models.sanm_kws_streaming.SanmKWSStreaming · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/sanm_kws_streaming/model.py#L161" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Init cache.</p>+<h4>Args:</h4>+<ul>+<li><code>cache</code> — State cache dict for streaming inference.</li>+<li><code>**kwargs</code> — Additional keyword arguments.</li>+</ul>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def init_cache(self, cache: dict = None, **kwargs):+ """Init cache.++ Args:+ cache: State cache dict for streaming inference.+ **kwargs: Additional keyword arguments.+ """+ if cache is None:+ cache = {}+ chunk_size = kwargs.get("chunk_size", [0, 10, 5])+ encoder_chunk_look_back = kwargs.get("encoder_chunk_look_back", 0)+ decoder_chunk_look_back = kwargs.get("decoder_chunk_look_back", 0)+ batch_size = 1++ enc_output_size = kwargs["encoder_conf"]["output_size"]+ feats_dims = kwargs["frontend_conf"]["n_mels"] * kwargs["frontend_conf"]["lfr_m"]+ cache_encoder = {+ "start_idx": 0,+ "cif_hidden": torch.zeros((batch_size, 1, enc_output_size)),+ "cif_alphas": torch.zeros((batch_size, 1)),+ "encoder_out": None,+ "encoder_out_lens": None,+ "chunk_size": chunk_size,+ "encoder_chunk_look_back": encoder_chunk_look_back,+ "last_chunk": False,+ "opt": None,+ "feats": torch.zeros((batch_size, chunk_size[0] + chunk_size[2], feats_dims)),+ "tail_chunk": False,+ }+ cache["encoder"] = cache_encoder+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/sanm_kws_streaming/model.py#L161" target="_blank" class="src-more">View full source on GitHub →</a></details></div>+<div class="api-detail" id="e206"><span class="dtag badge-m">method</span><h2><code>SanmKWSStreaming.generate_chunk(speech, speech_lengths, key, tokenizer, frontend, **kwargs)</code></h2><div class="dmod">funasr.models.sanm_kws_streaming.SanmKWSStreaming · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/sanm_kws_streaming/model.py#L204" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Generate chunk.</p>+<h4>Args:</h4>+<ul>+<li><code>speech</code> — Speech audio tensor, shape (batch, time).</li>+<li><code>speech_lengths</code> — Length of each speech sample.</li>+<li><code>key</code> — Sample identifiers.</li>+<li><code>tokenizer</code> — Tokenizer instance for text encoding/decoding.</li>+<li><code>frontend</code> — Audio frontend for feature extraction.</li>+<li><code>**kwargs</code> — Additional keyword arguments.</li>+</ul>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def generate_chunk(+ self,+ speech,+ speech_lengths=None,+ key: list = None,+ tokenizer=None,+ frontend=None,+ **kwargs,+ ):+ """Generate chunk.++ Args:+ speech: Speech audio tensor, shape (batch, time).+ speech_lengths: Length of each speech sample.+ key: Sample identifiers.+ tokenizer: Tokenizer instance for text encoding/decoding.+ frontend: Audio frontend for feature extraction.+ **kwargs: Additional keyword arguments.+ """+ cache = kwargs.get("cache", {})+ speech = speech.to(device=kwargs["device"])+ speech_lengths = speech_lengths.to(device=kwargs["device"])++ # Encoder+ is_final = kwargs.get("is_final", False)+ encoder_out, encoder_out_lens = self.encode_chunk(+ speech, speech_lengths, cache=cache, is_final=is_final+ )+ if isinstance(encoder_out, tuple):+ encoder_out = encoder_out[0]+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/sanm_kws_streaming/model.py#L204" target="_blank" class="src-more">View full source on GitHub →</a></details></div>+<div class="api-detail" id="e207"><span class="dtag badge-m">method</span><h2><code>SanmKWSStreaming.inference(data_in, data_lengths, key, tokenizer, frontend, cache, **kwargs)</code></h2><div class="dmod">funasr.models.sanm_kws_streaming.SanmKWSStreaming · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/sanm_kws_streaming/model.py#L287" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Run inference on input data.</p>+<h4>Args:</h4>+<ul>+<li><code>data_in</code> — Input data (audio samples, file paths, or text).</li>+<li><code>data_lengths</code> — Lengths of each input sample in the batch.</li>+<li><code>key</code> — Sample identifiers.</li>+<li><code>tokenizer</code> — Tokenizer instance for text encoding/decoding.</li>+<li><code>frontend</code> — Audio frontend for feature extraction.</li>+<li><code>cache</code> — State cache dict for streaming inference.</li>+<li><code>**kwargs</code> — Additional keyword arguments.</li>+</ul>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def inference(+ self,+ data_in,+ data_lengths=None,+ key: list = None,+ tokenizer=None,+ frontend=None,+ cache: dict = None,+ **kwargs,+ ):+ """Run inference on input data.++ Args:+ data_in: Input data (audio samples, file paths, or text).+ data_lengths: Lengths of each input sample in the batch.+ key: Sample identifiers.+ tokenizer: Tokenizer instance for text encoding/decoding.+ frontend: Audio frontend for feature extraction.+ cache: State cache dict for streaming inference.+ **kwargs: Additional keyword arguments.+ """+ if cache is None:+ cache = {}+ keywords = kwargs.get("keywords")+ from funasr.utils.kws_utils import KwsCtcPrefixDecoder+ self.kws_decoder = KwsCtcPrefixDecoder(+ ctc=self.ctc,+ keywords=keywords,+ token_list=tokenizer.token_list,+ seg_dict=tokenizer.seg_dict,+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/sanm_kws_streaming/model.py#L287" target="_blank" class="src-more">View full source on GitHub →</a></details></div>+<div class="api-detail" id="e208"><span class="dtag badge-m">method</span><h2><code>SanmKWSStreaming.export(**kwargs)</code></h2><div class="dmod">funasr.models.sanm_kws_streaming.SanmKWSStreaming · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/sanm_kws_streaming/model.py#L490" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Export.</p>+<h4>Args:</h4>+<ul>+<li><code>**kwargs</code> — Additional keyword arguments.</li>+</ul>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def export(self, **kwargs):+ """Export.++ Args:+ **kwargs: Additional keyword arguments.+ """+ from .export_meta import export_rebuild_model++ models = export_rebuild_model(model=self, **kwargs)+ return models+</code></pre></details></div>+<div class="api-detail" id="e209"><span class="dtag badge-c">class</span><h2>SCAMA</h2><div class="dmod">funasr.models.scama · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/scama/model.py#L45" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><ul>+<li><code>SCAMA</code> — Streaming Chunk-Aware Multi-head Attention ASR.</li>+</ul>+<p>Streaming ASR using chunk-based encoder with configurable latency.</p>+<p>Supports 2-pass decoding for accuracy refinement.</p>+<ul>+<li><code>Output</code> — {"key": str, "text": str}</li>+<li><code>Author</code> — Shiliang Zhang, Zhifu Gao, Haoneng Luo, Ming Lei, Jie Gao, Zhijie Yan, Lei Xie</li>+<li><code>SCAMA</code> — Streaming chunk-aware multihead attention for online end-to-end speech recognition</li>+</ul>+<p>https://arxiv.org/abs/2006.01712</p>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code>class SCAMA(nn.Module):+ """SCAMA: Streaming Chunk-Aware Multi-head Attention ASR.++ Streaming ASR using chunk-based encoder with configurable latency.+ Supports 2-pass decoding for accuracy refinement.++ Output: {"key": str, "text": str}++ Author: Shiliang Zhang, Zhifu Gao, Haoneng Luo, Ming Lei, Jie Gao, Zhijie Yan, Lei Xie+ SCAMA: Streaming chunk-aware multihead attention for online end-to-end speech recognition+ https://arxiv.org/abs/2006.01712+ """++ def __init__(+ self,+ specaug: str = None,+ specaug_conf: dict = None,+ normalize: str = None,+ normalize_conf: dict = None,+ encoder: str = None,+ encoder_conf: dict = None,+ decoder: str = None,+ decoder_conf: dict = None,+ ctc: str = None,+ ctc_conf: dict = None,+ ctc_weight: float = 0.5,+ predictor: str = None,+ predictor_conf: dict = None,+ predictor_bias: int = 0,+ predictor_weight: float = 0.0,+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/scama/model.py#L45" target="_blank" class="src-more">View full source on GitHub →</a></details><h3 class="mt">Methods</h3><div class="mblk"><code>.forward(speech, speech_lengths, text, text_lengths, **kwargs)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/scama/model.py#L200" target="_blank" class="gh-link">L200</a><div class="mdoc"><p>Encoder + Decoder + Calc loss</p>+<h4>Args:</h4>+<ul>+<li><code>speech</code> — (Batch, Length, ...)</li>+<li><code>speech_lengths</code> — (Batch, )</li>+<li><code>text</code> — (Batch, Length)</li>+<li><code>text_lengths</code> — (Batch,)</li>+</ul>+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def forward(+ self,+ speech: torch.Tensor,+ speech_lengths: torch.Tensor,+ text: torch.Tensor,+ text_lengths: torch.Tensor,+ **kwargs,+ ) -> Tuple[torch.Tensor, Dict[str, torch.Tensor], torch.Tensor]:+ """Encoder + Decoder + Calc loss+ Args:+ speech: (Batch, Length, ...)+ speech_lengths: (Batch, )+ text: (Batch, Length)+ text_lengths: (Batch,)+ """++ decoding_ind = kwargs.get("decoding_ind")+ if len(text_lengths.size()) > 1:+ text_lengths = text_lengths[:, 0]+ if len(speech_lengths.size()) > 1:+ speech_lengths = speech_lengths[:, 0]++ batch_size = speech.shape[0]++ # Encoder+ ind = self.encoder.overlap_chunk_cls.random_choice(self.training, decoding_ind)+ encoder_out, encoder_out_lens = self.encode(speech, speech_lengths, ind=ind)++ loss_ctc, cer_ctc = None, None+ loss_pre = None+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/scama/model.py#L200" target="_blank" class="src-more">View full source →</a></details></div><div class="mblk"><code>.encode(speech, speech_lengths, **kwargs)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/scama/model.py#L277" target="_blank" class="gh-link">L277</a><div class="mdoc"><p>Encoder. Note that this method is used by asr_inference.py</p>+<h4>Args:</h4>+<ul>+<li><code>speech</code> — (Batch, Length, ...)</li>+<li><code>speech_lengths</code> — (Batch, )</li>+<li><code>ind</code> — int</li>+</ul>+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def encode(+ self,+ speech: torch.Tensor,+ speech_lengths: torch.Tensor,+ **kwargs,+ ) -> Tuple[torch.Tensor, torch.Tensor]:+ """Encoder. Note that this method is used by asr_inference.py+ Args:+ speech: (Batch, Length, ...)+ speech_lengths: (Batch, )+ ind: int+ """+ with autocast(False):++ # Data augmentation+ if self.specaug is not None and self.training:+ speech, speech_lengths = self.specaug(speech, speech_lengths)++ # Normalization for feature: e.g. Global-CMVN, Utterance-CMVN+ if self.normalize is not None:+ speech, speech_lengths = self.normalize(speech, speech_lengths)++ # Forward encoder+ encoder_out, encoder_out_lens, _ = self.encoder(speech, speech_lengths)+ if isinstance(encoder_out, tuple):+ encoder_out = encoder_out[0]++ return encoder_out, encoder_out_lens+</code></pre></details></div><div class="mblk"><code>.encode_chunk(speech, speech_lengths, cache, **kwargs)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/scama/model.py#L306" target="_blank" class="gh-link">L306</a><div class="mdoc"><p>Encode chunk.</p>+<h4>Args:</h4>+<ul>+<li><code>speech</code> — Speech audio tensor, shape (batch, time).</li>+<li><code>speech_lengths</code> — Length of each speech sample.</li>+<li><code>cache</code> — State cache dict for streaming inference.</li>+<li><code>**kwargs</code> — Additional keyword arguments.</li>+</ul>+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def encode_chunk(+ self,+ speech: torch.Tensor,+ speech_lengths: torch.Tensor,+ cache: dict = None,+ **kwargs,+ ) -> Tuple[torch.Tensor, torch.Tensor]:+ """Encode chunk.++ Args:+ speech: Speech audio tensor, shape (batch, time).+ speech_lengths: Length of each speech sample.+ cache: State cache dict for streaming inference.+ **kwargs: Additional keyword arguments.+ """+ if cache is None:+ cache = {}+ """Frontend + Encoder. Note that this method is used by asr_inference.py+ Args:+ speech: (Batch, Length, ...)+ speech_lengths: (Batch, )+ ind: int+ """+ with autocast(False):++ # Data augmentation+ if self.specaug is not None and self.training:+ speech, speech_lengths = self.specaug(speech, speech_lengths)++ # Normalization for feature: e.g. Global-CMVN, Utterance-CMVN+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/scama/model.py#L306" target="_blank" class="src-more">View full source →</a></details></div><div class="mblk"><code>.calc_predictor_chunk(encoder_out, encoder_out_lens, cache, **kwargs)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/scama/model.py#L348" target="_blank" class="gh-link">L348</a><div class="mdoc"><p>Calc predictor chunk.</p>+<h4>Args:</h4>+<ul>+<li><code>encoder_out</code> — Encoder output tensor.</li>+<li><code>encoder_out_lens</code> — Encoder output lengths.</li>+<li><code>cache</code> — State cache dict for streaming inference.</li>+<li><code>**kwargs</code> — Additional keyword arguments.</li>+</ul>+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def calc_predictor_chunk(self, encoder_out, encoder_out_lens, cache=None, **kwargs):+ """Calc predictor chunk.++ Args:+ encoder_out: Encoder output tensor.+ encoder_out_lens: Encoder output lengths.+ cache: State cache dict for streaming inference.+ **kwargs: Additional keyword arguments.+ """+ is_final = kwargs.get("is_final", False)++ return self.predictor.forward_chunk(encoder_out, cache["encoder"], is_final=is_final)+</code></pre></details></div><div class="mblk"><code>.calc_predictor_mask(encoder_out, encoder_out_lens, ys_pad, ys_pad_lens)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/scama/model.py#L462" target="_blank" class="gh-link">L462</a><div class="mdoc"><p>Calc predictor mask.</p>+<h4>Args:</h4>+<ul>+<li><code>encoder_out</code> — Encoder output tensor.</li>+<li><code>encoder_out_lens</code> — Encoder output lengths.</li>+<li><code>ys_pad</code> — TODO.</li>+<li><code>ys_pad_lens</code> — Lengths of ys_pad.</li>+</ul>+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def calc_predictor_mask(+ self,+ encoder_out: torch.Tensor,+ encoder_out_lens: torch.Tensor,+ ys_pad: torch.Tensor = None,+ ys_pad_lens: torch.Tensor = None,+ ):+ # ys_in_pad, ys_out_pad = add_sos_eos(ys_pad, self.sos, self.eos, self.ignore_id)+ # ys_in_lens = ys_pad_lens + 1+ """Calc predictor mask.++ Args:+ encoder_out: Encoder output tensor.+ encoder_out_lens: Encoder output lengths.+ ys_pad: TODO.+ ys_pad_lens: Lengths of ys_pad.+ """+ ys_out_pad, ys_in_lens = None, None++ encoder_out_mask = sequence_mask(+ encoder_out_lens,+ maxlen=encoder_out.size(1),+ dtype=encoder_out.dtype,+ device=encoder_out.device,+ )[:, None, :]+ mask_chunk_predictor = None++ mask_chunk_predictor = self.encoder.overlap_chunk_cls.get_mask_chunk_predictor(+ None, device=encoder_out.device, batch_size=encoder_out.size(0)+ )+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/scama/model.py#L462" target="_blank" class="src-more">View full source →</a></details></div><div class="mblk"><code>.init_beam_search(**kwargs)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/scama/model.py#L543" target="_blank" class="gh-link">L543</a><div class="mdoc"><p>Init beam search.</p>+<h4>Args:</h4>+<ul>+<li><code>**kwargs</code> — Additional keyword arguments.</li>+</ul>+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def init_beam_search(+ self,+ **kwargs,+ ):++ """Init beam search.++ Args:+ **kwargs: Additional keyword arguments.+ """+ from funasr.models.scama.beam_search import BeamSearchScamaStreaming++ from funasr.models.transformer.scorers.ctc import CTCPrefixScorer+ from funasr.models.transformer.scorers.length_bonus import LengthBonus++ # 1. Build ASR model+ scorers = {}++ if self.ctc != None:+ ctc = CTCPrefixScorer(ctc=self.ctc, eos=self.eos)+ scorers.update(ctc=ctc)+ token_list = kwargs.get("token_list")+ scorers.update(+ decoder=self.decoder,+ length_bonus=LengthBonus(len(token_list)),+ )++ # 3. Build ngram model+ # ngram is not supported now+ ngram = None+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/scama/model.py#L543" target="_blank" class="src-more">View full source →</a></details></div><div class="mblk"><code>.generate_chunk(speech, speech_lengths, key, tokenizer, frontend, **kwargs)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/scama/model.py#L599" target="_blank" class="gh-link">L599</a><div class="mdoc"><p>Generate chunk.</p>+<h4>Args:</h4>+<ul>+<li><code>speech</code> — Speech audio tensor, shape (batch, time).</li>+<li><code>speech_lengths</code> — Length of each speech sample.</li>+<li><code>key</code> — Sample identifiers.</li>+<li><code>tokenizer</code> — Tokenizer instance for text encoding/decoding.</li>+<li><code>frontend</code> — Audio frontend for feature extraction.</li>+<li><code>**kwargs</code> — Additional keyword arguments.</li>+</ul>+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def generate_chunk(+ self,+ speech,+ speech_lengths=None,+ key: list = None,+ tokenizer=None,+ frontend=None,+ **kwargs,+ ):+ """Generate chunk.++ Args:+ speech: Speech audio tensor, shape (batch, time).+ speech_lengths: Length of each speech sample.+ key: Sample identifiers.+ tokenizer: Tokenizer instance for text encoding/decoding.+ frontend: Audio frontend for feature extraction.+ **kwargs: Additional keyword arguments.+ """+ cache = kwargs.get("cache", {})+ speech = speech.to(device=kwargs["device"])+ speech_lengths = speech_lengths.to(device=kwargs["device"])++ # Encoder+ encoder_out, encoder_out_lens = self.encode_chunk(+ speech, speech_lengths, cache=cache, is_final=kwargs.get("is_final", False)+ )+ if isinstance(encoder_out, tuple):+ encoder_out = encoder_out[0]+ if "running_hyps" not in cache:+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/scama/model.py#L599" target="_blank" class="src-more">View full source →</a></details></div><div class="mblk"><code>.init_cache(cache, **kwargs)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/scama/model.py#L694" target="_blank" class="gh-link">L694</a><div class="mdoc"><p>Init cache.</p>+<h4>Args:</h4>+<ul>+<li><code>cache</code> — State cache dict for streaming inference.</li>+<li><code>**kwargs</code> — Additional keyword arguments.</li>+</ul>+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def init_cache(self, cache: dict = None, **kwargs):+ """Init cache.++ Args:+ cache: State cache dict for streaming inference.+ **kwargs: Additional keyword arguments.+ """+ if cache is None:+ cache = {}+ device = kwargs.get("device", "cuda")++ chunk_size = kwargs.get("chunk_size", [0, 10, 5])+ encoder_chunk_look_back = kwargs.get("encoder_chunk_look_back", 0)+ decoder_chunk_look_back = kwargs.get("decoder_chunk_look_back", 0)+ batch_size = 1++ enc_output_size = kwargs["encoder_conf"]["output_size"]+ feats_dims = kwargs["frontend_conf"]["n_mels"] * kwargs["frontend_conf"]["lfr_m"]++ cache_encoder = {+ "start_idx": 0,+ "cif_hidden": torch.zeros((batch_size, 1, enc_output_size)).to(device=device),+ "cif_alphas": torch.zeros((batch_size, 1)).to(device=device),+ "chunk_size": chunk_size,+ "encoder_chunk_look_back": encoder_chunk_look_back,+ "last_chunk": False,+ "opt": None,+ "feats": torch.zeros((batch_size, chunk_size[0] + chunk_size[2], feats_dims)).to(+ device=device+ ),+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/scama/model.py#L694" target="_blank" class="src-more">View full source →</a></details></div><div class="mblk"><code>.inference(data_in, data_lengths, key, tokenizer, frontend, cache, **kwargs)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/scama/model.py#L741" target="_blank" class="gh-link">L741</a><div class="mdoc"><p>Run inference on input data.</p>+<h4>Args:</h4>+<ul>+<li><code>data_in</code> — Input data (audio samples, file paths, or text).</li>+<li><code>data_lengths</code> — Lengths of each input sample in the batch.</li>+<li><code>key</code> — Sample identifiers.</li>+<li><code>tokenizer</code> — Tokenizer instance for text encoding/decoding.</li>+<li><code>frontend</code> — Audio frontend for feature extraction.</li>+<li><code>cache</code> — State cache dict for streaming inference.</li>+<li><code>**kwargs</code> — Additional keyword arguments.</li>+</ul>+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def inference(+ self,+ data_in,+ data_lengths=None,+ key: list = None,+ tokenizer=None,+ frontend=None,+ cache: dict = None,+ **kwargs,+ ):+ """Run inference on input data.++ Args:+ data_in: Input data (audio samples, file paths, or text).+ data_lengths: Lengths of each input sample in the batch.+ key: Sample identifiers.+ tokenizer: Tokenizer instance for text encoding/decoding.+ frontend: Audio frontend for feature extraction.+ cache: State cache dict for streaming inference.+ **kwargs: Additional keyword arguments.+ """+ if cache is None:+ cache = {}++ # init beamsearch+ is_use_ctc = kwargs.get("decoding_ctc_weight", 0.0) > 0.00001 and self.ctc != None+ is_use_lm = (+ kwargs.get("lm_weight", 0.0) > 0.00001 and kwargs.get("lm_file", None) is not None+ )++</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/scama/model.py#L741" target="_blank" class="src-more">View full source →</a></details></div></div>+<div class="api-detail" id="e210"><span class="dtag badge-m">method</span><h2><code>SCAMA.forward(speech, speech_lengths, text, text_lengths, **kwargs)</code></h2><div class="dmod">funasr.models.scama.SCAMA · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/scama/model.py#L200" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Encoder + Decoder + Calc loss</p>+<h4>Args:</h4>+<ul>+<li><code>speech</code> — (Batch, Length, ...)</li>+<li><code>speech_lengths</code> — (Batch, )</li>+<li><code>text</code> — (Batch, Length)</li>+<li><code>text_lengths</code> — (Batch,)</li>+</ul>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def forward(+ self,+ speech: torch.Tensor,+ speech_lengths: torch.Tensor,+ text: torch.Tensor,+ text_lengths: torch.Tensor,+ **kwargs,+ ) -> Tuple[torch.Tensor, Dict[str, torch.Tensor], torch.Tensor]:+ """Encoder + Decoder + Calc loss+ Args:+ speech: (Batch, Length, ...)+ speech_lengths: (Batch, )+ text: (Batch, Length)+ text_lengths: (Batch,)+ """++ decoding_ind = kwargs.get("decoding_ind")+ if len(text_lengths.size()) > 1:+ text_lengths = text_lengths[:, 0]+ if len(speech_lengths.size()) > 1:+ speech_lengths = speech_lengths[:, 0]++ batch_size = speech.shape[0]++ # Encoder+ ind = self.encoder.overlap_chunk_cls.random_choice(self.training, decoding_ind)+ encoder_out, encoder_out_lens = self.encode(speech, speech_lengths, ind=ind)++ loss_ctc, cer_ctc = None, None+ loss_pre = None+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/scama/model.py#L200" target="_blank" class="src-more">View full source on GitHub →</a></details></div>+<div class="api-detail" id="e211"><span class="dtag badge-m">method</span><h2><code>SCAMA.encode(speech, speech_lengths, **kwargs)</code></h2><div class="dmod">funasr.models.scama.SCAMA · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/scama/model.py#L277" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Encoder. Note that this method is used by asr_inference.py</p>+<h4>Args:</h4>+<ul>+<li><code>speech</code> — (Batch, Length, ...)</li>+<li><code>speech_lengths</code> — (Batch, )</li>+<li><code>ind</code> — int</li>+</ul>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def encode(+ self,+ speech: torch.Tensor,+ speech_lengths: torch.Tensor,+ **kwargs,+ ) -> Tuple[torch.Tensor, torch.Tensor]:+ """Encoder. Note that this method is used by asr_inference.py+ Args:+ speech: (Batch, Length, ...)+ speech_lengths: (Batch, )+ ind: int+ """+ with autocast(False):++ # Data augmentation+ if self.specaug is not None and self.training:+ speech, speech_lengths = self.specaug(speech, speech_lengths)++ # Normalization for feature: e.g. Global-CMVN, Utterance-CMVN+ if self.normalize is not None:+ speech, speech_lengths = self.normalize(speech, speech_lengths)++ # Forward encoder+ encoder_out, encoder_out_lens, _ = self.encoder(speech, speech_lengths)+ if isinstance(encoder_out, tuple):+ encoder_out = encoder_out[0]++ return encoder_out, encoder_out_lens+</code></pre></details></div>+<div class="api-detail" id="e212"><span class="dtag badge-m">method</span><h2><code>SCAMA.encode_chunk(speech, speech_lengths, cache, **kwargs)</code></h2><div class="dmod">funasr.models.scama.SCAMA · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/scama/model.py#L306" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Encode chunk.</p>+<h4>Args:</h4>+<ul>+<li><code>speech</code> — Speech audio tensor, shape (batch, time).</li>+<li><code>speech_lengths</code> — Length of each speech sample.</li>+<li><code>cache</code> — State cache dict for streaming inference.</li>+<li><code>**kwargs</code> — Additional keyword arguments.</li>+</ul>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def encode_chunk(+ self,+ speech: torch.Tensor,+ speech_lengths: torch.Tensor,+ cache: dict = None,+ **kwargs,+ ) -> Tuple[torch.Tensor, torch.Tensor]:+ """Encode chunk.++ Args:+ speech: Speech audio tensor, shape (batch, time).+ speech_lengths: Length of each speech sample.+ cache: State cache dict for streaming inference.+ **kwargs: Additional keyword arguments.+ """+ if cache is None:+ cache = {}+ """Frontend + Encoder. Note that this method is used by asr_inference.py+ Args:+ speech: (Batch, Length, ...)+ speech_lengths: (Batch, )+ ind: int+ """+ with autocast(False):++ # Data augmentation+ if self.specaug is not None and self.training:+ speech, speech_lengths = self.specaug(speech, speech_lengths)++ # Normalization for feature: e.g. Global-CMVN, Utterance-CMVN+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/scama/model.py#L306" target="_blank" class="src-more">View full source on GitHub →</a></details></div>+<div class="api-detail" id="e213"><span class="dtag badge-m">method</span><h2><code>SCAMA.calc_predictor_chunk(encoder_out, encoder_out_lens, cache, **kwargs)</code></h2><div class="dmod">funasr.models.scama.SCAMA · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/scama/model.py#L348" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Calc predictor chunk.</p>+<h4>Args:</h4>+<ul>+<li><code>encoder_out</code> — Encoder output tensor.</li>+<li><code>encoder_out_lens</code> — Encoder output lengths.</li>+<li><code>cache</code> — State cache dict for streaming inference.</li>+<li><code>**kwargs</code> — Additional keyword arguments.</li>+</ul>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def calc_predictor_chunk(self, encoder_out, encoder_out_lens, cache=None, **kwargs):+ """Calc predictor chunk.++ Args:+ encoder_out: Encoder output tensor.+ encoder_out_lens: Encoder output lengths.+ cache: State cache dict for streaming inference.+ **kwargs: Additional keyword arguments.+ """+ is_final = kwargs.get("is_final", False)++ return self.predictor.forward_chunk(encoder_out, cache["encoder"], is_final=is_final)+</code></pre></details></div>+<div class="api-detail" id="e214"><span class="dtag badge-m">method</span><h2><code>SCAMA.calc_predictor_mask(encoder_out, encoder_out_lens, ys_pad, ys_pad_lens)</code></h2><div class="dmod">funasr.models.scama.SCAMA · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/scama/model.py#L462" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Calc predictor mask.</p>+<h4>Args:</h4>+<ul>+<li><code>encoder_out</code> — Encoder output tensor.</li>+<li><code>encoder_out_lens</code> — Encoder output lengths.</li>+<li><code>ys_pad</code> — TODO.</li>+<li><code>ys_pad_lens</code> — Lengths of ys_pad.</li>+</ul>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def calc_predictor_mask(+ self,+ encoder_out: torch.Tensor,+ encoder_out_lens: torch.Tensor,+ ys_pad: torch.Tensor = None,+ ys_pad_lens: torch.Tensor = None,+ ):+ # ys_in_pad, ys_out_pad = add_sos_eos(ys_pad, self.sos, self.eos, self.ignore_id)+ # ys_in_lens = ys_pad_lens + 1+ """Calc predictor mask.++ Args:+ encoder_out: Encoder output tensor.+ encoder_out_lens: Encoder output lengths.+ ys_pad: TODO.+ ys_pad_lens: Lengths of ys_pad.+ """+ ys_out_pad, ys_in_lens = None, None++ encoder_out_mask = sequence_mask(+ encoder_out_lens,+ maxlen=encoder_out.size(1),+ dtype=encoder_out.dtype,+ device=encoder_out.device,+ )[:, None, :]+ mask_chunk_predictor = None++ mask_chunk_predictor = self.encoder.overlap_chunk_cls.get_mask_chunk_predictor(+ None, device=encoder_out.device, batch_size=encoder_out.size(0)+ )+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/scama/model.py#L462" target="_blank" class="src-more">View full source on GitHub →</a></details></div>+<div class="api-detail" id="e215"><span class="dtag badge-m">method</span><h2><code>SCAMA.init_beam_search(**kwargs)</code></h2><div class="dmod">funasr.models.scama.SCAMA · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/scama/model.py#L543" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Init beam search.</p>+<h4>Args:</h4>+<ul>+<li><code>**kwargs</code> — Additional keyword arguments.</li>+</ul>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def init_beam_search(+ self,+ **kwargs,+ ):++ """Init beam search.++ Args:+ **kwargs: Additional keyword arguments.+ """+ from funasr.models.scama.beam_search import BeamSearchScamaStreaming++ from funasr.models.transformer.scorers.ctc import CTCPrefixScorer+ from funasr.models.transformer.scorers.length_bonus import LengthBonus++ # 1. Build ASR model+ scorers = {}++ if self.ctc != None:+ ctc = CTCPrefixScorer(ctc=self.ctc, eos=self.eos)+ scorers.update(ctc=ctc)+ token_list = kwargs.get("token_list")+ scorers.update(+ decoder=self.decoder,+ length_bonus=LengthBonus(len(token_list)),+ )++ # 3. Build ngram model+ # ngram is not supported now+ ngram = None+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/scama/model.py#L543" target="_blank" class="src-more">View full source on GitHub →</a></details></div>+<div class="api-detail" id="e216"><span class="dtag badge-m">method</span><h2><code>SCAMA.generate_chunk(speech, speech_lengths, key, tokenizer, frontend, **kwargs)</code></h2><div class="dmod">funasr.models.scama.SCAMA · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/scama/model.py#L599" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Generate chunk.</p>+<h4>Args:</h4>+<ul>+<li><code>speech</code> — Speech audio tensor, shape (batch, time).</li>+<li><code>speech_lengths</code> — Length of each speech sample.</li>+<li><code>key</code> — Sample identifiers.</li>+<li><code>tokenizer</code> — Tokenizer instance for text encoding/decoding.</li>+<li><code>frontend</code> — Audio frontend for feature extraction.</li>+<li><code>**kwargs</code> — Additional keyword arguments.</li>+</ul>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def generate_chunk(+ self,+ speech,+ speech_lengths=None,+ key: list = None,+ tokenizer=None,+ frontend=None,+ **kwargs,+ ):+ """Generate chunk.++ Args:+ speech: Speech audio tensor, shape (batch, time).+ speech_lengths: Length of each speech sample.+ key: Sample identifiers.+ tokenizer: Tokenizer instance for text encoding/decoding.+ frontend: Audio frontend for feature extraction.+ **kwargs: Additional keyword arguments.+ """+ cache = kwargs.get("cache", {})+ speech = speech.to(device=kwargs["device"])+ speech_lengths = speech_lengths.to(device=kwargs["device"])++ # Encoder+ encoder_out, encoder_out_lens = self.encode_chunk(+ speech, speech_lengths, cache=cache, is_final=kwargs.get("is_final", False)+ )+ if isinstance(encoder_out, tuple):+ encoder_out = encoder_out[0]+ if "running_hyps" not in cache:+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/scama/model.py#L599" target="_blank" class="src-more">View full source on GitHub →</a></details></div>+<div class="api-detail" id="e217"><span class="dtag badge-m">method</span><h2><code>SCAMA.init_cache(cache, **kwargs)</code></h2><div class="dmod">funasr.models.scama.SCAMA · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/scama/model.py#L694" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Init cache.</p>+<h4>Args:</h4>+<ul>+<li><code>cache</code> — State cache dict for streaming inference.</li>+<li><code>**kwargs</code> — Additional keyword arguments.</li>+</ul>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def init_cache(self, cache: dict = None, **kwargs):+ """Init cache.++ Args:+ cache: State cache dict for streaming inference.+ **kwargs: Additional keyword arguments.+ """+ if cache is None:+ cache = {}+ device = kwargs.get("device", "cuda")++ chunk_size = kwargs.get("chunk_size", [0, 10, 5])+ encoder_chunk_look_back = kwargs.get("encoder_chunk_look_back", 0)+ decoder_chunk_look_back = kwargs.get("decoder_chunk_look_back", 0)+ batch_size = 1++ enc_output_size = kwargs["encoder_conf"]["output_size"]+ feats_dims = kwargs["frontend_conf"]["n_mels"] * kwargs["frontend_conf"]["lfr_m"]++ cache_encoder = {+ "start_idx": 0,+ "cif_hidden": torch.zeros((batch_size, 1, enc_output_size)).to(device=device),+ "cif_alphas": torch.zeros((batch_size, 1)).to(device=device),+ "chunk_size": chunk_size,+ "encoder_chunk_look_back": encoder_chunk_look_back,+ "last_chunk": False,+ "opt": None,+ "feats": torch.zeros((batch_size, chunk_size[0] + chunk_size[2], feats_dims)).to(+ device=device+ ),+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/scama/model.py#L694" target="_blank" class="src-more">View full source on GitHub →</a></details></div>+<div class="api-detail" id="e218"><span class="dtag badge-m">method</span><h2><code>SCAMA.inference(data_in, data_lengths, key, tokenizer, frontend, cache, **kwargs)</code></h2><div class="dmod">funasr.models.scama.SCAMA · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/scama/model.py#L741" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Run inference on input data.</p>+<h4>Args:</h4>+<ul>+<li><code>data_in</code> — Input data (audio samples, file paths, or text).</li>+<li><code>data_lengths</code> — Lengths of each input sample in the batch.</li>+<li><code>key</code> — Sample identifiers.</li>+<li><code>tokenizer</code> — Tokenizer instance for text encoding/decoding.</li>+<li><code>frontend</code> — Audio frontend for feature extraction.</li>+<li><code>cache</code> — State cache dict for streaming inference.</li>+<li><code>**kwargs</code> — Additional keyword arguments.</li>+</ul>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def inference(+ self,+ data_in,+ data_lengths=None,+ key: list = None,+ tokenizer=None,+ frontend=None,+ cache: dict = None,+ **kwargs,+ ):+ """Run inference on input data.++ Args:+ data_in: Input data (audio samples, file paths, or text).+ data_lengths: Lengths of each input sample in the batch.+ key: Sample identifiers.+ tokenizer: Tokenizer instance for text encoding/decoding.+ frontend: Audio frontend for feature extraction.+ cache: State cache dict for streaming inference.+ **kwargs: Additional keyword arguments.+ """+ if cache is None:+ cache = {}++ # init beamsearch+ is_use_ctc = kwargs.get("decoding_ctc_weight", 0.0) > 0.00001 and self.ctc != None+ is_use_lm = (+ kwargs.get("lm_weight", 0.0) > 0.00001 and kwargs.get("lm_file", None) is not None+ )++</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/scama/model.py#L741" target="_blank" class="src-more">View full source on GitHub →</a></details></div>+<div class="api-detail" id="e219"><span class="dtag badge-c">class</span><h2>SeacoParaformer</h2><div class="dmod">funasr.models.seaco_paraformer · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/seaco_paraformer/model.py#L50" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><ul>+<li><code>SeACo</code> — Paraformer: Semantic-Aware Contextual Paraformer.</li>+</ul>+<p>The recommended Chinese ASR model. Combines Paraformer's non-autoregressive</p>+<p>architecture with semantic context biasing for hotword recognition.</p>+<p>Registered as 'paraformer-zh' alias.</p>+<ul>+<li><code>Output</code> — {"key": str, "text": str, "timestamp": [[start_ms, end_ms], ...]}</li>+</ul>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code>class SeacoParaformer(BiCifParaformer, Paraformer):+ """SeACo-Paraformer: Semantic-Aware Contextual Paraformer.++ The recommended Chinese ASR model. Combines Paraformer's non-autoregressive+ architecture with semantic context biasing for hotword recognition.++ Registered as 'paraformer-zh' alias.+ Output: {"key": str, "text": str, "timestamp": [[start_ms, end_ms], ...]}+ """++ def __init__(+ self,+ *args,+ **kwargs,+ ):+ """Initialize SeacoParaformer.++ Args:+ *args: Variable positional arguments.+ **kwargs: Additional keyword arguments.+ """+ super().__init__(*args, **kwargs)++ self.inner_dim = kwargs.get("inner_dim", 256)+ self.bias_encoder_type = kwargs.get("bias_encoder_type", "lstm")+ bias_encoder_dropout_rate = kwargs.get("bias_encoder_dropout_rate", 0.0)+ bias_encoder_bid = kwargs.get("bias_encoder_bid", False)+ seaco_lsm_weight = kwargs.get("seaco_lsm_weight", 0.0)+ seaco_length_normalized_loss = kwargs.get("seaco_length_normalized_loss", True)++</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/seaco_paraformer/model.py#L50" target="_blank" class="src-more">View full source on GitHub →</a></details><h3 class="mt">Methods</h3><div class="mblk"><code>.forward(speech, speech_lengths, text, text_lengths, **kwargs)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/seaco_paraformer/model.py#L122" target="_blank" class="gh-link">L122</a><div class="mdoc"><p>Frontend + Encoder + Decoder + Calc loss</p>+<h4>Args:</h4>+<ul>+<li><code>speech</code> — (Batch, Length, ...)</li>+<li><code>speech_lengths</code> — (Batch, )</li>+<li><code>text</code> — (Batch, Length)</li>+<li><code>text_lengths</code> — (Batch,)</li>+</ul>+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def forward(+ self,+ speech: torch.Tensor,+ speech_lengths: torch.Tensor,+ text: torch.Tensor,+ text_lengths: torch.Tensor,+ **kwargs,+ ) -> Tuple[torch.Tensor, Dict[str, torch.Tensor], torch.Tensor]:+ """Frontend + Encoder + Decoder + Calc loss++ Args:+ speech: (Batch, Length, ...)+ speech_lengths: (Batch, )+ text: (Batch, Length)+ text_lengths: (Batch,)+ """+ if len(text_lengths.size()) > 1:+ text_lengths = text_lengths[:, 0]+ if len(speech_lengths.size()) > 1:+ speech_lengths = speech_lengths[:, 0]+ # Check that batch_size is unified+ assert (+ speech.shape[0] == speech_lengths.shape[0] == text.shape[0] == text_lengths.shape[0]+ ), (speech.shape, speech_lengths.shape, text.shape, text_lengths.shape)++ hotword_pad = kwargs.get("hotword_pad")+ hotword_lengths = kwargs.get("hotword_lengths")+ seaco_label_pad = kwargs.get("seaco_label_pad")+ if len(hotword_lengths.size()) > 1:+ hotword_lengths = hotword_lengths[:, 0]+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/seaco_paraformer/model.py#L122" target="_blank" class="src-more">View full source →</a></details></div><div class="mblk"><code>.calc_predictor(encoder_out, encoder_out_lens)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/seaco_paraformer/model.py#L202" target="_blank" class="gh-link">L202</a><div class="mdoc"><p>Calc predictor.</p>+<h4>Args:</h4>+<ul>+<li><code>encoder_out</code> — Encoder output tensor.</li>+<li><code>encoder_out_lens</code> — Encoder output lengths.</li>+</ul>+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def calc_predictor(self, encoder_out, encoder_out_lens):+ """Calc predictor.++ Args:+ encoder_out: Encoder output tensor.+ encoder_out_lens: Encoder output lengths.+ """+ encoder_out_mask = (+ ~make_pad_mask(encoder_out_lens, maxlen=encoder_out.size(1))[:, None, :]+ ).to(encoder_out.device)+ predictor_outs = self.predictor(+ encoder_out, None, encoder_out_mask, ignore_id=self.ignore_id+ )+ return predictor_outs[:4]+</code></pre></details></div><div class="mblk"><code>.inference(data_in, data_lengths, key, tokenizer, frontend, **kwargs)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/seaco_paraformer/model.py#L422" target="_blank" class="gh-link">L422</a><div class="mdoc"><p>Run inference on input data.</p>+<h4>Args:</h4>+<ul>+<li><code>data_in</code> — Input data (audio samples, file paths, or text).</li>+<li><code>data_lengths</code> — Lengths of each input sample in the batch.</li>+<li><code>key</code> — Sample identifiers.</li>+<li><code>tokenizer</code> — Tokenizer instance for text encoding/decoding.</li>+<li><code>frontend</code> — Audio frontend for feature extraction.</li>+<li><code>**kwargs</code> — Additional keyword arguments.</li>+</ul>+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def inference(+ self,+ data_in,+ data_lengths=None,+ key: list = None,+ tokenizer=None,+ frontend=None,+ **kwargs,+ ):++ # init beamsearch+ """Run inference on input data.++ Args:+ data_in: Input data (audio samples, file paths, or text).+ data_lengths: Lengths of each input sample in the batch.+ key: Sample identifiers.+ tokenizer: Tokenizer instance for text encoding/decoding.+ frontend: Audio frontend for feature extraction.+ **kwargs: Additional keyword arguments.+ """+ is_use_ctc = kwargs.get("decoding_ctc_weight", 0.0) > 0.00001 and self.ctc != None+ is_use_lm = (+ kwargs.get("lm_weight", 0.0) > 0.00001 and kwargs.get("lm_file", None) is not None+ )+ if self.beam_search is None and (is_use_lm or is_use_ctc):+ logging.info("enable beam_search")+ self.init_beam_search(**kwargs)+ self.nbest = kwargs.get("nbest", 1)+ meta_data = {}+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/seaco_paraformer/model.py#L422" target="_blank" class="src-more">View full source →</a></details></div><div class="mblk"><code>.generate_hotwords_list(hotword_list_or_file, tokenizer, frontend)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/seaco_paraformer/model.py#L583" target="_blank" class="gh-link">L583</a><div class="mdoc"><p>Generate hotwords list.</p>+<h4>Args:</h4>+<ul>+<li><code>hotword_list_or_file</code> — TODO.</li>+<li><code>tokenizer</code> — Tokenizer instance for text encoding/decoding.</li>+<li><code>frontend</code> — Audio frontend for feature extraction.</li>+</ul>+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def generate_hotwords_list(self, hotword_list_or_file, tokenizer=None, frontend=None):++ """Generate hotwords list.++ Args:+ hotword_list_or_file: TODO.+ tokenizer: Tokenizer instance for text encoding/decoding.+ frontend: Audio frontend for feature extraction.+ """+ def seg_tokenize(txt, seg_dict):+ """Seg tokenize.++ Args:+ txt: TODO.+ seg_dict: TODO.+ """+ pattern = re.compile(r"^[\u4E00-\u9FA50-9]+$")+ out_txt = ""+ for word in txt:+ word = word.lower()+ if word in seg_dict:+ out_txt += seg_dict[word] + " "+ else:+ if pattern.match(word):+ for char in word:+ if char in seg_dict:+ out_txt += seg_dict[char] + " "+ else:+ out_txt += "<unk>" + " "+ else:+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/seaco_paraformer/model.py#L583" target="_blank" class="src-more">View full source →</a></details></div><div class="mblk"><code>.export(**kwargs)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/seaco_paraformer/model.py#L692" target="_blank" class="gh-link">L692</a><div class="mdoc"><p>Export.</p>+<h4>Args:</h4>+<ul>+<li><code>**kwargs</code> — Additional keyword arguments.</li>+</ul>+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def export(+ self,+ **kwargs,+ ):+ """Export.++ Args:+ **kwargs: Additional keyword arguments.+ """+ if "max_seq_len" not in kwargs:+ kwargs["max_seq_len"] = 512+ from .export_meta import export_rebuild_model++ models = export_rebuild_model(model=self, **kwargs)+ return models+</code></pre></details></div></div>+<div class="api-detail" id="e220"><span class="dtag badge-m">method</span><h2><code>SeacoParaformer.forward(speech, speech_lengths, text, text_lengths, **kwargs)</code></h2><div class="dmod">funasr.models.seaco_paraformer.SeacoParaformer · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/seaco_paraformer/model.py#L122" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Frontend + Encoder + Decoder + Calc loss</p>+<h4>Args:</h4>+<ul>+<li><code>speech</code> — (Batch, Length, ...)</li>+<li><code>speech_lengths</code> — (Batch, )</li>+<li><code>text</code> — (Batch, Length)</li>+<li><code>text_lengths</code> — (Batch,)</li>+</ul>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def forward(+ self,+ speech: torch.Tensor,+ speech_lengths: torch.Tensor,+ text: torch.Tensor,+ text_lengths: torch.Tensor,+ **kwargs,+ ) -> Tuple[torch.Tensor, Dict[str, torch.Tensor], torch.Tensor]:+ """Frontend + Encoder + Decoder + Calc loss++ Args:+ speech: (Batch, Length, ...)+ speech_lengths: (Batch, )+ text: (Batch, Length)+ text_lengths: (Batch,)+ """+ if len(text_lengths.size()) > 1:+ text_lengths = text_lengths[:, 0]+ if len(speech_lengths.size()) > 1:+ speech_lengths = speech_lengths[:, 0]+ # Check that batch_size is unified+ assert (+ speech.shape[0] == speech_lengths.shape[0] == text.shape[0] == text_lengths.shape[0]+ ), (speech.shape, speech_lengths.shape, text.shape, text_lengths.shape)++ hotword_pad = kwargs.get("hotword_pad")+ hotword_lengths = kwargs.get("hotword_lengths")+ seaco_label_pad = kwargs.get("seaco_label_pad")+ if len(hotword_lengths.size()) > 1:+ hotword_lengths = hotword_lengths[:, 0]+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/seaco_paraformer/model.py#L122" target="_blank" class="src-more">View full source on GitHub →</a></details></div>+<div class="api-detail" id="e221"><span class="dtag badge-m">method</span><h2><code>SeacoParaformer.calc_predictor(encoder_out, encoder_out_lens)</code></h2><div class="dmod">funasr.models.seaco_paraformer.SeacoParaformer · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/seaco_paraformer/model.py#L202" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Calc predictor.</p>+<h4>Args:</h4>+<ul>+<li><code>encoder_out</code> — Encoder output tensor.</li>+<li><code>encoder_out_lens</code> — Encoder output lengths.</li>+</ul>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def calc_predictor(self, encoder_out, encoder_out_lens):+ """Calc predictor.++ Args:+ encoder_out: Encoder output tensor.+ encoder_out_lens: Encoder output lengths.+ """+ encoder_out_mask = (+ ~make_pad_mask(encoder_out_lens, maxlen=encoder_out.size(1))[:, None, :]+ ).to(encoder_out.device)+ predictor_outs = self.predictor(+ encoder_out, None, encoder_out_mask, ignore_id=self.ignore_id+ )+ return predictor_outs[:4]+</code></pre></details></div>+<div class="api-detail" id="e222"><span class="dtag badge-m">method</span><h2><code>SeacoParaformer.inference(data_in, data_lengths, key, tokenizer, frontend, **kwargs)</code></h2><div class="dmod">funasr.models.seaco_paraformer.SeacoParaformer · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/seaco_paraformer/model.py#L422" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Run inference on input data.</p>+<h4>Args:</h4>+<ul>+<li><code>data_in</code> — Input data (audio samples, file paths, or text).</li>+<li><code>data_lengths</code> — Lengths of each input sample in the batch.</li>+<li><code>key</code> — Sample identifiers.</li>+<li><code>tokenizer</code> — Tokenizer instance for text encoding/decoding.</li>+<li><code>frontend</code> — Audio frontend for feature extraction.</li>+<li><code>**kwargs</code> — Additional keyword arguments.</li>+</ul>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def inference(+ self,+ data_in,+ data_lengths=None,+ key: list = None,+ tokenizer=None,+ frontend=None,+ **kwargs,+ ):++ # init beamsearch+ """Run inference on input data.++ Args:+ data_in: Input data (audio samples, file paths, or text).+ data_lengths: Lengths of each input sample in the batch.+ key: Sample identifiers.+ tokenizer: Tokenizer instance for text encoding/decoding.+ frontend: Audio frontend for feature extraction.+ **kwargs: Additional keyword arguments.+ """+ is_use_ctc = kwargs.get("decoding_ctc_weight", 0.0) > 0.00001 and self.ctc != None+ is_use_lm = (+ kwargs.get("lm_weight", 0.0) > 0.00001 and kwargs.get("lm_file", None) is not None+ )+ if self.beam_search is None and (is_use_lm or is_use_ctc):+ logging.info("enable beam_search")+ self.init_beam_search(**kwargs)+ self.nbest = kwargs.get("nbest", 1)+ meta_data = {}+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/seaco_paraformer/model.py#L422" target="_blank" class="src-more">View full source on GitHub →</a></details></div>+<div class="api-detail" id="e223"><span class="dtag badge-m">method</span><h2><code>SeacoParaformer.generate_hotwords_list(hotword_list_or_file, tokenizer, frontend)</code></h2><div class="dmod">funasr.models.seaco_paraformer.SeacoParaformer · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/seaco_paraformer/model.py#L583" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Generate hotwords list.</p>+<h4>Args:</h4>+<ul>+<li><code>hotword_list_or_file</code> — TODO.</li>+<li><code>tokenizer</code> — Tokenizer instance for text encoding/decoding.</li>+<li><code>frontend</code> — Audio frontend for feature extraction.</li>+</ul>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def generate_hotwords_list(self, hotword_list_or_file, tokenizer=None, frontend=None):++ """Generate hotwords list.++ Args:+ hotword_list_or_file: TODO.+ tokenizer: Tokenizer instance for text encoding/decoding.+ frontend: Audio frontend for feature extraction.+ """+ def seg_tokenize(txt, seg_dict):+ """Seg tokenize.++ Args:+ txt: TODO.+ seg_dict: TODO.+ """+ pattern = re.compile(r"^[\u4E00-\u9FA50-9]+$")+ out_txt = ""+ for word in txt:+ word = word.lower()+ if word in seg_dict:+ out_txt += seg_dict[word] + " "+ else:+ if pattern.match(word):+ for char in word:+ if char in seg_dict:+ out_txt += seg_dict[char] + " "+ else:+ out_txt += "<unk>" + " "+ else:+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/seaco_paraformer/model.py#L583" target="_blank" class="src-more">View full source on GitHub →</a></details></div>+<div class="api-detail" id="e224"><span class="dtag badge-m">method</span><h2><code>SeacoParaformer.export(**kwargs)</code></h2><div class="dmod">funasr.models.seaco_paraformer.SeacoParaformer · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/seaco_paraformer/model.py#L692" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Export.</p>+<h4>Args:</h4>+<ul>+<li><code>**kwargs</code> — Additional keyword arguments.</li>+</ul>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def export(+ self,+ **kwargs,+ ):+ """Export.++ Args:+ **kwargs: Additional keyword arguments.+ """+ if "max_seq_len" not in kwargs:+ kwargs["max_seq_len"] = 512+ from .export_meta import export_rebuild_model++ models = export_rebuild_model(model=self, **kwargs)+ return models+</code></pre></details></div>+<div class="api-detail" id="e225"><span class="dtag badge-f">function</span><h2><code>load_seg_dict(seg_dict_file)</code></h2><div class="dmod">funasr.models.seaco_paraformer · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/seaco_paraformer/model.py#L710" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Load seg dict.</p>+<h4>Args:</h4>+<ul>+<li><code>seg_dict_file</code> — TODO.</li>+</ul>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code>def load_seg_dict(seg_dict_file):+ """Load seg dict.++ Args:+ seg_dict_file: TODO.+ """+ seg_dict = {}+ assert isinstance(seg_dict_file, str)+ with open(seg_dict_file, "r", encoding="utf8") as f:+ lines = f.readlines()+ for line in lines:+ s = line.strip().split()+ key = s[0]+ value = s[1:]+ seg_dict[key] = " ".join(value)+ return seg_dict</code></pre></details></div>+<div class="api-detail" id="e226"><span class="dtag badge-f">function</span><h2><code>sequence_mask(lengths, maxlen, dtype, device)</code></h2><div class="dmod">funasr.models.sense_voice · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/sense_voice/model.py#L326" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Sequence mask.</p>+<h4>Args:</h4>+<ul>+<li><code>lengths</code> — TODO.</li>+<li><code>maxlen</code> — TODO.</li>+<li><code>dtype</code> — TODO.</li>+<li><code>device</code> — Target device ("cuda:0", "cpu", etc.).</li>+</ul>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code>def sequence_mask(lengths, maxlen=None, dtype=torch.float32, device=None):+ """Sequence mask.++ Args:+ lengths: TODO.+ maxlen: TODO.+ dtype: TODO.+ device: Target device ("cuda:0", "cpu", etc.).+ """+ if maxlen is None:+ maxlen = lengths.max()+ row_vector = torch.arange(0, maxlen, 1).to(lengths.device)+ matrix = torch.unsqueeze(lengths, dim=-1)+ mask = row_vector < matrix+ mask = mask.detach()++ return mask.type(dtype).to(device) if device is not None else mask.type(dtype)+</code></pre></details></div>+<div class="api-detail" id="e227"><span class="dtag badge-c">class</span><h2>SenseVoiceEncoderSmall</h2><div class="dmod">funasr.models.sense_voice · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/sense_voice/model.py#L489" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><ul>+<li><code>Author</code> — Speech Lab of DAMO Academy, Alibaba Group</li>+<li><code>SCAMA</code> — Streaming chunk-aware multihead attention for online end-to-end speech recognition</li>+</ul>+<p>https://arxiv.org/abs/2006.01713</p>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code>class SenseVoiceEncoderSmall(nn.Module):+ """+ Author: Speech Lab of DAMO Academy, Alibaba Group+ SCAMA: Streaming chunk-aware multihead attention for online end-to-end speech recognition+ https://arxiv.org/abs/2006.01713+ """++ def __init__(+ self,+ input_size: int,+ output_size: int = 256,+ attention_heads: int = 4,+ linear_units: int = 2048,+ num_blocks: int = 6,+ tp_blocks: int = 0,+ dropout_rate: float = 0.1,+ positional_dropout_rate: float = 0.1,+ attention_dropout_rate: float = 0.0,+ stochastic_depth_rate: float = 0.0,+ input_layer: Optional[str] = "conv2d",+ pos_enc_class=SinusoidalPositionEncoder,+ normalize_before: bool = True,+ concat_after: bool = False,+ positionwise_layer_type: str = "linear",+ positionwise_conv_kernel_size: int = 1,+ padding_idx: int = -1,+ kernel_size: int = 11,+ sanm_shfit: int = 0,+ selfattention_layer_type: str = "sanm",+ **kwargs,+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/sense_voice/model.py#L489" target="_blank" class="src-more">View full source on GitHub →</a></details><h3 class="mt">Methods</h3><div class="mblk"><code>.output_size()</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/sense_voice/model.py#L619" target="_blank" class="gh-link">L619</a><div class="mdoc"><p>Output size.</p>+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def output_size(self) -> int:+ """Output size."""+ return self._output_size+</code></pre></details></div><div class="mblk"><code>.forward(xs_pad, ilens)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/sense_voice/model.py#L623" target="_blank" class="gh-link">L623</a><div class="mdoc"><p>Embed positions in tensor.</p>+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def forward(+ self,+ xs_pad: torch.Tensor,+ ilens: torch.Tensor,+ ):+ """Embed positions in tensor."""+ maxlen = xs_pad.shape[1]+ masks = sequence_mask(ilens, maxlen=maxlen, device=ilens.device)[:, None, :]++ xs_pad *= self.output_size() ** 0.5++ xs_pad = self.embed(xs_pad)++ # forward encoder1+ for layer_idx, encoder_layer in enumerate(self.encoders0):+ encoder_outs = encoder_layer(xs_pad, masks)+ xs_pad, masks = encoder_outs[0], encoder_outs[1]++ for layer_idx, encoder_layer in enumerate(self.encoders):+ encoder_outs = encoder_layer(xs_pad, masks)+ xs_pad, masks = encoder_outs[0], encoder_outs[1]++ xs_pad = self.after_norm(xs_pad)++ # forward encoder2+ olens = masks.squeeze(1).sum(1).int()++ for layer_idx, encoder_layer in enumerate(self.tp_encoders):+ encoder_outs = encoder_layer(xs_pad, masks)+ xs_pad, masks = encoder_outs[0], encoder_outs[1]+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/sense_voice/model.py#L623" target="_blank" class="src-more">View full source →</a></details></div></div>+<div class="api-detail" id="e228"><span class="dtag badge-m">method</span><h2><code>SenseVoiceEncoderSmall.output_size()</code></h2><div class="dmod">funasr.models.sense_voice.SenseVoiceEncoderSmall · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/sense_voice/model.py#L619" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Output size.</p>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def output_size(self) -> int:+ """Output size."""+ return self._output_size+</code></pre></details></div>+<div class="api-detail" id="e229"><span class="dtag badge-m">method</span><h2><code>SenseVoiceEncoderSmall.forward(xs_pad, ilens)</code></h2><div class="dmod">funasr.models.sense_voice.SenseVoiceEncoderSmall · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/sense_voice/model.py#L623" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Embed positions in tensor.</p>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def forward(+ self,+ xs_pad: torch.Tensor,+ ilens: torch.Tensor,+ ):+ """Embed positions in tensor."""+ maxlen = xs_pad.shape[1]+ masks = sequence_mask(ilens, maxlen=maxlen, device=ilens.device)[:, None, :]++ xs_pad *= self.output_size() ** 0.5++ xs_pad = self.embed(xs_pad)++ # forward encoder1+ for layer_idx, encoder_layer in enumerate(self.encoders0):+ encoder_outs = encoder_layer(xs_pad, masks)+ xs_pad, masks = encoder_outs[0], encoder_outs[1]++ for layer_idx, encoder_layer in enumerate(self.encoders):+ encoder_outs = encoder_layer(xs_pad, masks)+ xs_pad, masks = encoder_outs[0], encoder_outs[1]++ xs_pad = self.after_norm(xs_pad)++ # forward encoder2+ olens = masks.squeeze(1).sum(1).int()++ for layer_idx, encoder_layer in enumerate(self.tp_encoders):+ encoder_outs = encoder_layer(xs_pad, masks)+ xs_pad, masks = encoder_outs[0], encoder_outs[1]+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/sense_voice/model.py#L623" target="_blank" class="src-more">View full source on GitHub →</a></details></div>+<div class="api-detail" id="e230"><span class="dtag badge-c">class</span><h2>SenseVoiceSmall</h2><div class="dmod">funasr.models.sense_voice · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/sense_voice/model.py#L659" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><ul>+<li><code>CTC</code> — attention hybrid Encoder-Decoder model</li>+</ul>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code>class SenseVoiceSmall(nn.Module):+ """CTC-attention hybrid Encoder-Decoder model"""++ def __init__(+ self,+ specaug: str = None,+ specaug_conf: dict = None,+ normalize: str = None,+ normalize_conf: dict = None,+ encoder: str = None,+ encoder_conf: dict = None,+ ctc_conf: dict = None,+ input_size: int = 80,+ vocab_size: int = -1,+ ignore_id: int = -1,+ blank_id: int = 0,+ sos: int = 1,+ eos: int = 2,+ length_normalized_loss: bool = False,+ **kwargs,+ ):++ """Initialize SenseVoiceSmall.++ Args:+ specaug: TODO.+ specaug_conf: Configuration dict for specaug.+ normalize: TODO.+ normalize_conf: Configuration dict for normalize.+ encoder: TODO.+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/sense_voice/model.py#L659" target="_blank" class="src-more">View full source on GitHub →</a></details><h3 class="mt">Methods</h3><div class="mblk"><code>.from_pretrained(model, **kwargs)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/sense_voice/model.py#L754" target="_blank" class="gh-link">L754</a><div class="mdoc"><p>From pretrained.</p>+<h4>Args:</h4>+<ul>+<li><code>model</code> — Model instance or model name.</li>+<li><code>**kwargs</code> — Additional keyword arguments.</li>+</ul>+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def from_pretrained(model: str = None, **kwargs):+ """From pretrained.++ Args:+ model: Model instance or model name.+ **kwargs: Additional keyword arguments.+ """+ from funasr import AutoModel++ model, kwargs = AutoModel.build_model(model=model, trust_remote_code=True, **kwargs)++ return model, kwargs+</code></pre></details></div><div class="mblk"><code>.forward(speech, speech_lengths, text, text_lengths, **kwargs)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/sense_voice/model.py#L767" target="_blank" class="gh-link">L767</a><div class="mdoc"><p>Encoder + Decoder + Calc loss</p>+<h4>Args:</h4>+<ul>+<li><code>speech</code> — (Batch, Length, ...)</li>+<li><code>speech_lengths</code> — (Batch, )</li>+<li><code>text</code> — (Batch, Length)</li>+<li><code>text_lengths</code> — (Batch,)</li>+</ul>+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def forward(+ self,+ speech: torch.Tensor,+ speech_lengths: torch.Tensor,+ text: torch.Tensor,+ text_lengths: torch.Tensor,+ **kwargs,+ ):+ """Encoder + Decoder + Calc loss+ Args:+ speech: (Batch, Length, ...)+ speech_lengths: (Batch, )+ text: (Batch, Length)+ text_lengths: (Batch,)+ """+ # import pdb;+ # pdb.set_trace()+ if len(text_lengths.size()) > 1:+ text_lengths = text_lengths[:, 0]+ if len(speech_lengths.size()) > 1:+ speech_lengths = speech_lengths[:, 0]++ batch_size = speech.shape[0]++ # 1. Encoder+ encoder_out, encoder_out_lens = self.encode(speech, speech_lengths, text)++ loss_ctc, cer_ctc = None, None+ loss_rich, acc_rich = None, None+ stats = dict()+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/sense_voice/model.py#L767" target="_blank" class="src-more">View full source →</a></details></div><div class="mblk"><code>.encode(speech, speech_lengths, text, **kwargs)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/sense_voice/model.py#L817" target="_blank" class="gh-link">L817</a><div class="mdoc"><p>Frontend + Encoder. Note that this method is used by asr_inference.py</p>+<h4>Args:</h4>+<ul>+<li><code>speech</code> — (Batch, Length, ...)</li>+<li><code>speech_lengths</code> — (Batch, )</li>+<li><code>ind</code> — int</li>+</ul>+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def encode(+ self,+ speech: torch.Tensor,+ speech_lengths: torch.Tensor,+ text: torch.Tensor,+ **kwargs,+ ):+ """Frontend + Encoder. Note that this method is used by asr_inference.py+ Args:+ speech: (Batch, Length, ...)+ speech_lengths: (Batch, )+ ind: int+ """++ # Data augmentation+ if self.specaug is not None and self.training:+ speech, speech_lengths = self.specaug(speech, speech_lengths)++ # Normalization for feature: e.g. Global-CMVN, Utterance-CMVN+ if self.normalize is not None:+ speech, speech_lengths = self.normalize(speech, speech_lengths)++ lids = torch.LongTensor(+ [+ [+ (+ self.lid_int_dict[int(lid)]+ if torch.rand(1) > 0.2 and int(lid) in self.lid_int_dict+ else 0+ )+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/sense_voice/model.py#L817" target="_blank" class="src-more">View full source →</a></details></div><div class="mblk"><code>.inference(data_in, data_lengths, key, tokenizer, frontend, **kwargs)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/sense_voice/model.py#L918" target="_blank" class="gh-link">L918</a><div class="mdoc"><p>Run inference on input data.</p>+<h4>Args:</h4>+<ul>+<li><code>data_in</code> — Input data (audio samples, file paths, or text).</li>+<li><code>data_lengths</code> — Lengths of each input sample in the batch.</li>+<li><code>key</code> — Sample identifiers.</li>+<li><code>tokenizer</code> — Tokenizer instance for text encoding/decoding.</li>+<li><code>frontend</code> — Audio frontend for feature extraction.</li>+<li><code>**kwargs</code> — Additional keyword arguments.</li>+</ul>+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def inference(+ self,+ data_in,+ data_lengths=None,+ key: list = ["wav_file_tmp_name"],+ tokenizer=None,+ frontend=None,+ **kwargs,+ ):++ """Run inference on input data.++ Args:+ data_in: Input data (audio samples, file paths, or text).+ data_lengths: Lengths of each input sample in the batch.+ key: Sample identifiers.+ tokenizer: Tokenizer instance for text encoding/decoding.+ frontend: Audio frontend for feature extraction.+ **kwargs: Additional keyword arguments.+ """+ meta_data = {}+ if (+ isinstance(data_in, torch.Tensor) and kwargs.get("data_type", "sound") == "fbank"+ ): # fbank+ speech, speech_lengths = data_in, data_lengths+ if len(speech.shape) < 3:+ speech = speech[None, :, :]+ if speech_lengths is None:+ speech_lengths = speech.shape[1]+ else:+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/sense_voice/model.py#L918" target="_blank" class="src-more">View full source →</a></details></div><div class="mblk"><code>.post(timestamp)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/sense_voice/model.py#L1082" target="_blank" class="gh-link">L1082</a><div class="mdoc"><p>Post.</p>+<h4>Args:</h4>+<ul>+<li><code>timestamp</code> — TODO.</li>+</ul>+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def post(self, timestamp):+ """Post.++ Args:+ timestamp: TODO.+ """+ timestamp_new = []+ words_new = []+ prev_word = None+ for i, t in enumerate(timestamp):+ word, start, end = t+ start = int(start * 1000)+ end = int(end * 1000)+ if word == "▁":+ continue+ if i == 0:+ # timestamp_new.append([word, start, end])+ timestamp_new.append([start, end])+ words_new.append(word)+ elif word.startswith("▁"):+ word = word[1:]+ timestamp_new.append([start, end])+ words_new.append(word)+ elif prev_word is not None and prev_word.isalpha() and prev_word.isascii() and word.isalpha() and word.isascii():+ word = prev_word + word+ timestamp_new[-1][1] = end+ words_new[-1] = word+ else:+ # timestamp_new[-1][0] += word+ timestamp_new.append([start, end])+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/sense_voice/model.py#L1082" target="_blank" class="src-more">View full source →</a></details></div><div class="mblk"><code>.export(**kwargs)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/sense_voice/model.py#L1116" target="_blank" class="gh-link">L1116</a><div class="mdoc"><p>Export.</p>+<h4>Args:</h4>+<ul>+<li><code>**kwargs</code> — Additional keyword arguments.</li>+</ul>+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def export(self, **kwargs):+ """Export.++ Args:+ **kwargs: Additional keyword arguments.+ """+ from .export_meta import export_rebuild_model++ if "max_seq_len" not in kwargs:+ kwargs["max_seq_len"] = 512+ models = export_rebuild_model(model=self, **kwargs)+ return models++ return results, meta_data+</code></pre></details></div></div>+<div class="api-detail" id="e231"><span class="dtag badge-m">method</span><h2><code>SenseVoiceSmall.from_pretrained(model, **kwargs)</code></h2><div class="dmod">funasr.models.sense_voice.SenseVoiceSmall · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/sense_voice/model.py#L754" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>From pretrained.</p>+<h4>Args:</h4>+<ul>+<li><code>model</code> — Model instance or model name.</li>+<li><code>**kwargs</code> — Additional keyword arguments.</li>+</ul>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def from_pretrained(model: str = None, **kwargs):+ """From pretrained.++ Args:+ model: Model instance or model name.+ **kwargs: Additional keyword arguments.+ """+ from funasr import AutoModel++ model, kwargs = AutoModel.build_model(model=model, trust_remote_code=True, **kwargs)++ return model, kwargs+</code></pre></details></div>+<div class="api-detail" id="e232"><span class="dtag badge-m">method</span><h2><code>SenseVoiceSmall.forward(speech, speech_lengths, text, text_lengths, **kwargs)</code></h2><div class="dmod">funasr.models.sense_voice.SenseVoiceSmall · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/sense_voice/model.py#L767" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Encoder + Decoder + Calc loss</p>+<h4>Args:</h4>+<ul>+<li><code>speech</code> — (Batch, Length, ...)</li>+<li><code>speech_lengths</code> — (Batch, )</li>+<li><code>text</code> — (Batch, Length)</li>+<li><code>text_lengths</code> — (Batch,)</li>+</ul>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def forward(+ self,+ speech: torch.Tensor,+ speech_lengths: torch.Tensor,+ text: torch.Tensor,+ text_lengths: torch.Tensor,+ **kwargs,+ ):+ """Encoder + Decoder + Calc loss+ Args:+ speech: (Batch, Length, ...)+ speech_lengths: (Batch, )+ text: (Batch, Length)+ text_lengths: (Batch,)+ """+ # import pdb;+ # pdb.set_trace()+ if len(text_lengths.size()) > 1:+ text_lengths = text_lengths[:, 0]+ if len(speech_lengths.size()) > 1:+ speech_lengths = speech_lengths[:, 0]++ batch_size = speech.shape[0]++ # 1. Encoder+ encoder_out, encoder_out_lens = self.encode(speech, speech_lengths, text)++ loss_ctc, cer_ctc = None, None+ loss_rich, acc_rich = None, None+ stats = dict()+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/sense_voice/model.py#L767" target="_blank" class="src-more">View full source on GitHub →</a></details></div>+<div class="api-detail" id="e233"><span class="dtag badge-m">method</span><h2><code>SenseVoiceSmall.encode(speech, speech_lengths, text, **kwargs)</code></h2><div class="dmod">funasr.models.sense_voice.SenseVoiceSmall · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/sense_voice/model.py#L817" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Frontend + Encoder. Note that this method is used by asr_inference.py</p>+<h4>Args:</h4>+<ul>+<li><code>speech</code> — (Batch, Length, ...)</li>+<li><code>speech_lengths</code> — (Batch, )</li>+<li><code>ind</code> — int</li>+</ul>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def encode(+ self,+ speech: torch.Tensor,+ speech_lengths: torch.Tensor,+ text: torch.Tensor,+ **kwargs,+ ):+ """Frontend + Encoder. Note that this method is used by asr_inference.py+ Args:+ speech: (Batch, Length, ...)+ speech_lengths: (Batch, )+ ind: int+ """++ # Data augmentation+ if self.specaug is not None and self.training:+ speech, speech_lengths = self.specaug(speech, speech_lengths)++ # Normalization for feature: e.g. Global-CMVN, Utterance-CMVN+ if self.normalize is not None:+ speech, speech_lengths = self.normalize(speech, speech_lengths)++ lids = torch.LongTensor(+ [+ [+ (+ self.lid_int_dict[int(lid)]+ if torch.rand(1) > 0.2 and int(lid) in self.lid_int_dict+ else 0+ )+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/sense_voice/model.py#L817" target="_blank" class="src-more">View full source on GitHub →</a></details></div>+<div class="api-detail" id="e234"><span class="dtag badge-m">method</span><h2><code>SenseVoiceSmall.inference(data_in, data_lengths, key, tokenizer, frontend, **kwargs)</code></h2><div class="dmod">funasr.models.sense_voice.SenseVoiceSmall · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/sense_voice/model.py#L918" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Run inference on input data.</p>+<h4>Args:</h4>+<ul>+<li><code>data_in</code> — Input data (audio samples, file paths, or text).</li>+<li><code>data_lengths</code> — Lengths of each input sample in the batch.</li>+<li><code>key</code> — Sample identifiers.</li>+<li><code>tokenizer</code> — Tokenizer instance for text encoding/decoding.</li>+<li><code>frontend</code> — Audio frontend for feature extraction.</li>+<li><code>**kwargs</code> — Additional keyword arguments.</li>+</ul>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def inference(+ self,+ data_in,+ data_lengths=None,+ key: list = ["wav_file_tmp_name"],+ tokenizer=None,+ frontend=None,+ **kwargs,+ ):++ """Run inference on input data.++ Args:+ data_in: Input data (audio samples, file paths, or text).+ data_lengths: Lengths of each input sample in the batch.+ key: Sample identifiers.+ tokenizer: Tokenizer instance for text encoding/decoding.+ frontend: Audio frontend for feature extraction.+ **kwargs: Additional keyword arguments.+ """+ meta_data = {}+ if (+ isinstance(data_in, torch.Tensor) and kwargs.get("data_type", "sound") == "fbank"+ ): # fbank+ speech, speech_lengths = data_in, data_lengths+ if len(speech.shape) < 3:+ speech = speech[None, :, :]+ if speech_lengths is None:+ speech_lengths = speech.shape[1]+ else:+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/sense_voice/model.py#L918" target="_blank" class="src-more">View full source on GitHub →</a></details></div>+<div class="api-detail" id="e235"><span class="dtag badge-m">method</span><h2><code>SenseVoiceSmall.post(timestamp)</code></h2><div class="dmod">funasr.models.sense_voice.SenseVoiceSmall · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/sense_voice/model.py#L1082" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Post.</p>+<h4>Args:</h4>+<ul>+<li><code>timestamp</code> — TODO.</li>+</ul>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def post(self, timestamp):+ """Post.++ Args:+ timestamp: TODO.+ """+ timestamp_new = []+ words_new = []+ prev_word = None+ for i, t in enumerate(timestamp):+ word, start, end = t+ start = int(start * 1000)+ end = int(end * 1000)+ if word == "▁":+ continue+ if i == 0:+ # timestamp_new.append([word, start, end])+ timestamp_new.append([start, end])+ words_new.append(word)+ elif word.startswith("▁"):+ word = word[1:]+ timestamp_new.append([start, end])+ words_new.append(word)+ elif prev_word is not None and prev_word.isalpha() and prev_word.isascii() and word.isalpha() and word.isascii():+ word = prev_word + word+ timestamp_new[-1][1] = end+ words_new[-1] = word+ else:+ # timestamp_new[-1][0] += word+ timestamp_new.append([start, end])+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/sense_voice/model.py#L1082" target="_blank" class="src-more">View full source on GitHub →</a></details></div>+<div class="api-detail" id="e236"><span class="dtag badge-m">method</span><h2><code>SenseVoiceSmall.export(**kwargs)</code></h2><div class="dmod">funasr.models.sense_voice.SenseVoiceSmall · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/sense_voice/model.py#L1116" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Export.</p>+<h4>Args:</h4>+<ul>+<li><code>**kwargs</code> — Additional keyword arguments.</li>+</ul>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def export(self, **kwargs):+ """Export.++ Args:+ **kwargs: Additional keyword arguments.+ """+ from .export_meta import export_rebuild_model++ if "max_seq_len" not in kwargs:+ kwargs["max_seq_len"] = 512+ models = export_rebuild_model(model=self, **kwargs)+ return models++ return results, meta_data+</code></pre></details></div>+<div class="api-detail" id="e237"><span class="dtag badge-c">class</span><h2>Transducer</h2><div class="dmod">funasr.models.transducer · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/transducer/model.py#L40" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><ul>+<li><code>Transducer (RNN-T)</code> — Streaming ASR using encoder-predictor-joint architecture.</li>+</ul>+<p>Combines encoder (audio frames), predictor (text history), and joint network.</p>+<p>Supports beam search decoding. Suitable for low-latency streaming applications.</p>+<ul>+<li><code>Output</code> — {"key": str, "text": str}</li>+</ul>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code>class Transducer(torch.nn.Module):+ """Transducer (RNN-T): Streaming ASR using encoder-predictor-joint architecture.++ Combines encoder (audio frames), predictor (text history), and joint network.+ Supports beam search decoding. Suitable for low-latency streaming applications.++ Output: {"key": str, "text": str}+ """++ def __init__(+ self,+ frontend: Optional[str] = None,+ frontend_conf: Optional[Dict] = None,+ specaug: Optional[str] = None,+ specaug_conf: Optional[Dict] = None,+ normalize: str = None,+ normalize_conf: Optional[Dict] = None,+ encoder: str = None,+ encoder_conf: Optional[Dict] = None,+ decoder: str = None,+ decoder_conf: Optional[Dict] = None,+ joint_network: str = None,+ joint_network_conf: Optional[Dict] = None,+ transducer_weight: float = 1.0,+ fastemit_lambda: float = 0.0,+ auxiliary_ctc_weight: float = 0.0,+ auxiliary_ctc_dropout_rate: float = 0.0,+ auxiliary_lm_loss_weight: float = 0.0,+ auxiliary_lm_loss_smoothing: float = 0.0,+ input_size: int = 80,+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/transducer/model.py#L40" target="_blank" class="src-more">View full source on GitHub →</a></details><h3 class="mt">Methods</h3><div class="mblk"><code>.forward(speech, speech_lengths, text, text_lengths, **kwargs)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/transducer/model.py#L190" target="_blank" class="gh-link">L190</a><div class="mdoc"><p>Encoder + Decoder + Calc loss</p>+<h4>Args:</h4>+<ul>+<li><code>speech</code> — (Batch, Length, ...)</li>+<li><code>speech_lengths</code> — (Batch, )</li>+<li><code>text</code> — (Batch, Length)</li>+<li><code>text_lengths</code> — (Batch,)</li>+</ul>+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def forward(+ self,+ speech: torch.Tensor,+ speech_lengths: torch.Tensor,+ text: torch.Tensor,+ text_lengths: torch.Tensor,+ **kwargs,+ ) -> Tuple[torch.Tensor, Dict[str, torch.Tensor], torch.Tensor]:+ """Encoder + Decoder + Calc loss+ Args:+ speech: (Batch, Length, ...)+ speech_lengths: (Batch, )+ text: (Batch, Length)+ text_lengths: (Batch,)+ """+ if len(text_lengths.size()) > 1:+ text_lengths = text_lengths[:, 0]+ if len(speech_lengths.size()) > 1:+ speech_lengths = speech_lengths[:, 0]++ batch_size = speech.shape[0]+ # 1. Encoder+ encoder_out, encoder_out_lens = self.encode(speech, speech_lengths)+ if (+ hasattr(self.encoder, "overlap_chunk_cls")+ and self.encoder.overlap_chunk_cls is not None+ ):+ encoder_out, encoder_out_lens = self.encoder.overlap_chunk_cls.remove_chunk(+ encoder_out, encoder_out_lens, chunk_outs=None+ )+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/transducer/model.py#L190" target="_blank" class="src-more">View full source →</a></details></div><div class="mblk"><code>.encode(speech, speech_lengths, **kwargs)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/transducer/model.py#L276" target="_blank" class="gh-link">L276</a><div class="mdoc"><p>Frontend + Encoder. Note that this method is used by asr_inference.py</p>+<h4>Args:</h4>+<ul>+<li><code>speech</code> — (Batch, Length, ...)</li>+<li><code>speech_lengths</code> — (Batch, )</li>+<li><code>ind</code> — int</li>+</ul>+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def encode(+ self,+ speech: torch.Tensor,+ speech_lengths: torch.Tensor,+ **kwargs,+ ) -> Tuple[torch.Tensor, torch.Tensor]:+ """Frontend + Encoder. Note that this method is used by asr_inference.py+ Args:+ speech: (Batch, Length, ...)+ speech_lengths: (Batch, )+ ind: int+ """+ with autocast(False):++ # Data augmentation+ if self.specaug is not None and self.training:+ speech, speech_lengths = self.specaug(speech, speech_lengths)++ # Normalization for feature: e.g. Global-CMVN, Utterance-CMVN+ if self.normalize is not None:+ speech, speech_lengths = self.normalize(speech, speech_lengths)++ # Forward encoder+ # feats: (Batch, Length, Dim)+ # -> encoder_out: (Batch, Length2, Dim2)+ encoder_out, encoder_out_lens, _ = self.encoder(speech, speech_lengths)+ intermediate_outs = None+ if isinstance(encoder_out, tuple):+ intermediate_outs = encoder_out[1]+ encoder_out = encoder_out[0]+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/transducer/model.py#L276" target="_blank" class="src-more">View full source →</a></details></div><div class="mblk"><code>.init_beam_search(**kwargs)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/transducer/model.py#L455" target="_blank" class="gh-link">L455</a><div class="mdoc"><p>Init beam search.</p>+<h4>Args:</h4>+<ul>+<li><code>**kwargs</code> — Additional keyword arguments.</li>+</ul>+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def init_beam_search(+ self,+ **kwargs,+ ):++ # 1. Build ASR model+ """Init beam search.++ Args:+ **kwargs: Additional keyword arguments.+ """+ scorers = {}++ if self.ctc != None:+ ctc = CTCPrefixScorer(ctc=self.ctc, eos=self.eos)+ scorers.update(ctc=ctc)+ token_list = kwargs.get("token_list")+ scorers.update(+ length_bonus=LengthBonus(len(token_list)),+ )++ # 3. Build ngram model+ # ngram is not supported now+ ngram = None+ scorers["ngram"] = ngram++ beam_search = BeamSearchTransducer(+ self.decoder,+ self.joint_network,+ kwargs.get("beam_size", 2),+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/transducer/model.py#L455" target="_blank" class="src-more">View full source →</a></details></div><div class="mblk"><code>.inference(data_in, data_lengths, key, tokenizer, **kwargs)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/transducer/model.py#L493" target="_blank" class="gh-link">L493</a><div class="mdoc"><p>Run inference on input data.</p>+<h4>Args:</h4>+<ul>+<li><code>data_in</code> — Input data (audio samples, file paths, or text).</li>+<li><code>data_lengths</code> — Lengths of each input sample in the batch.</li>+<li><code>key</code> — Sample identifiers.</li>+<li><code>tokenizer</code> — Tokenizer instance for text encoding/decoding.</li>+<li><code>**kwargs</code> — Additional keyword arguments.</li>+</ul>+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def inference(+ self,+ data_in: list,+ data_lengths: list = None,+ key: list = None,+ tokenizer=None,+ **kwargs,+ ):++ """Run inference on input data.++ Args:+ data_in: Input data (audio samples, file paths, or text).+ data_lengths: Lengths of each input sample in the batch.+ key: Sample identifiers.+ tokenizer: Tokenizer instance for text encoding/decoding.+ **kwargs: Additional keyword arguments.+ """+ if kwargs.get("batch_size", 1) > 1:+ raise NotImplementedError("batch decoding is not implemented")++ # init beamsearch+ is_use_ctc = kwargs.get("decoding_ctc_weight", 0.0) > 0.00001 and self.ctc != None+ is_use_lm = (+ kwargs.get("lm_weight", 0.0) > 0.00001 and kwargs.get("lm_file", None) is not None+ )+ # if self.beam_search is None and (is_use_lm or is_use_ctc):+ logging.info("enable beam_search")+ self.init_beam_search(**kwargs)+ self.nbest = kwargs.get("nbest", 1)+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/transducer/model.py#L493" target="_blank" class="src-more">View full source →</a></details></div></div>+<div class="api-detail" id="e238"><span class="dtag badge-m">method</span><h2><code>Transducer.forward(speech, speech_lengths, text, text_lengths, **kwargs)</code></h2><div class="dmod">funasr.models.transducer.Transducer · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/transducer/model.py#L190" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Encoder + Decoder + Calc loss</p>+<h4>Args:</h4>+<ul>+<li><code>speech</code> — (Batch, Length, ...)</li>+<li><code>speech_lengths</code> — (Batch, )</li>+<li><code>text</code> — (Batch, Length)</li>+<li><code>text_lengths</code> — (Batch,)</li>+</ul>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def forward(+ self,+ speech: torch.Tensor,+ speech_lengths: torch.Tensor,+ text: torch.Tensor,+ text_lengths: torch.Tensor,+ **kwargs,+ ) -> Tuple[torch.Tensor, Dict[str, torch.Tensor], torch.Tensor]:+ """Encoder + Decoder + Calc loss+ Args:+ speech: (Batch, Length, ...)+ speech_lengths: (Batch, )+ text: (Batch, Length)+ text_lengths: (Batch,)+ """+ if len(text_lengths.size()) > 1:+ text_lengths = text_lengths[:, 0]+ if len(speech_lengths.size()) > 1:+ speech_lengths = speech_lengths[:, 0]++ batch_size = speech.shape[0]+ # 1. Encoder+ encoder_out, encoder_out_lens = self.encode(speech, speech_lengths)+ if (+ hasattr(self.encoder, "overlap_chunk_cls")+ and self.encoder.overlap_chunk_cls is not None+ ):+ encoder_out, encoder_out_lens = self.encoder.overlap_chunk_cls.remove_chunk(+ encoder_out, encoder_out_lens, chunk_outs=None+ )+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/transducer/model.py#L190" target="_blank" class="src-more">View full source on GitHub →</a></details></div>+<div class="api-detail" id="e239"><span class="dtag badge-m">method</span><h2><code>Transducer.encode(speech, speech_lengths, **kwargs)</code></h2><div class="dmod">funasr.models.transducer.Transducer · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/transducer/model.py#L276" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Frontend + Encoder. Note that this method is used by asr_inference.py</p>+<h4>Args:</h4>+<ul>+<li><code>speech</code> — (Batch, Length, ...)</li>+<li><code>speech_lengths</code> — (Batch, )</li>+<li><code>ind</code> — int</li>+</ul>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def encode(+ self,+ speech: torch.Tensor,+ speech_lengths: torch.Tensor,+ **kwargs,+ ) -> Tuple[torch.Tensor, torch.Tensor]:+ """Frontend + Encoder. Note that this method is used by asr_inference.py+ Args:+ speech: (Batch, Length, ...)+ speech_lengths: (Batch, )+ ind: int+ """+ with autocast(False):++ # Data augmentation+ if self.specaug is not None and self.training:+ speech, speech_lengths = self.specaug(speech, speech_lengths)++ # Normalization for feature: e.g. Global-CMVN, Utterance-CMVN+ if self.normalize is not None:+ speech, speech_lengths = self.normalize(speech, speech_lengths)++ # Forward encoder+ # feats: (Batch, Length, Dim)+ # -> encoder_out: (Batch, Length2, Dim2)+ encoder_out, encoder_out_lens, _ = self.encoder(speech, speech_lengths)+ intermediate_outs = None+ if isinstance(encoder_out, tuple):+ intermediate_outs = encoder_out[1]+ encoder_out = encoder_out[0]+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/transducer/model.py#L276" target="_blank" class="src-more">View full source on GitHub →</a></details></div>+<div class="api-detail" id="e240"><span class="dtag badge-m">method</span><h2><code>Transducer.init_beam_search(**kwargs)</code></h2><div class="dmod">funasr.models.transducer.Transducer · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/transducer/model.py#L455" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Init beam search.</p>+<h4>Args:</h4>+<ul>+<li><code>**kwargs</code> — Additional keyword arguments.</li>+</ul>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def init_beam_search(+ self,+ **kwargs,+ ):++ # 1. Build ASR model+ """Init beam search.++ Args:+ **kwargs: Additional keyword arguments.+ """+ scorers = {}++ if self.ctc != None:+ ctc = CTCPrefixScorer(ctc=self.ctc, eos=self.eos)+ scorers.update(ctc=ctc)+ token_list = kwargs.get("token_list")+ scorers.update(+ length_bonus=LengthBonus(len(token_list)),+ )++ # 3. Build ngram model+ # ngram is not supported now+ ngram = None+ scorers["ngram"] = ngram++ beam_search = BeamSearchTransducer(+ self.decoder,+ self.joint_network,+ kwargs.get("beam_size", 2),+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/transducer/model.py#L455" target="_blank" class="src-more">View full source on GitHub →</a></details></div>+<div class="api-detail" id="e241"><span class="dtag badge-m">method</span><h2><code>Transducer.inference(data_in, data_lengths, key, tokenizer, **kwargs)</code></h2><div class="dmod">funasr.models.transducer.Transducer · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/transducer/model.py#L493" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Run inference on input data.</p>+<h4>Args:</h4>+<ul>+<li><code>data_in</code> — Input data (audio samples, file paths, or text).</li>+<li><code>data_lengths</code> — Lengths of each input sample in the batch.</li>+<li><code>key</code> — Sample identifiers.</li>+<li><code>tokenizer</code> — Tokenizer instance for text encoding/decoding.</li>+<li><code>**kwargs</code> — Additional keyword arguments.</li>+</ul>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def inference(+ self,+ data_in: list,+ data_lengths: list = None,+ key: list = None,+ tokenizer=None,+ **kwargs,+ ):++ """Run inference on input data.++ Args:+ data_in: Input data (audio samples, file paths, or text).+ data_lengths: Lengths of each input sample in the batch.+ key: Sample identifiers.+ tokenizer: Tokenizer instance for text encoding/decoding.+ **kwargs: Additional keyword arguments.+ """+ if kwargs.get("batch_size", 1) > 1:+ raise NotImplementedError("batch decoding is not implemented")++ # init beamsearch+ is_use_ctc = kwargs.get("decoding_ctc_weight", 0.0) > 0.00001 and self.ctc != None+ is_use_lm = (+ kwargs.get("lm_weight", 0.0) > 0.00001 and kwargs.get("lm_file", None) is not None+ )+ # if self.beam_search is None and (is_use_lm or is_use_ctc):+ logging.info("enable beam_search")+ self.init_beam_search(**kwargs)+ self.nbest = kwargs.get("nbest", 1)+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/transducer/model.py#L493" target="_blank" class="src-more">View full source on GitHub →</a></details></div>+<div class="api-detail" id="e242"><span class="dtag badge-c">class</span><h2>Transformer</h2><div class="dmod">funasr.models.transformer · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/transformer/model.py#L23" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><ul>+<li><code>Transformer</code> — Base encoder-decoder ASR model.</li>+</ul>+<p>Standard CTC-attention hybrid architecture with:</p>+<ul>+<li>Encoder (self-attention + position encoding)</li>+<li>CTC branch for auxiliary loss</li>+<li>Attention decoder for sequence generation</li>+<li>Beam search with LM fusion</li>+</ul>+<p>Base class for Conformer, Branchformer, etc.</p>+<ul>+<li><code>Output</code> — {"key": str, "text": str}</li>+</ul>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code>class Transformer(nn.Module):+ """Transformer: Base encoder-decoder ASR model.++ Standard CTC-attention hybrid architecture with:+ - Encoder (self-attention + position encoding)+ - CTC branch for auxiliary loss+ - Attention decoder for sequence generation+ - Beam search with LM fusion++ Base class for Conformer, Branchformer, etc.+ Output: {"key": str, "text": str}+ """++ def __init__(+ self,+ specaug: str = None,+ specaug_conf: dict = None,+ normalize: str = None,+ normalize_conf: dict = None,+ encoder: str = None,+ encoder_conf: dict = None,+ decoder: str = None,+ decoder_conf: dict = None,+ ctc: str = None,+ ctc_conf: dict = None,+ ctc_weight: float = 0.5,+ interctc_weight: float = 0.0,+ input_size: int = 80,+ vocab_size: int = -1,+ ignore_id: int = -1,+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/transformer/model.py#L23" target="_blank" class="src-more">View full source on GitHub →</a></details><h3 class="mt">Methods</h3><div class="mblk"><code>.forward(speech, speech_lengths, text, text_lengths, **kwargs)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/transformer/model.py#L173" target="_blank" class="gh-link">L173</a><div class="mdoc"><p>Encoder + Decoder + Calc loss</p>+<h4>Args:</h4>+<ul>+<li><code>speech</code> — (Batch, Length, ...)</li>+<li><code>speech_lengths</code> — (Batch, )</li>+<li><code>text</code> — (Batch, Length)</li>+<li><code>text_lengths</code> — (Batch,)</li>+</ul>+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def forward(+ self,+ speech: torch.Tensor,+ speech_lengths: torch.Tensor,+ text: torch.Tensor,+ text_lengths: torch.Tensor,+ **kwargs,+ ) -> Tuple[torch.Tensor, Dict[str, torch.Tensor], torch.Tensor]:+ """Encoder + Decoder + Calc loss+ Args:+ speech: (Batch, Length, ...)+ speech_lengths: (Batch, )+ text: (Batch, Length)+ text_lengths: (Batch,)+ """+ if len(text_lengths.size()) > 1:+ text_lengths = text_lengths[:, 0]+ if len(speech_lengths.size()) > 1:+ speech_lengths = speech_lengths[:, 0]++ batch_size = speech.shape[0]++ # 1. Encoder+ encoder_out, encoder_out_lens = self.encode(speech, speech_lengths)+ intermediate_outs = None+ if isinstance(encoder_out, tuple):+ intermediate_outs = encoder_out[1]+ encoder_out = encoder_out[0]++ loss_att, acc_att, cer_att, wer_att = None, None, None, None+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/transformer/model.py#L173" target="_blank" class="src-more">View full source →</a></details></div><div class="mblk"><code>.encode(speech, speech_lengths, **kwargs)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/transformer/model.py#L266" target="_blank" class="gh-link">L266</a><div class="mdoc"><p>Frontend + Encoder. Note that this method is used by asr_inference.py</p>+<h4>Args:</h4>+<ul>+<li><code>speech</code> — (Batch, Length, ...)</li>+<li><code>speech_lengths</code> — (Batch, )</li>+<li><code>ind</code> — int</li>+</ul>+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def encode(+ self,+ speech: torch.Tensor,+ speech_lengths: torch.Tensor,+ **kwargs,+ ) -> Tuple[torch.Tensor, torch.Tensor]:+ """Frontend + Encoder. Note that this method is used by asr_inference.py+ Args:+ speech: (Batch, Length, ...)+ speech_lengths: (Batch, )+ ind: int+ """+ with autocast(False):++ # Data augmentation+ if self.specaug is not None and self.training:+ speech, speech_lengths = self.specaug(speech, speech_lengths)++ # Normalization for feature: e.g. Global-CMVN, Utterance-CMVN+ if self.normalize is not None:+ speech, speech_lengths = self.normalize(speech, speech_lengths)++ # Forward encoder+ # feats: (Batch, Length, Dim)+ # -> encoder_out: (Batch, Length2, Dim2)+ if self.encoder.interctc_use_conditioning:+ encoder_out, encoder_out_lens, _ = self.encoder(speech, speech_lengths, ctc=self.ctc)+ else:+ encoder_out, encoder_out_lens, _ = self.encoder(speech, speech_lengths)+ intermediate_outs = None+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/transformer/model.py#L266" target="_blank" class="src-more">View full source →</a></details></div><div class="mblk"><code>.init_beam_search(**kwargs)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/transformer/model.py#L368" target="_blank" class="gh-link">L368</a><div class="mdoc"><p>Init beam search.</p>+<h4>Args:</h4>+<ul>+<li><code>**kwargs</code> — Additional keyword arguments.</li>+</ul>+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def init_beam_search(+ self,+ **kwargs,+ ):+ """Init beam search.++ Args:+ **kwargs: Additional keyword arguments.+ """+ from funasr.models.transformer.search import BeamSearch+ from funasr.models.transformer.scorers.ctc import CTCPrefixScorer+ from funasr.models.transformer.scorers.length_bonus import LengthBonus++ # 1. Build ASR model+ scorers = {}++ if self.ctc != None:+ ctc = CTCPrefixScorer(ctc=self.ctc, eos=self.eos)+ scorers.update(ctc=ctc)+ token_list = kwargs.get("token_list")+ scorers.update(+ decoder=self.decoder,+ length_bonus=LengthBonus(len(token_list)),+ )++ # 3. Build ngram model+ # ngram is not supported now+ ngram = None+ scorers["ngram"] = ngram++</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/transformer/model.py#L368" target="_blank" class="src-more">View full source →</a></details></div><div class="mblk"><code>.inference(data_in, data_lengths, key, tokenizer, frontend, **kwargs)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/transformer/model.py#L418" target="_blank" class="gh-link">L418</a><div class="mdoc"><p>Run inference on input data.</p>+<h4>Args:</h4>+<ul>+<li><code>data_in</code> — Input data (audio samples, file paths, or text).</li>+<li><code>data_lengths</code> — Lengths of each input sample in the batch.</li>+<li><code>key</code> — Sample identifiers.</li>+<li><code>tokenizer</code> — Tokenizer instance for text encoding/decoding.</li>+<li><code>frontend</code> — Audio frontend for feature extraction.</li>+<li><code>**kwargs</code> — Additional keyword arguments.</li>+</ul>+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def inference(+ self,+ data_in,+ data_lengths=None,+ key: list = None,+ tokenizer=None,+ frontend=None,+ **kwargs,+ ):++ """Run inference on input data.++ Args:+ data_in: Input data (audio samples, file paths, or text).+ data_lengths: Lengths of each input sample in the batch.+ key: Sample identifiers.+ tokenizer: Tokenizer instance for text encoding/decoding.+ frontend: Audio frontend for feature extraction.+ **kwargs: Additional keyword arguments.+ """+ if kwargs.get("batch_size", 1) > 1:+ raise NotImplementedError("batch decoding is not implemented")++ # init beamsearch+ if self.beam_search is None:+ logging.info("enable beam_search")+ self.init_beam_search(**kwargs)+ self.nbest = kwargs.get("nbest", 1)++ meta_data = {}+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/transformer/model.py#L418" target="_blank" class="src-more">View full source →</a></details></div></div>+<div class="api-detail" id="e243"><span class="dtag badge-m">method</span><h2><code>Transformer.forward(speech, speech_lengths, text, text_lengths, **kwargs)</code></h2><div class="dmod">funasr.models.transformer.Transformer · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/transformer/model.py#L173" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Encoder + Decoder + Calc loss</p>+<h4>Args:</h4>+<ul>+<li><code>speech</code> — (Batch, Length, ...)</li>+<li><code>speech_lengths</code> — (Batch, )</li>+<li><code>text</code> — (Batch, Length)</li>+<li><code>text_lengths</code> — (Batch,)</li>+</ul>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def forward(+ self,+ speech: torch.Tensor,+ speech_lengths: torch.Tensor,+ text: torch.Tensor,+ text_lengths: torch.Tensor,+ **kwargs,+ ) -> Tuple[torch.Tensor, Dict[str, torch.Tensor], torch.Tensor]:+ """Encoder + Decoder + Calc loss+ Args:+ speech: (Batch, Length, ...)+ speech_lengths: (Batch, )+ text: (Batch, Length)+ text_lengths: (Batch,)+ """+ if len(text_lengths.size()) > 1:+ text_lengths = text_lengths[:, 0]+ if len(speech_lengths.size()) > 1:+ speech_lengths = speech_lengths[:, 0]++ batch_size = speech.shape[0]++ # 1. Encoder+ encoder_out, encoder_out_lens = self.encode(speech, speech_lengths)+ intermediate_outs = None+ if isinstance(encoder_out, tuple):+ intermediate_outs = encoder_out[1]+ encoder_out = encoder_out[0]++ loss_att, acc_att, cer_att, wer_att = None, None, None, None+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/transformer/model.py#L173" target="_blank" class="src-more">View full source on GitHub →</a></details></div>+<div class="api-detail" id="e244"><span class="dtag badge-m">method</span><h2><code>Transformer.encode(speech, speech_lengths, **kwargs)</code></h2><div class="dmod">funasr.models.transformer.Transformer · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/transformer/model.py#L266" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Frontend + Encoder. Note that this method is used by asr_inference.py</p>+<h4>Args:</h4>+<ul>+<li><code>speech</code> — (Batch, Length, ...)</li>+<li><code>speech_lengths</code> — (Batch, )</li>+<li><code>ind</code> — int</li>+</ul>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def encode(+ self,+ speech: torch.Tensor,+ speech_lengths: torch.Tensor,+ **kwargs,+ ) -> Tuple[torch.Tensor, torch.Tensor]:+ """Frontend + Encoder. Note that this method is used by asr_inference.py+ Args:+ speech: (Batch, Length, ...)+ speech_lengths: (Batch, )+ ind: int+ """+ with autocast(False):++ # Data augmentation+ if self.specaug is not None and self.training:+ speech, speech_lengths = self.specaug(speech, speech_lengths)++ # Normalization for feature: e.g. Global-CMVN, Utterance-CMVN+ if self.normalize is not None:+ speech, speech_lengths = self.normalize(speech, speech_lengths)++ # Forward encoder+ # feats: (Batch, Length, Dim)+ # -> encoder_out: (Batch, Length2, Dim2)+ if self.encoder.interctc_use_conditioning:+ encoder_out, encoder_out_lens, _ = self.encoder(speech, speech_lengths, ctc=self.ctc)+ else:+ encoder_out, encoder_out_lens, _ = self.encoder(speech, speech_lengths)+ intermediate_outs = None+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/transformer/model.py#L266" target="_blank" class="src-more">View full source on GitHub →</a></details></div>+<div class="api-detail" id="e245"><span class="dtag badge-m">method</span><h2><code>Transformer.init_beam_search(**kwargs)</code></h2><div class="dmod">funasr.models.transformer.Transformer · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/transformer/model.py#L368" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Init beam search.</p>+<h4>Args:</h4>+<ul>+<li><code>**kwargs</code> — Additional keyword arguments.</li>+</ul>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def init_beam_search(+ self,+ **kwargs,+ ):+ """Init beam search.++ Args:+ **kwargs: Additional keyword arguments.+ """+ from funasr.models.transformer.search import BeamSearch+ from funasr.models.transformer.scorers.ctc import CTCPrefixScorer+ from funasr.models.transformer.scorers.length_bonus import LengthBonus++ # 1. Build ASR model+ scorers = {}++ if self.ctc != None:+ ctc = CTCPrefixScorer(ctc=self.ctc, eos=self.eos)+ scorers.update(ctc=ctc)+ token_list = kwargs.get("token_list")+ scorers.update(+ decoder=self.decoder,+ length_bonus=LengthBonus(len(token_list)),+ )++ # 3. Build ngram model+ # ngram is not supported now+ ngram = None+ scorers["ngram"] = ngram++</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/transformer/model.py#L368" target="_blank" class="src-more">View full source on GitHub →</a></details></div>+<div class="api-detail" id="e246"><span class="dtag badge-m">method</span><h2><code>Transformer.inference(data_in, data_lengths, key, tokenizer, frontend, **kwargs)</code></h2><div class="dmod">funasr.models.transformer.Transformer · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/transformer/model.py#L418" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Run inference on input data.</p>+<h4>Args:</h4>+<ul>+<li><code>data_in</code> — Input data (audio samples, file paths, or text).</li>+<li><code>data_lengths</code> — Lengths of each input sample in the batch.</li>+<li><code>key</code> — Sample identifiers.</li>+<li><code>tokenizer</code> — Tokenizer instance for text encoding/decoding.</li>+<li><code>frontend</code> — Audio frontend for feature extraction.</li>+<li><code>**kwargs</code> — Additional keyword arguments.</li>+</ul>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def inference(+ self,+ data_in,+ data_lengths=None,+ key: list = None,+ tokenizer=None,+ frontend=None,+ **kwargs,+ ):++ """Run inference on input data.++ Args:+ data_in: Input data (audio samples, file paths, or text).+ data_lengths: Lengths of each input sample in the batch.+ key: Sample identifiers.+ tokenizer: Tokenizer instance for text encoding/decoding.+ frontend: Audio frontend for feature extraction.+ **kwargs: Additional keyword arguments.+ """+ if kwargs.get("batch_size", 1) > 1:+ raise NotImplementedError("batch decoding is not implemented")++ # init beamsearch+ if self.beam_search is None:+ logging.info("enable beam_search")+ self.init_beam_search(**kwargs)+ self.nbest = kwargs.get("nbest", 1)++ meta_data = {}+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/transformer/model.py#L418" target="_blank" class="src-more">View full source on GitHub →</a></details></div>+<div class="api-detail" id="e247"><span class="dtag badge-c">class</span><h2>UniASR</h2><div class="dmod">funasr.models.uniasr · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/uniasr/model.py#L27" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><ul>+<li><code>UniASR</code> — Unified Streaming and Non-streaming ASR model.</li>+</ul>+<p>Single model that supports both streaming (online) and non-streaming</p>+<p>(offline) decoding through dynamic masking in the encoder.</p>+<p>Inherits Paraformer pipeline.</p>+<ul>+<li><code>Output</code> — {"key": str, "text": str}</li>+</ul>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code>class UniASR(torch.nn.Module):+ """UniASR: Unified Streaming and Non-streaming ASR model.++ Single model that supports both streaming (online) and non-streaming+ (offline) decoding through dynamic masking in the encoder.++ Inherits Paraformer pipeline.+ Output: {"key": str, "text": str}+ """++ def __init__(+ self,+ specaug: str = None,+ specaug_conf: dict = None,+ normalize: str = None,+ normalize_conf: dict = None,+ encoder: str = None,+ encoder_conf: dict = None,+ encoder2: str = None,+ encoder2_conf: dict = None,+ decoder: str = None,+ decoder_conf: dict = None,+ decoder2: str = None,+ decoder2_conf: dict = None,+ predictor: str = None,+ predictor_conf: dict = None,+ predictor_bias: int = 0,+ predictor_weight: float = 0.0,+ predictor2: str = None,+ predictor2_conf: dict = None,+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/uniasr/model.py#L27" target="_blank" class="src-more">View full source on GitHub →</a></details><h3 class="mt">Methods</h3><div class="mblk"><code>.forward(speech, speech_lengths, text, text_lengths, **kwargs)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/uniasr/model.py#L235" target="_blank" class="gh-link">L235</a><div class="mdoc"><p>Frontend + Encoder + Decoder + Calc loss</p>+<h4>Args:</h4>+<ul>+<li><code>speech</code> — (Batch, Length, ...)</li>+<li><code>speech_lengths</code> — (Batch, )</li>+<li><code>text</code> — (Batch, Length)</li>+<li><code>text_lengths</code> — (Batch,)</li>+</ul>+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def forward(+ self,+ speech: torch.Tensor,+ speech_lengths: torch.Tensor,+ text: torch.Tensor,+ text_lengths: torch.Tensor,+ **kwargs,+ ) -> Tuple[torch.Tensor, Dict[str, torch.Tensor], torch.Tensor]:+ """Frontend + Encoder + Decoder + Calc loss+ Args:+ speech: (Batch, Length, ...)+ speech_lengths: (Batch, )+ text: (Batch, Length)+ text_lengths: (Batch,)+ """+ decoding_ind = kwargs.get("decoding_ind", None)+ if len(text_lengths.size()) > 1:+ text_lengths = text_lengths[:, 0]+ if len(speech_lengths.size()) > 1:+ speech_lengths = speech_lengths[:, 0]++ batch_size = speech.shape[0]++ ind = self.encoder.overlap_chunk_cls.random_choice(self.training, decoding_ind)+ # 1. Encoder+ if self.enable_maas_finetune:+ with torch.no_grad():+ speech_raw, encoder_out, encoder_out_lens = self.encode(+ speech, speech_lengths, ind=ind+ )+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/uniasr/model.py#L235" target="_blank" class="src-more">View full source →</a></details></div><div class="mblk"><code>.collect_feats(speech, speech_lengths, text, text_lengths)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/uniasr/model.py#L354" target="_blank" class="gh-link">L354</a><div class="mdoc"><p>Collect feats.</p>+<h4>Args:</h4>+<ul>+<li><code>speech</code> — Speech audio tensor, shape (batch, time).</li>+<li><code>speech_lengths</code> — Length of each speech sample.</li>+<li><code>text</code> — Text tensor or string input.</li>+<li><code>text_lengths</code> — Length of each text sample.</li>+</ul>+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def collect_feats(+ self,+ speech: torch.Tensor,+ speech_lengths: torch.Tensor,+ text: torch.Tensor,+ text_lengths: torch.Tensor,+ ) -> Dict[str, torch.Tensor]:+ """Collect feats.++ Args:+ speech: Speech audio tensor, shape (batch, time).+ speech_lengths: Length of each speech sample.+ text: Text tensor or string input.+ text_lengths: Length of each text sample.+ """+ if self.extract_feats_in_collect_stats:+ feats, feats_lengths = self._extract_feats(speech, speech_lengths)+ else:+ # Generate dummy stats if extract_feats_in_collect_stats is False+ logging.warning(+ "Generating dummy stats for feats and feats_lengths, "+ "because encoder_conf.extract_feats_in_collect_stats is "+ f"{self.extract_feats_in_collect_stats}"+ )+ feats, feats_lengths = speech, speech_lengths+ return {"feats": feats, "feats_lengths": feats_lengths}+</code></pre></details></div><div class="mblk"><code>.encode(speech, speech_lengths, **kwargs)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/uniasr/model.py#L381" target="_blank" class="gh-link">L381</a><div class="mdoc"><p>Frontend + Encoder. Note that this method is used by asr_inference.py</p>+<h4>Args:</h4>+<ul>+<li><code>speech</code> — (Batch, Length, ...)</li>+<li><code>speech_lengths</code> — (Batch, )</li>+</ul>+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def encode(+ self,+ speech: torch.Tensor,+ speech_lengths: torch.Tensor,+ **kwargs,+ ):+ """Frontend + Encoder. Note that this method is used by asr_inference.py+ Args:+ speech: (Batch, Length, ...)+ speech_lengths: (Batch, )+ """+ ind = kwargs.get("ind", 0)+ with autocast(False):+ # Data augmentation+ if self.specaug is not None and self.training:+ speech, speech_lengths = self.specaug(speech, speech_lengths)++ # Normalization for feature: e.g. Global-CMVN, Utterance-CMVN+ if self.normalize is not None:+ speech, speech_lengths = self.normalize(speech, speech_lengths)++ speech_raw = speech.clone().to(speech.device)++ # 4. Forward encoder+ encoder_out, encoder_out_lens, _ = self.encoder(speech, speech_lengths, ind=ind)+ if isinstance(encoder_out, tuple):+ encoder_out = encoder_out[0]++ return speech_raw, encoder_out, encoder_out_lens+</code></pre></details></div><div class="mblk"><code>.encode2(encoder_out, encoder_out_lens, speech, speech_lengths, **kwargs)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/uniasr/model.py#L411" target="_blank" class="gh-link">L411</a><div class="mdoc"><p>Frontend + Encoder. Note that this method is used by asr_inference.py</p>+<h4>Args:</h4>+<ul>+<li><code>speech</code> — (Batch, Length, ...)</li>+<li><code>speech_lengths</code> — (Batch, )</li>+</ul>+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def encode2(+ self,+ encoder_out: torch.Tensor,+ encoder_out_lens: torch.Tensor,+ speech: torch.Tensor,+ speech_lengths: torch.Tensor,+ **kwargs,+ ):+ """Frontend + Encoder. Note that this method is used by asr_inference.py+ Args:+ speech: (Batch, Length, ...)+ speech_lengths: (Batch, )+ """++ ind = kwargs.get("ind", 0)+ encoder_out_rm, encoder_out_lens_rm = self.encoder.overlap_chunk_cls.remove_chunk(+ encoder_out,+ encoder_out_lens,+ chunk_outs=None,+ )+ # residual_input+ encoder_out = torch.cat((speech, encoder_out_rm), dim=-1)+ encoder_out_lens = encoder_out_lens_rm+ if self.stride_conv is not None:+ speech, speech_lengths = self.stride_conv(encoder_out, encoder_out_lens)+ if not self.encoder1_encoder2_joint_training:+ speech = speech.detach()+ speech_lengths = speech_lengths.detach()+ # 4. Forward encoder+ # feats: (Batch, Length, Dim)+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/uniasr/model.py#L411" target="_blank" class="src-more">View full source →</a></details></div><div class="mblk"><code>.nll(encoder_out, encoder_out_lens, ys_pad, ys_pad_lens)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/uniasr/model.py#L449" target="_blank" class="gh-link">L449</a><div class="mdoc"><p>Compute negative log likelihood(nll) from transformer-decoder</p>+<p>Normally, this function is called in batchify_nll.</p>+<h4>Args:</h4>+<ul>+<li><code>encoder_out</code> — (Batch, Length, Dim)</li>+<li><code>encoder_out_lens</code> — (Batch,)</li>+<li><code>ys_pad</code> — (Batch, Length)</li>+<li><code>ys_pad_lens</code> — (Batch,)</li>+</ul>+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def nll(+ self,+ encoder_out: torch.Tensor,+ encoder_out_lens: torch.Tensor,+ ys_pad: torch.Tensor,+ ys_pad_lens: torch.Tensor,+ ) -> torch.Tensor:+ """Compute negative log likelihood(nll) from transformer-decoder+ Normally, this function is called in batchify_nll.+ Args:+ encoder_out: (Batch, Length, Dim)+ encoder_out_lens: (Batch,)+ ys_pad: (Batch, Length)+ ys_pad_lens: (Batch,)+ """+ ys_in_pad, ys_out_pad = add_sos_eos(ys_pad, self.sos, self.eos, self.ignore_id)+ ys_in_lens = ys_pad_lens + 1++ # 1. Forward decoder+ decoder_out, _ = self.decoder(+ encoder_out, encoder_out_lens, ys_in_pad, ys_in_lens+ ) # [batch, seqlen, dim]+ batch_size = decoder_out.size(0)+ decoder_num_class = decoder_out.size(2)+ # nll: negative log-likelihood+ nll = torch.nn.functional.cross_entropy(+ decoder_out.view(-1, decoder_num_class),+ ys_out_pad.view(-1),+ ignore_index=self.ignore_id,+ reduction="none",+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/uniasr/model.py#L449" target="_blank" class="src-more">View full source →</a></details></div><div class="mblk"><code>.batchify_nll(encoder_out, encoder_out_lens, ys_pad, ys_pad_lens, batch_size)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/uniasr/model.py#L485" target="_blank" class="gh-link">L485</a><div class="mdoc"><p>Compute negative log likelihood(nll) from transformer-decoder</p>+<p>To avoid OOM, this fuction seperate the input into batches.</p>+<p>Then call nll for each batch and combine and return results.</p>+<h4>Args:</h4>+<ul>+<li><code>encoder_out</code> — (Batch, Length, Dim)</li>+<li><code>encoder_out_lens</code> — (Batch,)</li>+<li><code>ys_pad</code> — (Batch, Length)</li>+<li><code>ys_pad_lens</code> — (Batch,)</li>+<li><code>batch_size</code> — int, samples each batch contain when computing nll,</li>+<li class="sub">you may change this to avoid OOM or increase</li>+<li class="sub">GPU memory usage</li>+</ul>+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def batchify_nll(+ self,+ encoder_out: torch.Tensor,+ encoder_out_lens: torch.Tensor,+ ys_pad: torch.Tensor,+ ys_pad_lens: torch.Tensor,+ batch_size: int = 100,+ ):+ """Compute negative log likelihood(nll) from transformer-decoder+ To avoid OOM, this fuction seperate the input into batches.+ Then call nll for each batch and combine and return results.+ Args:+ encoder_out: (Batch, Length, Dim)+ encoder_out_lens: (Batch,)+ ys_pad: (Batch, Length)+ ys_pad_lens: (Batch,)+ batch_size: int, samples each batch contain when computing nll,+ you may change this to avoid OOM or increase+ GPU memory usage+ """+ total_num = encoder_out.size(0)+ if total_num <= batch_size:+ nll = self.nll(encoder_out, encoder_out_lens, ys_pad, ys_pad_lens)+ else:+ nll = []+ start_idx = 0+ while True:+ end_idx = min(start_idx + batch_size, total_num)+ batch_encoder_out = encoder_out[start_idx:end_idx, :, :]+ batch_encoder_out_lens = encoder_out_lens[start_idx:end_idx]+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/uniasr/model.py#L485" target="_blank" class="src-more">View full source →</a></details></div><div class="mblk"><code>.calc_predictor_mask(encoder_out, encoder_out_lens, ys_pad, ys_pad_lens)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/uniasr/model.py#L787" target="_blank" class="gh-link">L787</a><div class="mdoc"><p>Calc predictor mask.</p>+<h4>Args:</h4>+<ul>+<li><code>encoder_out</code> — Encoder output tensor.</li>+<li><code>encoder_out_lens</code> — Encoder output lengths.</li>+<li><code>ys_pad</code> — TODO.</li>+<li><code>ys_pad_lens</code> — Lengths of ys_pad.</li>+</ul>+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def calc_predictor_mask(+ self,+ encoder_out: torch.Tensor,+ encoder_out_lens: torch.Tensor,+ ys_pad: torch.Tensor = None,+ ys_pad_lens: torch.Tensor = None,+ ):+ # ys_in_pad, ys_out_pad = add_sos_eos(ys_pad, self.sos, self.eos, self.ignore_id)+ # ys_in_lens = ys_pad_lens + 1+ """Calc predictor mask.++ Args:+ encoder_out: Encoder output tensor.+ encoder_out_lens: Encoder output lengths.+ ys_pad: TODO.+ ys_pad_lens: Lengths of ys_pad.+ """+ ys_out_pad, ys_in_lens = None, None++ encoder_out_mask = sequence_mask(+ encoder_out_lens,+ maxlen=encoder_out.size(1),+ dtype=encoder_out.dtype,+ device=encoder_out.device,+ )[:, None, :]+ mask_chunk_predictor = None+ if self.encoder.overlap_chunk_cls is not None:+ mask_chunk_predictor = self.encoder.overlap_chunk_cls.get_mask_chunk_predictor(+ None, device=encoder_out.device, batch_size=encoder_out.size(0)+ )+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/uniasr/model.py#L787" target="_blank" class="src-more">View full source →</a></details></div><div class="mblk"><code>.calc_predictor_mask2(encoder_out, encoder_out_lens, ys_pad, ys_pad_lens)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/uniasr/model.py#L877" target="_blank" class="gh-link">L877</a><div class="mdoc"><p>Calc predictor mask2.</p>+<h4>Args:</h4>+<ul>+<li><code>encoder_out</code> — Encoder output tensor.</li>+<li><code>encoder_out_lens</code> — Encoder output lengths.</li>+<li><code>ys_pad</code> — TODO.</li>+<li><code>ys_pad_lens</code> — Lengths of ys_pad.</li>+</ul>+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def calc_predictor_mask2(+ self,+ encoder_out: torch.Tensor,+ encoder_out_lens: torch.Tensor,+ ys_pad: torch.Tensor = None,+ ys_pad_lens: torch.Tensor = None,+ ):+ # ys_in_pad, ys_out_pad = add_sos_eos(ys_pad, self.sos, self.eos, self.ignore_id)+ # ys_in_lens = ys_pad_lens + 1+ """Calc predictor mask2.++ Args:+ encoder_out: Encoder output tensor.+ encoder_out_lens: Encoder output lengths.+ ys_pad: TODO.+ ys_pad_lens: Lengths of ys_pad.+ """+ ys_out_pad, ys_in_lens = None, None++ encoder_out_mask = sequence_mask(+ encoder_out_lens,+ maxlen=encoder_out.size(1),+ dtype=encoder_out.dtype,+ device=encoder_out.device,+ )[:, None, :]+ mask_chunk_predictor = None+ if self.encoder2.overlap_chunk_cls is not None:+ mask_chunk_predictor = self.encoder2.overlap_chunk_cls.get_mask_chunk_predictor(+ None, device=encoder_out.device, batch_size=encoder_out.size(0)+ )+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/uniasr/model.py#L877" target="_blank" class="src-more">View full source →</a></details></div><div class="mblk"><code>.init_beam_search(**kwargs)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/uniasr/model.py#L967" target="_blank" class="gh-link">L967</a><div class="mdoc"><p>Init beam search.</p>+<h4>Args:</h4>+<ul>+<li><code>**kwargs</code> — Additional keyword arguments.</li>+</ul>+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def init_beam_search(+ self,+ **kwargs,+ ):+ """Init beam search.++ Args:+ **kwargs: Additional keyword arguments.+ """+ from funasr.models.uniasr.beam_search import BeamSearchScama+ from funasr.models.transformer.scorers.ctc import CTCPrefixScorer+ from funasr.models.transformer.scorers.length_bonus import LengthBonus++ decoding_mode = kwargs.get("decoding_mode", "model1")+ if decoding_mode == "model1":+ decoder = self.decoder+ else:+ decoder = self.decoder2+ # 1. Build ASR model+ scorers = {}++ if self.ctc != None:+ ctc = CTCPrefixScorer(ctc=self.ctc, eos=self.eos)+ scorers.update(ctc=ctc)+ token_list = kwargs.get("token_list")+ scorers.update(+ decoder=decoder,+ length_bonus=LengthBonus(len(token_list)),+ )++</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/uniasr/model.py#L967" target="_blank" class="src-more">View full source →</a></details></div><div class="mblk"><code>.inference(data_in, data_lengths, key, tokenizer, frontend, **kwargs)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/uniasr/model.py#L1022" target="_blank" class="gh-link">L1022</a><div class="mdoc"><p>Run inference on input data.</p>+<h4>Args:</h4>+<ul>+<li><code>data_in</code> — Input data (audio samples, file paths, or text).</li>+<li><code>data_lengths</code> — Lengths of each input sample in the batch.</li>+<li><code>key</code> — Sample identifiers.</li>+<li><code>tokenizer</code> — Tokenizer instance for text encoding/decoding.</li>+<li><code>frontend</code> — Audio frontend for feature extraction.</li>+<li><code>**kwargs</code> — Additional keyword arguments.</li>+</ul>+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def inference(+ self,+ data_in,+ data_lengths=None,+ key: list = None,+ tokenizer=None,+ frontend=None,+ **kwargs,+ ):++ """Run inference on input data.++ Args:+ data_in: Input data (audio samples, file paths, or text).+ data_lengths: Lengths of each input sample in the batch.+ key: Sample identifiers.+ tokenizer: Tokenizer instance for text encoding/decoding.+ frontend: Audio frontend for feature extraction.+ **kwargs: Additional keyword arguments.+ """+ decoding_model = kwargs.get("decoding_model", "normal")+ token_num_relax = kwargs.get("token_num_relax", 5)+ if decoding_model == "fast":+ decoding_ind = 0+ decoding_mode = "model1"+ elif decoding_model == "offline":+ decoding_ind = 1+ decoding_mode = "model2"+ else:+ decoding_ind = 0+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/uniasr/model.py#L1022" target="_blank" class="src-more">View full source →</a></details></div></div>+<div class="api-detail" id="e248"><span class="dtag badge-m">method</span><h2><code>UniASR.forward(speech, speech_lengths, text, text_lengths, **kwargs)</code></h2><div class="dmod">funasr.models.uniasr.UniASR · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/uniasr/model.py#L235" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Frontend + Encoder + Decoder + Calc loss</p>+<h4>Args:</h4>+<ul>+<li><code>speech</code> — (Batch, Length, ...)</li>+<li><code>speech_lengths</code> — (Batch, )</li>+<li><code>text</code> — (Batch, Length)</li>+<li><code>text_lengths</code> — (Batch,)</li>+</ul>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def forward(+ self,+ speech: torch.Tensor,+ speech_lengths: torch.Tensor,+ text: torch.Tensor,+ text_lengths: torch.Tensor,+ **kwargs,+ ) -> Tuple[torch.Tensor, Dict[str, torch.Tensor], torch.Tensor]:+ """Frontend + Encoder + Decoder + Calc loss+ Args:+ speech: (Batch, Length, ...)+ speech_lengths: (Batch, )+ text: (Batch, Length)+ text_lengths: (Batch,)+ """+ decoding_ind = kwargs.get("decoding_ind", None)+ if len(text_lengths.size()) > 1:+ text_lengths = text_lengths[:, 0]+ if len(speech_lengths.size()) > 1:+ speech_lengths = speech_lengths[:, 0]++ batch_size = speech.shape[0]++ ind = self.encoder.overlap_chunk_cls.random_choice(self.training, decoding_ind)+ # 1. Encoder+ if self.enable_maas_finetune:+ with torch.no_grad():+ speech_raw, encoder_out, encoder_out_lens = self.encode(+ speech, speech_lengths, ind=ind+ )+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/uniasr/model.py#L235" target="_blank" class="src-more">View full source on GitHub →</a></details></div>+<div class="api-detail" id="e249"><span class="dtag badge-m">method</span><h2><code>UniASR.collect_feats(speech, speech_lengths, text, text_lengths)</code></h2><div class="dmod">funasr.models.uniasr.UniASR · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/uniasr/model.py#L354" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Collect feats.</p>+<h4>Args:</h4>+<ul>+<li><code>speech</code> — Speech audio tensor, shape (batch, time).</li>+<li><code>speech_lengths</code> — Length of each speech sample.</li>+<li><code>text</code> — Text tensor or string input.</li>+<li><code>text_lengths</code> — Length of each text sample.</li>+</ul>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def collect_feats(+ self,+ speech: torch.Tensor,+ speech_lengths: torch.Tensor,+ text: torch.Tensor,+ text_lengths: torch.Tensor,+ ) -> Dict[str, torch.Tensor]:+ """Collect feats.++ Args:+ speech: Speech audio tensor, shape (batch, time).+ speech_lengths: Length of each speech sample.+ text: Text tensor or string input.+ text_lengths: Length of each text sample.+ """+ if self.extract_feats_in_collect_stats:+ feats, feats_lengths = self._extract_feats(speech, speech_lengths)+ else:+ # Generate dummy stats if extract_feats_in_collect_stats is False+ logging.warning(+ "Generating dummy stats for feats and feats_lengths, "+ "because encoder_conf.extract_feats_in_collect_stats is "+ f"{self.extract_feats_in_collect_stats}"+ )+ feats, feats_lengths = speech, speech_lengths+ return {"feats": feats, "feats_lengths": feats_lengths}+</code></pre></details></div>+<div class="api-detail" id="e250"><span class="dtag badge-m">method</span><h2><code>UniASR.encode(speech, speech_lengths, **kwargs)</code></h2><div class="dmod">funasr.models.uniasr.UniASR · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/uniasr/model.py#L381" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Frontend + Encoder. Note that this method is used by asr_inference.py</p>+<h4>Args:</h4>+<ul>+<li><code>speech</code> — (Batch, Length, ...)</li>+<li><code>speech_lengths</code> — (Batch, )</li>+</ul>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def encode(+ self,+ speech: torch.Tensor,+ speech_lengths: torch.Tensor,+ **kwargs,+ ):+ """Frontend + Encoder. Note that this method is used by asr_inference.py+ Args:+ speech: (Batch, Length, ...)+ speech_lengths: (Batch, )+ """+ ind = kwargs.get("ind", 0)+ with autocast(False):+ # Data augmentation+ if self.specaug is not None and self.training:+ speech, speech_lengths = self.specaug(speech, speech_lengths)++ # Normalization for feature: e.g. Global-CMVN, Utterance-CMVN+ if self.normalize is not None:+ speech, speech_lengths = self.normalize(speech, speech_lengths)++ speech_raw = speech.clone().to(speech.device)++ # 4. Forward encoder+ encoder_out, encoder_out_lens, _ = self.encoder(speech, speech_lengths, ind=ind)+ if isinstance(encoder_out, tuple):+ encoder_out = encoder_out[0]++ return speech_raw, encoder_out, encoder_out_lens+</code></pre></details></div>+<div class="api-detail" id="e251"><span class="dtag badge-m">method</span><h2><code>UniASR.encode2(encoder_out, encoder_out_lens, speech, speech_lengths, **kwargs)</code></h2><div class="dmod">funasr.models.uniasr.UniASR · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/uniasr/model.py#L411" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Frontend + Encoder. Note that this method is used by asr_inference.py</p>+<h4>Args:</h4>+<ul>+<li><code>speech</code> — (Batch, Length, ...)</li>+<li><code>speech_lengths</code> — (Batch, )</li>+</ul>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def encode2(+ self,+ encoder_out: torch.Tensor,+ encoder_out_lens: torch.Tensor,+ speech: torch.Tensor,+ speech_lengths: torch.Tensor,+ **kwargs,+ ):+ """Frontend + Encoder. Note that this method is used by asr_inference.py+ Args:+ speech: (Batch, Length, ...)+ speech_lengths: (Batch, )+ """++ ind = kwargs.get("ind", 0)+ encoder_out_rm, encoder_out_lens_rm = self.encoder.overlap_chunk_cls.remove_chunk(+ encoder_out,+ encoder_out_lens,+ chunk_outs=None,+ )+ # residual_input+ encoder_out = torch.cat((speech, encoder_out_rm), dim=-1)+ encoder_out_lens = encoder_out_lens_rm+ if self.stride_conv is not None:+ speech, speech_lengths = self.stride_conv(encoder_out, encoder_out_lens)+ if not self.encoder1_encoder2_joint_training:+ speech = speech.detach()+ speech_lengths = speech_lengths.detach()+ # 4. Forward encoder+ # feats: (Batch, Length, Dim)+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/uniasr/model.py#L411" target="_blank" class="src-more">View full source on GitHub →</a></details></div>+<div class="api-detail" id="e252"><span class="dtag badge-m">method</span><h2><code>UniASR.nll(encoder_out, encoder_out_lens, ys_pad, ys_pad_lens)</code></h2><div class="dmod">funasr.models.uniasr.UniASR · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/uniasr/model.py#L449" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Compute negative log likelihood(nll) from transformer-decoder</p>+<p>Normally, this function is called in batchify_nll.</p>+<h4>Args:</h4>+<ul>+<li><code>encoder_out</code> — (Batch, Length, Dim)</li>+<li><code>encoder_out_lens</code> — (Batch,)</li>+<li><code>ys_pad</code> — (Batch, Length)</li>+<li><code>ys_pad_lens</code> — (Batch,)</li>+</ul>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def nll(+ self,+ encoder_out: torch.Tensor,+ encoder_out_lens: torch.Tensor,+ ys_pad: torch.Tensor,+ ys_pad_lens: torch.Tensor,+ ) -> torch.Tensor:+ """Compute negative log likelihood(nll) from transformer-decoder+ Normally, this function is called in batchify_nll.+ Args:+ encoder_out: (Batch, Length, Dim)+ encoder_out_lens: (Batch,)+ ys_pad: (Batch, Length)+ ys_pad_lens: (Batch,)+ """+ ys_in_pad, ys_out_pad = add_sos_eos(ys_pad, self.sos, self.eos, self.ignore_id)+ ys_in_lens = ys_pad_lens + 1++ # 1. Forward decoder+ decoder_out, _ = self.decoder(+ encoder_out, encoder_out_lens, ys_in_pad, ys_in_lens+ ) # [batch, seqlen, dim]+ batch_size = decoder_out.size(0)+ decoder_num_class = decoder_out.size(2)+ # nll: negative log-likelihood+ nll = torch.nn.functional.cross_entropy(+ decoder_out.view(-1, decoder_num_class),+ ys_out_pad.view(-1),+ ignore_index=self.ignore_id,+ reduction="none",+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/uniasr/model.py#L449" target="_blank" class="src-more">View full source on GitHub →</a></details></div>+<div class="api-detail" id="e253"><span class="dtag badge-m">method</span><h2><code>UniASR.batchify_nll(encoder_out, encoder_out_lens, ys_pad, ys_pad_lens, batch_size)</code></h2><div class="dmod">funasr.models.uniasr.UniASR · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/uniasr/model.py#L485" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Compute negative log likelihood(nll) from transformer-decoder</p>+<p>To avoid OOM, this fuction seperate the input into batches.</p>+<p>Then call nll for each batch and combine and return results.</p>+<h4>Args:</h4>+<ul>+<li><code>encoder_out</code> — (Batch, Length, Dim)</li>+<li><code>encoder_out_lens</code> — (Batch,)</li>+<li><code>ys_pad</code> — (Batch, Length)</li>+<li><code>ys_pad_lens</code> — (Batch,)</li>+<li><code>batch_size</code> — int, samples each batch contain when computing nll,</li>+<li class="sub">you may change this to avoid OOM or increase</li>+<li class="sub">GPU memory usage</li>+</ul>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def batchify_nll(+ self,+ encoder_out: torch.Tensor,+ encoder_out_lens: torch.Tensor,+ ys_pad: torch.Tensor,+ ys_pad_lens: torch.Tensor,+ batch_size: int = 100,+ ):+ """Compute negative log likelihood(nll) from transformer-decoder+ To avoid OOM, this fuction seperate the input into batches.+ Then call nll for each batch and combine and return results.+ Args:+ encoder_out: (Batch, Length, Dim)+ encoder_out_lens: (Batch,)+ ys_pad: (Batch, Length)+ ys_pad_lens: (Batch,)+ batch_size: int, samples each batch contain when computing nll,+ you may change this to avoid OOM or increase+ GPU memory usage+ """+ total_num = encoder_out.size(0)+ if total_num <= batch_size:+ nll = self.nll(encoder_out, encoder_out_lens, ys_pad, ys_pad_lens)+ else:+ nll = []+ start_idx = 0+ while True:+ end_idx = min(start_idx + batch_size, total_num)+ batch_encoder_out = encoder_out[start_idx:end_idx, :, :]+ batch_encoder_out_lens = encoder_out_lens[start_idx:end_idx]+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/uniasr/model.py#L485" target="_blank" class="src-more">View full source on GitHub →</a></details></div>+<div class="api-detail" id="e254"><span class="dtag badge-m">method</span><h2><code>UniASR.calc_predictor_mask(encoder_out, encoder_out_lens, ys_pad, ys_pad_lens)</code></h2><div class="dmod">funasr.models.uniasr.UniASR · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/uniasr/model.py#L787" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Calc predictor mask.</p>+<h4>Args:</h4>+<ul>+<li><code>encoder_out</code> — Encoder output tensor.</li>+<li><code>encoder_out_lens</code> — Encoder output lengths.</li>+<li><code>ys_pad</code> — TODO.</li>+<li><code>ys_pad_lens</code> — Lengths of ys_pad.</li>+</ul>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def calc_predictor_mask(+ self,+ encoder_out: torch.Tensor,+ encoder_out_lens: torch.Tensor,+ ys_pad: torch.Tensor = None,+ ys_pad_lens: torch.Tensor = None,+ ):+ # ys_in_pad, ys_out_pad = add_sos_eos(ys_pad, self.sos, self.eos, self.ignore_id)+ # ys_in_lens = ys_pad_lens + 1+ """Calc predictor mask.++ Args:+ encoder_out: Encoder output tensor.+ encoder_out_lens: Encoder output lengths.+ ys_pad: TODO.+ ys_pad_lens: Lengths of ys_pad.+ """+ ys_out_pad, ys_in_lens = None, None++ encoder_out_mask = sequence_mask(+ encoder_out_lens,+ maxlen=encoder_out.size(1),+ dtype=encoder_out.dtype,+ device=encoder_out.device,+ )[:, None, :]+ mask_chunk_predictor = None+ if self.encoder.overlap_chunk_cls is not None:+ mask_chunk_predictor = self.encoder.overlap_chunk_cls.get_mask_chunk_predictor(+ None, device=encoder_out.device, batch_size=encoder_out.size(0)+ )+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/uniasr/model.py#L787" target="_blank" class="src-more">View full source on GitHub →</a></details></div>+<div class="api-detail" id="e255"><span class="dtag badge-m">method</span><h2><code>UniASR.calc_predictor_mask2(encoder_out, encoder_out_lens, ys_pad, ys_pad_lens)</code></h2><div class="dmod">funasr.models.uniasr.UniASR · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/uniasr/model.py#L877" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Calc predictor mask2.</p>+<h4>Args:</h4>+<ul>+<li><code>encoder_out</code> — Encoder output tensor.</li>+<li><code>encoder_out_lens</code> — Encoder output lengths.</li>+<li><code>ys_pad</code> — TODO.</li>+<li><code>ys_pad_lens</code> — Lengths of ys_pad.</li>+</ul>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def calc_predictor_mask2(+ self,+ encoder_out: torch.Tensor,+ encoder_out_lens: torch.Tensor,+ ys_pad: torch.Tensor = None,+ ys_pad_lens: torch.Tensor = None,+ ):+ # ys_in_pad, ys_out_pad = add_sos_eos(ys_pad, self.sos, self.eos, self.ignore_id)+ # ys_in_lens = ys_pad_lens + 1+ """Calc predictor mask2.++ Args:+ encoder_out: Encoder output tensor.+ encoder_out_lens: Encoder output lengths.+ ys_pad: TODO.+ ys_pad_lens: Lengths of ys_pad.+ """+ ys_out_pad, ys_in_lens = None, None++ encoder_out_mask = sequence_mask(+ encoder_out_lens,+ maxlen=encoder_out.size(1),+ dtype=encoder_out.dtype,+ device=encoder_out.device,+ )[:, None, :]+ mask_chunk_predictor = None+ if self.encoder2.overlap_chunk_cls is not None:+ mask_chunk_predictor = self.encoder2.overlap_chunk_cls.get_mask_chunk_predictor(+ None, device=encoder_out.device, batch_size=encoder_out.size(0)+ )+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/uniasr/model.py#L877" target="_blank" class="src-more">View full source on GitHub →</a></details></div>+<div class="api-detail" id="e256"><span class="dtag badge-m">method</span><h2><code>UniASR.init_beam_search(**kwargs)</code></h2><div class="dmod">funasr.models.uniasr.UniASR · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/uniasr/model.py#L967" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Init beam search.</p>+<h4>Args:</h4>+<ul>+<li><code>**kwargs</code> — Additional keyword arguments.</li>+</ul>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def init_beam_search(+ self,+ **kwargs,+ ):+ """Init beam search.++ Args:+ **kwargs: Additional keyword arguments.+ """+ from funasr.models.uniasr.beam_search import BeamSearchScama+ from funasr.models.transformer.scorers.ctc import CTCPrefixScorer+ from funasr.models.transformer.scorers.length_bonus import LengthBonus++ decoding_mode = kwargs.get("decoding_mode", "model1")+ if decoding_mode == "model1":+ decoder = self.decoder+ else:+ decoder = self.decoder2+ # 1. Build ASR model+ scorers = {}++ if self.ctc != None:+ ctc = CTCPrefixScorer(ctc=self.ctc, eos=self.eos)+ scorers.update(ctc=ctc)+ token_list = kwargs.get("token_list")+ scorers.update(+ decoder=decoder,+ length_bonus=LengthBonus(len(token_list)),+ )++</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/uniasr/model.py#L967" target="_blank" class="src-more">View full source on GitHub →</a></details></div>+<div class="api-detail" id="e257"><span class="dtag badge-m">method</span><h2><code>UniASR.inference(data_in, data_lengths, key, tokenizer, frontend, **kwargs)</code></h2><div class="dmod">funasr.models.uniasr.UniASR · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/uniasr/model.py#L1022" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Run inference on input data.</p>+<h4>Args:</h4>+<ul>+<li><code>data_in</code> — Input data (audio samples, file paths, or text).</li>+<li><code>data_lengths</code> — Lengths of each input sample in the batch.</li>+<li><code>key</code> — Sample identifiers.</li>+<li><code>tokenizer</code> — Tokenizer instance for text encoding/decoding.</li>+<li><code>frontend</code> — Audio frontend for feature extraction.</li>+<li><code>**kwargs</code> — Additional keyword arguments.</li>+</ul>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def inference(+ self,+ data_in,+ data_lengths=None,+ key: list = None,+ tokenizer=None,+ frontend=None,+ **kwargs,+ ):++ """Run inference on input data.++ Args:+ data_in: Input data (audio samples, file paths, or text).+ data_lengths: Lengths of each input sample in the batch.+ key: Sample identifiers.+ tokenizer: Tokenizer instance for text encoding/decoding.+ frontend: Audio frontend for feature extraction.+ **kwargs: Additional keyword arguments.+ """+ decoding_model = kwargs.get("decoding_model", "normal")+ token_num_relax = kwargs.get("token_num_relax", 5)+ if decoding_model == "fast":+ decoding_ind = 0+ decoding_mode = "model1"+ elif decoding_model == "offline":+ decoding_ind = 1+ decoding_mode = "model2"+ else:+ decoding_ind = 0+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/uniasr/model.py#L1022" target="_blank" class="src-more">View full source on GitHub →</a></details></div>+<div class="api-detail" id="e258"><span class="dtag badge-c">class</span><h2>WhisperWarp</h2><div class="dmod">funasr.models.whisper · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/whisper/model.py#L33" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><ul>+<li><code>Whisper</code> — OpenAI Whisper model integration.</li>+</ul>+<p>Wraps Whisper for multilingual speech recognition and translation</p>+<p>within FunASR's AutoModel interface.</p>+<ul>+<li><code>Supports</code> — whisper-tiny through whisper-large-v3-turbo.</li>+<li><code>Output</code> — {"key": str, "text": str}</li>+</ul>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code>class WhisperWarp(nn.Module):+ """Whisper: OpenAI Whisper model integration.++ Wraps Whisper for multilingual speech recognition and translation+ within FunASR's AutoModel interface.++ Supports: whisper-tiny through whisper-large-v3-turbo.+ Output: {"key": str, "text": str}+ """++ def __init__(self, *args, **kwargs):+ """Initialize WhisperWarp.++ Args:+ *args: Variable positional arguments.+ **kwargs: Additional keyword arguments.+ """+ super().__init__()+ hub = kwargs.get("hub", "funasr")+ if hub == "openai":+ model_or_path = kwargs.get("model_path", "Whisper-large-v3")+ if model_or_path.startswith("Whisper-"):+ model_or_path = model_or_path.replace("Whisper-", "")+ model = whisper.load_model(model_or_path)+ else:+ dims = kwargs.get("dims", {})+ dims = whisper.model.ModelDimensions(**dims)+ model = whisper.model.Whisper(dims=dims)++ self.model = model+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/whisper/model.py#L33" target="_blank" class="src-more">View full source on GitHub →</a></details><h3 class="mt">Methods</h3><div class="mblk"><code>.forward()</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/whisper/model.py#L66" target="_blank" class="gh-link">L66</a><div class="mdoc"><p>Forward pass for training.</p>+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def forward(+ self,+ ):+ """Forward pass for training."""+ pass+</code></pre></details></div><div class="mblk"><code>.inference(data_in, data_lengths, key, tokenizer, frontend, **kwargs)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/whisper/model.py#L72" target="_blank" class="gh-link">L72</a><div class="mdoc"><p>Run inference on input data.</p>+<h4>Args:</h4>+<ul>+<li><code>data_in</code> — Input data (audio samples, file paths, or text).</li>+<li><code>data_lengths</code> — Lengths of each input sample in the batch.</li>+<li><code>key</code> — Sample identifiers.</li>+<li><code>tokenizer</code> — Tokenizer instance for text encoding/decoding.</li>+<li><code>frontend</code> — Audio frontend for feature extraction.</li>+<li><code>**kwargs</code> — Additional keyword arguments.</li>+</ul>+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def inference(+ self,+ data_in,+ data_lengths=None,+ key: list = None,+ tokenizer=None,+ frontend=None,+ **kwargs,+ ):+ """Run inference on input data.++ Args:+ data_in: Input data (audio samples, file paths, or text).+ data_lengths: Lengths of each input sample in the batch.+ key: Sample identifiers.+ tokenizer: Tokenizer instance for text encoding/decoding.+ frontend: Audio frontend for feature extraction.+ **kwargs: Additional keyword arguments.+ """+ if kwargs.get("batch_size", 1) > 1:+ raise NotImplementedError("batch decoding is not implemented")++ if frontend is None and not hasattr(self, "frontend"):+ frontend_class = tables.frontend_classes.get("WhisperFrontend")+ frontend = frontend_class(+ n_mels=self.model.dims.n_mels, do_pad_trim=kwargs.get("do_pad_trim", True)+ )+ self.frontend = frontend+ else:+ frontend = frontend if frontend is not None else self.frontend+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/whisper/model.py#L72" target="_blank" class="src-more">View full source →</a></details></div></div>+<div class="api-detail" id="e259"><span class="dtag badge-m">method</span><h2><code>WhisperWarp.forward()</code></h2><div class="dmod">funasr.models.whisper.WhisperWarp · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/whisper/model.py#L66" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Forward pass for training.</p>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def forward(+ self,+ ):+ """Forward pass for training."""+ pass+</code></pre></details></div>+<div class="api-detail" id="e260"><span class="dtag badge-m">method</span><h2><code>WhisperWarp.inference(data_in, data_lengths, key, tokenizer, frontend, **kwargs)</code></h2><div class="dmod">funasr.models.whisper.WhisperWarp · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/whisper/model.py#L72" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Run inference on input data.</p>+<h4>Args:</h4>+<ul>+<li><code>data_in</code> — Input data (audio samples, file paths, or text).</li>+<li><code>data_lengths</code> — Lengths of each input sample in the batch.</li>+<li><code>key</code> — Sample identifiers.</li>+<li><code>tokenizer</code> — Tokenizer instance for text encoding/decoding.</li>+<li><code>frontend</code> — Audio frontend for feature extraction.</li>+<li><code>**kwargs</code> — Additional keyword arguments.</li>+</ul>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def inference(+ self,+ data_in,+ data_lengths=None,+ key: list = None,+ tokenizer=None,+ frontend=None,+ **kwargs,+ ):+ """Run inference on input data.++ Args:+ data_in: Input data (audio samples, file paths, or text).+ data_lengths: Lengths of each input sample in the batch.+ key: Sample identifiers.+ tokenizer: Tokenizer instance for text encoding/decoding.+ frontend: Audio frontend for feature extraction.+ **kwargs: Additional keyword arguments.+ """+ if kwargs.get("batch_size", 1) > 1:+ raise NotImplementedError("batch decoding is not implemented")++ if frontend is None and not hasattr(self, "frontend"):+ frontend_class = tables.frontend_classes.get("WhisperFrontend")+ frontend = frontend_class(+ n_mels=self.model.dims.n_mels, do_pad_trim=kwargs.get("do_pad_trim", True)+ )+ self.frontend = frontend+ else:+ frontend = frontend if frontend is not None else self.frontend+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/whisper/model.py#L72" target="_blank" class="src-more">View full source on GitHub →</a></details></div>+<div class="api-detail" id="e261"><span class="dtag badge-c">class</span><h2>OpenAIWhisperModel</h2><div class="dmod">funasr.models.whisper_lid · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/whisper_lid/model.py#L22" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><ul>+<li><code>Whisper</code> — LID: Language Identification using OpenAI Whisper encoder.</li>+</ul>+<p>Detects spoken language from audio input. Supports 99 languages.</p>+<p>Uses Whisper encoder features with a classification head.</p>+<ul>+<li><code>Output</code> — {"key": str, "text": str (language code)}</li>+</ul>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code>class OpenAIWhisperModel(nn.Module):+ """Whisper-LID: Language Identification using OpenAI Whisper encoder.++ Detects spoken language from audio input. Supports 99 languages.+ Uses Whisper encoder features with a classification head.++ Output: {"key": str, "text": str (language code)}+ """++ def __init__(+ self,+ specaug: str = None,+ specaug_conf: dict = None,+ normalize: str = None,+ normalize_conf: dict = None,+ encoder: str = None,+ encoder_conf: dict = None,+ decoder: str = None,+ decoder_conf: dict = None,+ ctc: str = None,+ ctc_conf: dict = None,+ ctc_weight: float = 0.5,+ interctc_weight: float = 0.0,+ input_size: int = 80,+ vocab_size: int = -1,+ ignore_id: int = -1,+ blank_id: int = 0,+ sos: int = 1,+ eos: int = 2,+ lsm_weight: float = 0.0,+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/whisper_lid/model.py#L22" target="_blank" class="src-more">View full source on GitHub →</a></details><h3 class="mt">Methods</h3><div class="mblk"><code>.forward(speech, speech_lengths, text, text_lengths, **kwargs)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/whisper_lid/model.py#L164" target="_blank" class="gh-link">L164</a><div class="mdoc"><p>Encoder + Decoder + Calc loss</p>+<h4>Args:</h4>+<ul>+<li><code>speech</code> — (Batch, Length, ...)</li>+<li><code>speech_lengths</code> — (Batch, )</li>+<li><code>text</code> — (Batch, Length)</li>+<li><code>text_lengths</code> — (Batch,)</li>+</ul>+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def forward(+ self,+ speech: torch.Tensor,+ speech_lengths: torch.Tensor,+ text: torch.Tensor,+ text_lengths: torch.Tensor,+ **kwargs,+ ) -> Tuple[torch.Tensor, Dict[str, torch.Tensor], torch.Tensor]:+ """Encoder + Decoder + Calc loss+ Args:+ speech: (Batch, Length, ...)+ speech_lengths: (Batch, )+ text: (Batch, Length)+ text_lengths: (Batch,)+ """+ if len(text_lengths.size()) > 1:+ text_lengths = text_lengths[:, 0]+ if len(speech_lengths.size()) > 1:+ speech_lengths = speech_lengths[:, 0]++ batch_size = speech.shape[0]++ # 1. Encoder+ encoder_out, encoder_out_lens = self.encode(speech, speech_lengths)+ intermediate_outs = None+ if isinstance(encoder_out, tuple):+ intermediate_outs = encoder_out[1]+ encoder_out = encoder_out[0]++ loss_att, acc_att, cer_att, wer_att = None, None, None, None+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/whisper_lid/model.py#L164" target="_blank" class="src-more">View full source →</a></details></div><div class="mblk"><code>.encode(speech, speech_lengths, **kwargs)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/whisper_lid/model.py#L257" target="_blank" class="gh-link">L257</a><div class="mdoc"><p>Frontend + Encoder. Note that this method is used by asr_inference.py</p>+<h4>Args:</h4>+<ul>+<li><code>speech</code> — (Batch, Length, ...)</li>+<li><code>speech_lengths</code> — (Batch, )</li>+<li><code>ind</code> — int</li>+</ul>+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def encode(+ self,+ speech: torch.Tensor,+ speech_lengths: torch.Tensor,+ **kwargs,+ ) -> Tuple[torch.Tensor, torch.Tensor]:+ """Frontend + Encoder. Note that this method is used by asr_inference.py+ Args:+ speech: (Batch, Length, ...)+ speech_lengths: (Batch, )+ ind: int+ """+ with autocast(False):++ # Data augmentation+ if self.specaug is not None and self.training:+ speech, speech_lengths = self.specaug(speech, speech_lengths)++ # Normalization for feature: e.g. Global-CMVN, Utterance-CMVN+ if self.normalize is not None:+ speech, speech_lengths = self.normalize(speech, speech_lengths)++ # Forward encoder+ # feats: (Batch, Length, Dim)+ # -> encoder_out: (Batch, Length2, Dim2)+ if self.encoder.interctc_use_conditioning:+ encoder_out, encoder_out_lens, _ = self.encoder(speech, speech_lengths, ctc=self.ctc)+ else:+ encoder_out, encoder_out_lens, _ = self.encoder(speech, speech_lengths)+ intermediate_outs = None+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/whisper_lid/model.py#L257" target="_blank" class="src-more">View full source →</a></details></div><div class="mblk"><code>.init_beam_search(**kwargs)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/whisper_lid/model.py#L359" target="_blank" class="gh-link">L359</a><div class="mdoc"><p>Init beam search.</p>+<h4>Args:</h4>+<ul>+<li><code>**kwargs</code> — Additional keyword arguments.</li>+</ul>+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def init_beam_search(+ self,+ **kwargs,+ ):+ """Init beam search.++ Args:+ **kwargs: Additional keyword arguments.+ """+ from funasr.models.transformer.search import BeamSearch+ from funasr.models.transformer.scorers.ctc import CTCPrefixScorer+ from funasr.models.transformer.scorers.length_bonus import LengthBonus++ # 1. Build ASR model+ scorers = {}++ if self.ctc != None:+ ctc = CTCPrefixScorer(ctc=self.ctc, eos=self.eos)+ scorers.update(ctc=ctc)+ token_list = kwargs.get("token_list")+ scorers.update(+ decoder=self.decoder,+ length_bonus=LengthBonus(len(token_list)),+ )++ # 3. Build ngram model+ # ngram is not supported now+ ngram = None+ scorers["ngram"] = ngram++</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/whisper_lid/model.py#L359" target="_blank" class="src-more">View full source →</a></details></div><div class="mblk"><code>.inference(data_in, data_lengths, key, tokenizer, frontend, **kwargs)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/whisper_lid/model.py#L409" target="_blank" class="gh-link">L409</a><div class="mdoc"><p>Run inference on input data.</p>+<h4>Args:</h4>+<ul>+<li><code>data_in</code> — Input data (audio samples, file paths, or text).</li>+<li><code>data_lengths</code> — Lengths of each input sample in the batch.</li>+<li><code>key</code> — Sample identifiers.</li>+<li><code>tokenizer</code> — Tokenizer instance for text encoding/decoding.</li>+<li><code>frontend</code> — Audio frontend for feature extraction.</li>+<li><code>**kwargs</code> — Additional keyword arguments.</li>+</ul>+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def inference(+ self,+ data_in,+ data_lengths=None,+ key: list = None,+ tokenizer=None,+ frontend=None,+ **kwargs,+ ):++ """Run inference on input data.++ Args:+ data_in: Input data (audio samples, file paths, or text).+ data_lengths: Lengths of each input sample in the batch.+ key: Sample identifiers.+ tokenizer: Tokenizer instance for text encoding/decoding.+ frontend: Audio frontend for feature extraction.+ **kwargs: Additional keyword arguments.+ """+ if kwargs.get("batch_size", 1) > 1:+ raise NotImplementedError("batch decoding is not implemented")++ # init beamsearch+ if self.beam_search is None:+ logging.info("enable beam_search")+ self.init_beam_search(**kwargs)+ self.nbest = kwargs.get("nbest", 1)++ meta_data = {}+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/whisper_lid/model.py#L409" target="_blank" class="src-more">View full source →</a></details></div></div>+<div class="api-detail" id="e262"><span class="dtag badge-m">method</span><h2><code>OpenAIWhisperModel.forward(speech, speech_lengths, text, text_lengths, **kwargs)</code></h2><div class="dmod">funasr.models.whisper_lid.OpenAIWhisperModel · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/whisper_lid/model.py#L164" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Encoder + Decoder + Calc loss</p>+<h4>Args:</h4>+<ul>+<li><code>speech</code> — (Batch, Length, ...)</li>+<li><code>speech_lengths</code> — (Batch, )</li>+<li><code>text</code> — (Batch, Length)</li>+<li><code>text_lengths</code> — (Batch,)</li>+</ul>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def forward(+ self,+ speech: torch.Tensor,+ speech_lengths: torch.Tensor,+ text: torch.Tensor,+ text_lengths: torch.Tensor,+ **kwargs,+ ) -> Tuple[torch.Tensor, Dict[str, torch.Tensor], torch.Tensor]:+ """Encoder + Decoder + Calc loss+ Args:+ speech: (Batch, Length, ...)+ speech_lengths: (Batch, )+ text: (Batch, Length)+ text_lengths: (Batch,)+ """+ if len(text_lengths.size()) > 1:+ text_lengths = text_lengths[:, 0]+ if len(speech_lengths.size()) > 1:+ speech_lengths = speech_lengths[:, 0]++ batch_size = speech.shape[0]++ # 1. Encoder+ encoder_out, encoder_out_lens = self.encode(speech, speech_lengths)+ intermediate_outs = None+ if isinstance(encoder_out, tuple):+ intermediate_outs = encoder_out[1]+ encoder_out = encoder_out[0]++ loss_att, acc_att, cer_att, wer_att = None, None, None, None+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/whisper_lid/model.py#L164" target="_blank" class="src-more">View full source on GitHub →</a></details></div>+<div class="api-detail" id="e263"><span class="dtag badge-m">method</span><h2><code>OpenAIWhisperModel.encode(speech, speech_lengths, **kwargs)</code></h2><div class="dmod">funasr.models.whisper_lid.OpenAIWhisperModel · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/whisper_lid/model.py#L257" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Frontend + Encoder. Note that this method is used by asr_inference.py</p>+<h4>Args:</h4>+<ul>+<li><code>speech</code> — (Batch, Length, ...)</li>+<li><code>speech_lengths</code> — (Batch, )</li>+<li><code>ind</code> — int</li>+</ul>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def encode(+ self,+ speech: torch.Tensor,+ speech_lengths: torch.Tensor,+ **kwargs,+ ) -> Tuple[torch.Tensor, torch.Tensor]:+ """Frontend + Encoder. Note that this method is used by asr_inference.py+ Args:+ speech: (Batch, Length, ...)+ speech_lengths: (Batch, )+ ind: int+ """+ with autocast(False):++ # Data augmentation+ if self.specaug is not None and self.training:+ speech, speech_lengths = self.specaug(speech, speech_lengths)++ # Normalization for feature: e.g. Global-CMVN, Utterance-CMVN+ if self.normalize is not None:+ speech, speech_lengths = self.normalize(speech, speech_lengths)++ # Forward encoder+ # feats: (Batch, Length, Dim)+ # -> encoder_out: (Batch, Length2, Dim2)+ if self.encoder.interctc_use_conditioning:+ encoder_out, encoder_out_lens, _ = self.encoder(speech, speech_lengths, ctc=self.ctc)+ else:+ encoder_out, encoder_out_lens, _ = self.encoder(speech, speech_lengths)+ intermediate_outs = None+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/whisper_lid/model.py#L257" target="_blank" class="src-more">View full source on GitHub →</a></details></div>+<div class="api-detail" id="e264"><span class="dtag badge-m">method</span><h2><code>OpenAIWhisperModel.init_beam_search(**kwargs)</code></h2><div class="dmod">funasr.models.whisper_lid.OpenAIWhisperModel · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/whisper_lid/model.py#L359" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Init beam search.</p>+<h4>Args:</h4>+<ul>+<li><code>**kwargs</code> — Additional keyword arguments.</li>+</ul>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def init_beam_search(+ self,+ **kwargs,+ ):+ """Init beam search.++ Args:+ **kwargs: Additional keyword arguments.+ """+ from funasr.models.transformer.search import BeamSearch+ from funasr.models.transformer.scorers.ctc import CTCPrefixScorer+ from funasr.models.transformer.scorers.length_bonus import LengthBonus++ # 1. Build ASR model+ scorers = {}++ if self.ctc != None:+ ctc = CTCPrefixScorer(ctc=self.ctc, eos=self.eos)+ scorers.update(ctc=ctc)+ token_list = kwargs.get("token_list")+ scorers.update(+ decoder=self.decoder,+ length_bonus=LengthBonus(len(token_list)),+ )++ # 3. Build ngram model+ # ngram is not supported now+ ngram = None+ scorers["ngram"] = ngram++</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/whisper_lid/model.py#L359" target="_blank" class="src-more">View full source on GitHub →</a></details></div>+<div class="api-detail" id="e265"><span class="dtag badge-m">method</span><h2><code>OpenAIWhisperModel.inference(data_in, data_lengths, key, tokenizer, frontend, **kwargs)</code></h2><div class="dmod">funasr.models.whisper_lid.OpenAIWhisperModel · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/whisper_lid/model.py#L409" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Run inference on input data.</p>+<h4>Args:</h4>+<ul>+<li><code>data_in</code> — Input data (audio samples, file paths, or text).</li>+<li><code>data_lengths</code> — Lengths of each input sample in the batch.</li>+<li><code>key</code> — Sample identifiers.</li>+<li><code>tokenizer</code> — Tokenizer instance for text encoding/decoding.</li>+<li><code>frontend</code> — Audio frontend for feature extraction.</li>+<li><code>**kwargs</code> — Additional keyword arguments.</li>+</ul>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def inference(+ self,+ data_in,+ data_lengths=None,+ key: list = None,+ tokenizer=None,+ frontend=None,+ **kwargs,+ ):++ """Run inference on input data.++ Args:+ data_in: Input data (audio samples, file paths, or text).+ data_lengths: Lengths of each input sample in the batch.+ key: Sample identifiers.+ tokenizer: Tokenizer instance for text encoding/decoding.+ frontend: Audio frontend for feature extraction.+ **kwargs: Additional keyword arguments.+ """+ if kwargs.get("batch_size", 1) > 1:+ raise NotImplementedError("batch decoding is not implemented")++ # init beamsearch+ if self.beam_search is None:+ logging.info("enable beam_search")+ self.init_beam_search(**kwargs)+ self.nbest = kwargs.get("nbest", 1)++ meta_data = {}+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/whisper_lid/model.py#L409" target="_blank" class="src-more">View full source on GitHub →</a></details></div>+<div class="api-detail" id="e266"><span class="dtag badge-c">class</span><h2>OpenAIWhisperLIDModel</h2><div class="dmod">funasr.models.whisper_lid · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/whisper_lid/model.py#L525" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>WhisperEncoder and EResNet based LID Model</p>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code>class OpenAIWhisperLIDModel(nn.Module):+ """WhisperEncoder and EResNet based LID Model"""++ def __init__(+ self,+ vocab_size: int,+ specaug: str = None,+ specaug_conf: dict = None,+ encoder: str = None,+ encoder_conf: dict = None,+ lid_predictor: str = None,+ lid_predictor_conf: dict = None,+ proj_dim: int = None,+ clip_frames: int = None,+ random_clip: bool = False,+ **kwargs,+ ):+ """Initialize OpenAIWhisperLIDModel.++ Args:+ vocab_size: Size/dimension parameter.+ specaug: TODO.+ specaug_conf: Configuration dict for specaug.+ encoder: TODO.+ encoder_conf: Configuration dict for encoder.+ lid_predictor: TODO.+ lid_predictor_conf: Configuration dict for lid_predictor.+ proj_dim: Size/dimension parameter.+ clip_frames: TODO.+ random_clip: TODO.+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/whisper_lid/model.py#L525" target="_blank" class="src-more">View full source on GitHub →</a></details><h3 class="mt">Methods</h3><div class="mblk"><code>.forward(speech, speech_lengths, lid, lid_lengths)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/whisper_lid/model.py#L587" target="_blank" class="gh-link">L587</a><div class="mdoc"><p>Forward pass for training.</p>+<h4>Args:</h4>+<ul>+<li><code>speech</code> — Speech audio tensor, shape (batch, time).</li>+<li><code>speech_lengths</code> — Length of each speech sample.</li>+<li><code>lid</code> — TODO.</li>+<li><code>lid_lengths</code> — Lengths of lid.</li>+</ul>+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def forward(+ self,+ speech: torch.Tensor, # may be padding+ speech_lengths: torch.Tensor, # actual length+ lid: torch.Tensor, # lid label, (batch_size, 1)+ lid_lengths: torch.Tensor,+ ):+ """Forward pass for training.++ Args:+ speech: Speech audio tensor, shape (batch, time).+ speech_lengths: Length of each speech sample.+ lid: TODO.+ lid_lengths: Lengths of lid.+ """+ assert lid.shape[1] == 1+ batch_size = speech.shape[0]+ encoder_out, encoder_out_lens = self.encode(speech, speech_lengths)++ # re-generate encoder_out+ if self.clip_frames is None:+ reduced_encoder_out = (+ torch.zeros(batch_size, encoder_out_lens.max(), encoder_out.shape[-1])+ .to(encoder_out.dtype)+ .to(encoder_out.device)+ )+ for i, enc_length in enumerate(encoder_out_lens):+ reduced_encoder_out[i, :enc_length] = encoder_out[i, :enc_length]+ else:+ reduced_encoder_out = (+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/whisper_lid/model.py#L587" target="_blank" class="src-more">View full source →</a></details></div><div class="mblk"><code>.encode(speech, speech_lengths)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/whisper_lid/model.py#L655" target="_blank" class="gh-link">L655</a><div class="mdoc"><p>Frontend + Encoder. Note that this method is used by asr_inference.py</p>+<h4>Args:</h4>+<ul>+<li><code>speech</code> — (Batch, Length, ...)</li>+<li><code>speech_lengths</code> — (Batch, )</li>+</ul>+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def encode(+ self, speech: torch.Tensor, speech_lengths: torch.Tensor+ ) -> Tuple[torch.Tensor, torch.Tensor]:+ """Frontend + Encoder. Note that this method is used by asr_inference.py+ Args:+ speech: (Batch, Length, ...)+ speech_lengths: (Batch, )+ """+ with autocast(False):++ # Data augmentation+ if self.specaug is not None and self.training:+ speech = speech.permute(0, 2, 1)+ # suit for whisper padding+ padded_speech_lengths = torch.ones_like(speech_lengths) * speech.shape[1]+ speech, padded_speech_lengths = self.specaug(speech, padded_speech_lengths)+ speech = speech.permute(0, 2, 1)++ # Normalization for feature: e.g. Global-CMVN, Utterance-CMVN+ if self.normalize is not None:+ speech, speech_lengths = self.normalize(speech, speech_lengths)++ # Forward encoder+ # feats: (Batch, Length, Dim)+ # -> encoder_out: (Batch, Length2, Dim2)+ if self.encoder.interctc_use_conditioning:+ encoder_out, encoder_out_lens, _ = self.encoder(speech, speech_lengths, ctc=self.ctc)+ else:+ encoder_out, encoder_out_lens, _ = self.encoder(speech, speech_lengths)+ intermediate_outs = None+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/whisper_lid/model.py#L655" target="_blank" class="src-more">View full source →</a></details></div><div class="mblk"><code>.inference(data_in, data_lengths, key, tokenizer, frontend, **kwargs)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/whisper_lid/model.py#L694" target="_blank" class="gh-link">L694</a><div class="mdoc"><p>Run inference on input data.</p>+<h4>Args:</h4>+<ul>+<li><code>data_in</code> — Input data (audio samples, file paths, or text).</li>+<li><code>data_lengths</code> — Lengths of each input sample in the batch.</li>+<li><code>key</code> — Sample identifiers.</li>+<li><code>tokenizer</code> — Tokenizer instance for text encoding/decoding.</li>+<li><code>frontend</code> — Audio frontend for feature extraction.</li>+<li><code>**kwargs</code> — Additional keyword arguments.</li>+</ul>+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def inference(+ self,+ data_in,+ data_lengths=None,+ key: list = None,+ tokenizer=None,+ frontend=None,+ **kwargs,+ ):++ """Run inference on input data.++ Args:+ data_in: Input data (audio samples, file paths, or text).+ data_lengths: Lengths of each input sample in the batch.+ key: Sample identifiers.+ tokenizer: Tokenizer instance for text encoding/decoding.+ frontend: Audio frontend for feature extraction.+ **kwargs: Additional keyword arguments.+ """+ if kwargs.get("batch_size", 1) > 1:+ raise NotImplementedError("batch decoding is not implemented")++ meta_data = {}+ if (+ isinstance(data_in, torch.Tensor) and kwargs.get("data_type", "sound") == "fbank"+ ): # fbank+ speech, speech_lengths = data_in, data_lengths+ if len(speech.shape) < 3:+ speech = speech[None, :, :]+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/whisper_lid/model.py#L694" target="_blank" class="src-more">View full source →</a></details></div></div>+<div class="api-detail" id="e267"><span class="dtag badge-m">method</span><h2><code>OpenAIWhisperLIDModel.forward(speech, speech_lengths, lid, lid_lengths)</code></h2><div class="dmod">funasr.models.whisper_lid.OpenAIWhisperLIDModel · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/whisper_lid/model.py#L587" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Forward pass for training.</p>+<h4>Args:</h4>+<ul>+<li><code>speech</code> — Speech audio tensor, shape (batch, time).</li>+<li><code>speech_lengths</code> — Length of each speech sample.</li>+<li><code>lid</code> — TODO.</li>+<li><code>lid_lengths</code> — Lengths of lid.</li>+</ul>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def forward(+ self,+ speech: torch.Tensor, # may be padding+ speech_lengths: torch.Tensor, # actual length+ lid: torch.Tensor, # lid label, (batch_size, 1)+ lid_lengths: torch.Tensor,+ ):+ """Forward pass for training.++ Args:+ speech: Speech audio tensor, shape (batch, time).+ speech_lengths: Length of each speech sample.+ lid: TODO.+ lid_lengths: Lengths of lid.+ """+ assert lid.shape[1] == 1+ batch_size = speech.shape[0]+ encoder_out, encoder_out_lens = self.encode(speech, speech_lengths)++ # re-generate encoder_out+ if self.clip_frames is None:+ reduced_encoder_out = (+ torch.zeros(batch_size, encoder_out_lens.max(), encoder_out.shape[-1])+ .to(encoder_out.dtype)+ .to(encoder_out.device)+ )+ for i, enc_length in enumerate(encoder_out_lens):+ reduced_encoder_out[i, :enc_length] = encoder_out[i, :enc_length]+ else:+ reduced_encoder_out = (+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/whisper_lid/model.py#L587" target="_blank" class="src-more">View full source on GitHub →</a></details></div>+<div class="api-detail" id="e268"><span class="dtag badge-m">method</span><h2><code>OpenAIWhisperLIDModel.encode(speech, speech_lengths)</code></h2><div class="dmod">funasr.models.whisper_lid.OpenAIWhisperLIDModel · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/whisper_lid/model.py#L655" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Frontend + Encoder. Note that this method is used by asr_inference.py</p>+<h4>Args:</h4>+<ul>+<li><code>speech</code> — (Batch, Length, ...)</li>+<li><code>speech_lengths</code> — (Batch, )</li>+</ul>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def encode(+ self, speech: torch.Tensor, speech_lengths: torch.Tensor+ ) -> Tuple[torch.Tensor, torch.Tensor]:+ """Frontend + Encoder. Note that this method is used by asr_inference.py+ Args:+ speech: (Batch, Length, ...)+ speech_lengths: (Batch, )+ """+ with autocast(False):++ # Data augmentation+ if self.specaug is not None and self.training:+ speech = speech.permute(0, 2, 1)+ # suit for whisper padding+ padded_speech_lengths = torch.ones_like(speech_lengths) * speech.shape[1]+ speech, padded_speech_lengths = self.specaug(speech, padded_speech_lengths)+ speech = speech.permute(0, 2, 1)++ # Normalization for feature: e.g. Global-CMVN, Utterance-CMVN+ if self.normalize is not None:+ speech, speech_lengths = self.normalize(speech, speech_lengths)++ # Forward encoder+ # feats: (Batch, Length, Dim)+ # -> encoder_out: (Batch, Length2, Dim2)+ if self.encoder.interctc_use_conditioning:+ encoder_out, encoder_out_lens, _ = self.encoder(speech, speech_lengths, ctc=self.ctc)+ else:+ encoder_out, encoder_out_lens, _ = self.encoder(speech, speech_lengths)+ intermediate_outs = None+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/whisper_lid/model.py#L655" target="_blank" class="src-more">View full source on GitHub →</a></details></div>+<div class="api-detail" id="e269"><span class="dtag badge-m">method</span><h2><code>OpenAIWhisperLIDModel.inference(data_in, data_lengths, key, tokenizer, frontend, **kwargs)</code></h2><div class="dmod">funasr.models.whisper_lid.OpenAIWhisperLIDModel · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/whisper_lid/model.py#L694" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Run inference on input data.</p>+<h4>Args:</h4>+<ul>+<li><code>data_in</code> — Input data (audio samples, file paths, or text).</li>+<li><code>data_lengths</code> — Lengths of each input sample in the batch.</li>+<li><code>key</code> — Sample identifiers.</li>+<li><code>tokenizer</code> — Tokenizer instance for text encoding/decoding.</li>+<li><code>frontend</code> — Audio frontend for feature extraction.</li>+<li><code>**kwargs</code> — Additional keyword arguments.</li>+</ul>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def inference(+ self,+ data_in,+ data_lengths=None,+ key: list = None,+ tokenizer=None,+ frontend=None,+ **kwargs,+ ):++ """Run inference on input data.++ Args:+ data_in: Input data (audio samples, file paths, or text).+ data_lengths: Lengths of each input sample in the batch.+ key: Sample identifiers.+ tokenizer: Tokenizer instance for text encoding/decoding.+ frontend: Audio frontend for feature extraction.+ **kwargs: Additional keyword arguments.+ """+ if kwargs.get("batch_size", 1) > 1:+ raise NotImplementedError("batch decoding is not implemented")++ meta_data = {}+ if (+ isinstance(data_in, torch.Tensor) and kwargs.get("data_type", "sound") == "fbank"+ ): # fbank+ speech, speech_lengths = data_in, data_lengths+ if len(speech.shape) < 3:+ speech = speech[None, :, :]+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/models/whisper_lid/model.py#L694" target="_blank" class="src-more">View full source on GitHub →</a></details></div>+<div class="api-detail" id="e270"><span class="dtag badge-c">class</span><h2>DefaultFrontend</h2><div class="dmod">funasr.frontends.default · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/frontends/default.py#L24" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Conventional frontend structure for ASR.</p>+<ul>+<li><code>Stft</code> — > WPE -> MVDR-Beamformer -> Power-spec -> Mel-Fbank -> CMVN</li>+</ul>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code>class DefaultFrontend(nn.Module):+ """Conventional frontend structure for ASR.+ Stft -> WPE -> MVDR-Beamformer -> Power-spec -> Mel-Fbank -> CMVN+ """++ def __init__(+ self,+ fs: int = 16000,+ n_fft: int = 512,+ win_length: int = None,+ hop_length: int = 128,+ window: Optional[str] = "hann",+ center: bool = True,+ normalized: bool = False,+ onesided: bool = True,+ n_mels: int = 80,+ fmin: int = None,+ fmax: int = None,+ htk: bool = False,+ frontend_conf: Optional[dict] = None,+ apply_stft: bool = True,+ use_channel: int = None,+ **kwargs,+ ):+ """Initialize DefaultFrontend.++ Args:+ fs: TODO.+ n_fft: TODO.+ win_length: TODO.+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/frontends/default.py#L24" target="_blank" class="src-more">View full source on GitHub →</a></details><h3 class="mt">Methods</h3><div class="mblk"><code>.output_size()</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/frontends/default.py#L106" target="_blank" class="gh-link">L106</a><div class="mdoc"><p>Output size.</p>+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def output_size(self) -> int:+ """Output size."""+ return self.n_mels+</code></pre></details></div><div class="mblk"><code>.forward(input, input_lengths)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/frontends/default.py#L110" target="_blank" class="gh-link">L110</a><div class="mdoc"><p>Forward pass for training.</p>+<h4>Args:</h4>+<ul>+<li><code>input</code> — Input audio/text data.</li>+<li><code>input_lengths</code> — Lengths of input.</li>+</ul>+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def forward(+ self, input: torch.Tensor, input_lengths: Union[torch.Tensor, list]+ ) -> Tuple[torch.Tensor, torch.Tensor]:+ """Forward pass for training.++ Args:+ input: Input audio/text data.+ input_lengths: Lengths of input.+ """+ if isinstance(input_lengths, list):+ input_lengths = torch.tensor(input_lengths)+ if input.dtype == torch.float64:+ input = input.float()+ # 1. Domain-conversion: e.g. Stft: time -> time-freq+ if self.stft is not None:+ input_stft, feats_lens = self._compute_stft(input, input_lengths)+ else:+ input_stft = ComplexTensor(input[..., 0], input[..., 1])+ feats_lens = input_lengths+ # 2. [Option] Speech enhancement+ if self.frontend is not None:+ assert isinstance(input_stft, ComplexTensor), type(input_stft)+ # input_stft: (Batch, Length, [Channel], Freq)+ input_stft, _, mask = self.frontend(input_stft, feats_lens)++ # 3. [Multi channel case]: Select a channel+ if input_stft.dim() == 4:+ # h: (B, T, C, F) -> h: (B, T, F)+ if self.training:+ if self.use_channel is not None:+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/frontends/default.py#L110" target="_blank" class="src-more">View full source →</a></details></div></div>+<div class="api-detail" id="e271"><span class="dtag badge-m">method</span><h2><code>DefaultFrontend.output_size()</code></h2><div class="dmod">funasr.frontends.default.DefaultFrontend · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/frontends/default.py#L106" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Output size.</p>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def output_size(self) -> int:+ """Output size."""+ return self.n_mels+</code></pre></details></div>+<div class="api-detail" id="e272"><span class="dtag badge-m">method</span><h2><code>DefaultFrontend.forward(input, input_lengths)</code></h2><div class="dmod">funasr.frontends.default.DefaultFrontend · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/frontends/default.py#L110" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Forward pass for training.</p>+<h4>Args:</h4>+<ul>+<li><code>input</code> — Input audio/text data.</li>+<li><code>input_lengths</code> — Lengths of input.</li>+</ul>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def forward(+ self, input: torch.Tensor, input_lengths: Union[torch.Tensor, list]+ ) -> Tuple[torch.Tensor, torch.Tensor]:+ """Forward pass for training.++ Args:+ input: Input audio/text data.+ input_lengths: Lengths of input.+ """+ if isinstance(input_lengths, list):+ input_lengths = torch.tensor(input_lengths)+ if input.dtype == torch.float64:+ input = input.float()+ # 1. Domain-conversion: e.g. Stft: time -> time-freq+ if self.stft is not None:+ input_stft, feats_lens = self._compute_stft(input, input_lengths)+ else:+ input_stft = ComplexTensor(input[..., 0], input[..., 1])+ feats_lens = input_lengths+ # 2. [Option] Speech enhancement+ if self.frontend is not None:+ assert isinstance(input_stft, ComplexTensor), type(input_stft)+ # input_stft: (Batch, Length, [Channel], Freq)+ input_stft, _, mask = self.frontend(input_stft, feats_lens)++ # 3. [Multi channel case]: Select a channel+ if input_stft.dim() == 4:+ # h: (B, T, C, F) -> h: (B, T, F)+ if self.training:+ if self.use_channel is not None:+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/frontends/default.py#L110" target="_blank" class="src-more">View full source on GitHub →</a></details></div>+<div class="api-detail" id="e273"><span class="dtag badge-c">class</span><h2>MultiChannelFrontend</h2><div class="dmod">funasr.frontends.default · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/frontends/default.py#L179" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Conventional frontend structure for ASR.</p>+<ul>+<li><code>Stft</code> — > WPE -> MVDR-Beamformer -> Power-spec -> Mel-Fbank -> CMVN</li>+</ul>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code>class MultiChannelFrontend(nn.Module):+ """Conventional frontend structure for ASR.+ Stft -> WPE -> MVDR-Beamformer -> Power-spec -> Mel-Fbank -> CMVN+ """++ def __init__(+ self,+ fs: int = 16000,+ n_fft: int = 512,+ win_length: int = None,+ hop_length: int = None,+ frame_length: int = None,+ frame_shift: int = None,+ window: Optional[str] = "hann",+ center: bool = True,+ normalized: bool = False,+ onesided: bool = True,+ n_mels: int = 80,+ fmin: int = None,+ fmax: int = None,+ htk: bool = False,+ frontend_conf: Optional[dict] = None,+ apply_stft: bool = True,+ use_channel: int = None,+ lfr_m: int = 1,+ lfr_n: int = 1,+ cmvn_file: str = None,+ mc: bool = True,+ ):+ """Initialize MultiChannelFrontend.+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/frontends/default.py#L179" target="_blank" class="src-more">View full source on GitHub →</a></details><h3 class="mt">Methods</h3><div class="mblk"><code>.output_size()</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/frontends/default.py#L290" target="_blank" class="gh-link">L290</a><div class="mdoc"><p>Output size.</p>+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def output_size(self) -> int:+ """Output size."""+ return self.n_mels+</code></pre></details></div><div class="mblk"><code>.forward(input, input_lengths)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/frontends/default.py#L294" target="_blank" class="gh-link">L294</a><div class="mdoc"><p>Forward pass for training.</p>+<h4>Args:</h4>+<ul>+<li><code>input</code> — Input audio/text data.</li>+<li><code>input_lengths</code> — Lengths of input.</li>+</ul>+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def forward(+ self, input: torch.Tensor, input_lengths: torch.Tensor+ ) -> Tuple[torch.Tensor, torch.Tensor]:+ # 1. Domain-conversion: e.g. Stft: time -> time-freq+ """Forward pass for training.++ Args:+ input: Input audio/text data.+ input_lengths: Lengths of input.+ """+ if self.stft is not None:+ input_stft, feats_lens = self._compute_stft(input, input_lengths)+ else:+ input_stft = ComplexTensor(input[..., 0], input[..., 1])+ feats_lens = input_lengths+ # 2. [Option] Speech enhancement+ if self.frontend is not None:+ assert isinstance(input_stft, ComplexTensor), type(input_stft)+ # input_stft: (Batch, Length, [Channel], Freq)+ input_stft, _, mask = self.frontend(input_stft, feats_lens)++ # 3. [Multi channel case]: Select a channel(sa_asr)+ if input_stft.dim() == 4 and not self.mc:+ # h: (B, T, C, F) -> h: (B, T, F)+ if self.training:+ if self.use_channel is not None:+ input_stft = input_stft[:, :, self.use_channel, :]++ else:+ # Select 1ch randomly+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/frontends/default.py#L294" target="_blank" class="src-more">View full source →</a></details></div></div>+<div class="api-detail" id="e274"><span class="dtag badge-m">method</span><h2><code>MultiChannelFrontend.output_size()</code></h2><div class="dmod">funasr.frontends.default.MultiChannelFrontend · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/frontends/default.py#L290" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Output size.</p>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def output_size(self) -> int:+ """Output size."""+ return self.n_mels+</code></pre></details></div>+<div class="api-detail" id="e275"><span class="dtag badge-m">method</span><h2><code>MultiChannelFrontend.forward(input, input_lengths)</code></h2><div class="dmod">funasr.frontends.default.MultiChannelFrontend · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/frontends/default.py#L294" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Forward pass for training.</p>+<h4>Args:</h4>+<ul>+<li><code>input</code> — Input audio/text data.</li>+<li><code>input_lengths</code> — Lengths of input.</li>+</ul>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def forward(+ self, input: torch.Tensor, input_lengths: torch.Tensor+ ) -> Tuple[torch.Tensor, torch.Tensor]:+ # 1. Domain-conversion: e.g. Stft: time -> time-freq+ """Forward pass for training.++ Args:+ input: Input audio/text data.+ input_lengths: Lengths of input.+ """+ if self.stft is not None:+ input_stft, feats_lens = self._compute_stft(input, input_lengths)+ else:+ input_stft = ComplexTensor(input[..., 0], input[..., 1])+ feats_lens = input_lengths+ # 2. [Option] Speech enhancement+ if self.frontend is not None:+ assert isinstance(input_stft, ComplexTensor), type(input_stft)+ # input_stft: (Batch, Length, [Channel], Freq)+ input_stft, _, mask = self.frontend(input_stft, feats_lens)++ # 3. [Multi channel case]: Select a channel(sa_asr)+ if input_stft.dim() == 4 and not self.mc:+ # h: (B, T, C, F) -> h: (B, T, F)+ if self.training:+ if self.use_channel is not None:+ input_stft = input_stft[:, :, self.use_channel, :]++ else:+ # Select 1ch randomly+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/frontends/default.py#L294" target="_blank" class="src-more">View full source on GitHub →</a></details></div>+<div class="api-detail" id="e276"><span class="dtag badge-f">function</span><h2><code>transform(Y, dtype)</code></h2><div class="dmod">funasr.frontends.eend_ola_feature · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/frontends/eend_ola_feature.py#L10" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Transform.</p>+<h4>Args:</h4>+<ul>+<li><code>Y</code> — TODO.</li>+<li><code>dtype</code> — TODO.</li>+</ul>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code>def transform(Y, dtype=np.float32):+ """Transform.++ Args:+ Y: TODO.+ dtype: TODO.+ """+ Y = np.abs(Y)+ n_fft = 2 * (Y.shape[1] - 1)+ sr = 8000+ n_mels = 23+ mel_basis = librosa.filters.mel(sr, n_fft, n_mels)+ Y = np.dot(Y**2, mel_basis.T)+ Y = np.log10(np.maximum(Y, 1e-10))+ mean = np.mean(Y, axis=0)+ Y = Y - mean+ return Y.astype(dtype)+</code></pre></details></div>+<div class="api-detail" id="e277"><span class="dtag badge-f">function</span><h2><code>subsample(Y, T, subsampling)</code></h2><div class="dmod">funasr.frontends.eend_ola_feature · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/frontends/eend_ola_feature.py#L29" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Subsample.</p>+<h4>Args:</h4>+<ul>+<li><code>Y</code> — TODO.</li>+<li><code>T</code> — TODO.</li>+<li><code>subsampling</code> — TODO.</li>+</ul>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code>def subsample(Y, T, subsampling=1):+ """Subsample.++ Args:+ Y: TODO.+ T: TODO.+ subsampling: TODO.+ """+ Y_ss = Y[::subsampling]+ T_ss = T[::subsampling]+ return Y_ss, T_ss+</code></pre></details></div>+<div class="api-detail" id="e278"><span class="dtag badge-f">function</span><h2><code>splice(Y, context_size)</code></h2><div class="dmod">funasr.frontends.eend_ola_feature · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/frontends/eend_ola_feature.py#L42" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Splice.</p>+<h4>Args:</h4>+<ul>+<li><code>Y</code> — TODO.</li>+<li><code>context_size</code> — Size/dimension parameter.</li>+</ul>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code>def splice(Y, context_size=0):+ """Splice.++ Args:+ Y: TODO.+ context_size: Size/dimension parameter.+ """+ Y_pad = np.pad(Y, [(context_size, context_size), (0, 0)], "constant")+ Y_spliced = np.lib.stride_tricks.as_strided(+ np.ascontiguousarray(Y_pad),+ (Y.shape[0], Y.shape[1] * (2 * context_size + 1)),+ (Y.itemsize * Y.shape[1], Y.itemsize),+ writeable=False,+ )+ return Y_spliced+</code></pre></details></div>+<div class="api-detail" id="e279"><span class="dtag badge-f">function</span><h2><code>stft(data, frame_size, frame_shift)</code></h2><div class="dmod">funasr.frontends.eend_ola_feature · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/frontends/eend_ola_feature.py#L59" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Stft.</p>+<h4>Args:</h4>+<ul>+<li><code>data</code> — TODO.</li>+<li><code>frame_size</code> — Size/dimension parameter.</li>+<li><code>frame_shift</code> — TODO.</li>+</ul>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code>def stft(data, frame_size=1024, frame_shift=256):+ """Stft.++ Args:+ data: TODO.+ frame_size: Size/dimension parameter.+ frame_shift: TODO.+ """+ fft_size = 1 << (frame_size - 1).bit_length()+ if len(data) % frame_shift == 0:+ return librosa.stft(data, n_fft=fft_size, win_length=frame_size, hop_length=frame_shift).T[+ :-1+ ]+ else:+ return librosa.stft(data, n_fft=fft_size, win_length=frame_size, hop_length=frame_shift).T+</code></pre></details></div>+<div class="api-detail" id="e280"><span class="dtag badge-c">class</span><h2>FusedFrontends</h2><div class="dmod">funasr.frontends.fused · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/frontends/fused.py#L9" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p class="muted">No documentation yet.</p></div><details class="src-block"><summary>📄 Source code</summary><pre><code>class FusedFrontends(nn.Module):+ def __init__(self, frontends=None, align_method="linear_projection", proj_dim=100, fs=16000):++ """Initialize FusedFrontends.++ Args:+ frontends: TODO.+ align_method: TODO.+ proj_dim: Size/dimension parameter.+ fs: TODO.+ """+ super().__init__()+ self.align_method = align_method # fusing method : linear_projection only for now+ self.proj_dim = proj_dim # dim of the projection done on each frontend+ self.frontends = [] # list of the frontends to combine++ for i, frontend in enumerate(frontends):+ frontend_type = frontend["frontend_type"]+ if frontend_type == "default":+ n_mels, fs, n_fft, win_length, hop_length = (+ frontend.get("n_mels", 80),+ fs,+ frontend.get("n_fft", 512),+ frontend.get("win_length"),+ frontend.get("hop_length", 128),+ )+ window, center, normalized, onesided = (+ frontend.get("window", "hann"),+ frontend.get("center", True),+ frontend.get("normalized", False),+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/frontends/fused.py#L9" target="_blank" class="src-more">View full source on GitHub →</a></details><h3 class="mt">Methods</h3><div class="mblk"><code>.output_size()</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/frontends/fused.py#L106" target="_blank" class="gh-link">L106</a><div class="mdoc"><p>Output size.</p>+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def output_size(self) -> int:+ """Output size."""+ return len(self.frontends) * self.proj_dim+</code></pre></details></div><div class="mblk"><code>.forward(input, input_lengths)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/frontends/fused.py#L110" target="_blank" class="gh-link">L110</a><div class="mdoc"><p>Forward pass for training.</p>+<h4>Args:</h4>+<ul>+<li><code>input</code> — Input audio/text data.</li>+<li><code>input_lengths</code> — Lengths of input.</li>+</ul>+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def forward(+ self, input: torch.Tensor, input_lengths: torch.Tensor+ ) -> Tuple[torch.Tensor, torch.Tensor]:++ # step 0 : get all frontends features+ """Forward pass for training.++ Args:+ input: Input audio/text data.+ input_lengths: Lengths of input.+ """+ self.feats = []+ for frontend in self.frontends:+ with torch.no_grad():+ input_feats, feats_lens = frontend.forward(input, input_lengths)+ self.feats.append([input_feats, feats_lens])++ if self.align_method == "linear_projection": # TODO(Dan): to add other align methods++ # first step : projections+ self.feats_proj = []+ for i, frontend in enumerate(self.frontends):+ input_feats = self.feats[i][0]+ self.feats_proj.append(self.projection_layers[i](input_feats))++ # 2nd step : reshape+ self.feats_reshaped = []+ for i, frontend in enumerate(self.frontends):+ input_feats_proj = self.feats_proj[i]+ bs, nf, dim = input_feats_proj.shape+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/frontends/fused.py#L110" target="_blank" class="src-more">View full source →</a></details></div></div>+<div class="api-detail" id="e281"><span class="dtag badge-m">method</span><h2><code>FusedFrontends.output_size()</code></h2><div class="dmod">funasr.frontends.fused.FusedFrontends · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/frontends/fused.py#L106" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Output size.</p>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def output_size(self) -> int:+ """Output size."""+ return len(self.frontends) * self.proj_dim+</code></pre></details></div>+<div class="api-detail" id="e282"><span class="dtag badge-m">method</span><h2><code>FusedFrontends.forward(input, input_lengths)</code></h2><div class="dmod">funasr.frontends.fused.FusedFrontends · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/frontends/fused.py#L110" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Forward pass for training.</p>+<h4>Args:</h4>+<ul>+<li><code>input</code> — Input audio/text data.</li>+<li><code>input_lengths</code> — Lengths of input.</li>+</ul>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def forward(+ self, input: torch.Tensor, input_lengths: torch.Tensor+ ) -> Tuple[torch.Tensor, torch.Tensor]:++ # step 0 : get all frontends features+ """Forward pass for training.++ Args:+ input: Input audio/text data.+ input_lengths: Lengths of input.+ """+ self.feats = []+ for frontend in self.frontends:+ with torch.no_grad():+ input_feats, feats_lens = frontend.forward(input, input_lengths)+ self.feats.append([input_feats, feats_lens])++ if self.align_method == "linear_projection": # TODO(Dan): to add other align methods++ # first step : projections+ self.feats_proj = []+ for i, frontend in enumerate(self.frontends):+ input_feats = self.feats[i][0]+ self.feats_proj.append(self.projection_layers[i](input_feats))++ # 2nd step : reshape+ self.feats_reshaped = []+ for i, frontend in enumerate(self.frontends):+ input_feats_proj = self.feats_proj[i]+ bs, nf, dim = input_feats_proj.shape+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/frontends/fused.py#L110" target="_blank" class="src-more">View full source on GitHub →</a></details></div>+<div class="api-detail" id="e283"><span class="dtag badge-f">function</span><h2><code>base_s3prl_setup(args)</code></h2><div class="dmod">funasr.frontends.s3prl · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/frontends/s3prl.py#L17" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Base s3prl setup.</p>+<h4>Args:</h4>+<ul>+<li><code>args</code> — TODO.</li>+</ul>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code>def base_s3prl_setup(args):+ """Base s3prl setup.++ Args:+ args: TODO.+ """+ args.upstream_feature_selection = getattr(args, "upstream_feature_selection", None)+ args.upstream_model_config = getattr(args, "upstream_model_config", None)+ args.upstream_refresh = getattr(args, "upstream_refresh", False)+ args.upstream_ckpt = getattr(args, "upstream_ckpt", None)+ args.init_ckpt = getattr(args, "init_ckpt", None)+ args.verbose = getattr(args, "verbose", False)+ args.tile_factor = getattr(args, "tile_factor", 1)+ return args+</code></pre></details></div>+<div class="api-detail" id="e284"><span class="dtag badge-c">class</span><h2>S3prlFrontend</h2><div class="dmod">funasr.frontends.s3prl · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/frontends/s3prl.py#L33" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Speech Pretrained Representation frontend structure for ASR.</p>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code>class S3prlFrontend(nn.Module):+ """Speech Pretrained Representation frontend structure for ASR."""++ def __init__(+ self,+ fs: Union[int, str] = 16000,+ frontend_conf: Optional[dict] = None,+ download_dir: str = None,+ multilayer_feature: bool = False,+ ):+ """Initialize S3prlFrontend.++ Args:+ fs: TODO.+ frontend_conf: Configuration dict for frontend.+ download_dir: TODO.+ multilayer_feature: TODO.+ """+ super().__init__()+ if isinstance(fs, str):+ fs = humanfriendly.parse_size(fs)++ if download_dir is not None:+ torch.hub.set_dir(download_dir)++ self.multilayer_feature = multilayer_feature+ self.upstream, self.featurizer = self._get_upstream(frontend_conf)+ self.pretrained_params = copy.deepcopy(self.upstream.state_dict())+ self.output_dim = self.featurizer.output_dim+ self.frontend_type = "s3prl"+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/frontends/s3prl.py#L33" target="_blank" class="src-more">View full source on GitHub →</a></details><h3 class="mt">Methods</h3><div class="mblk"><code>.output_size()</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/frontends/s3prl.py#L127" target="_blank" class="gh-link">L127</a><div class="mdoc"><p>Output size.</p>+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def output_size(self) -> int:+ """Output size."""+ return self.output_dim+</code></pre></details></div><div class="mblk"><code>.forward(input, input_lengths)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/frontends/s3prl.py#L131" target="_blank" class="gh-link">L131</a><div class="mdoc"><p>Forward pass for training.</p>+<h4>Args:</h4>+<ul>+<li><code>input</code> — Input audio/text data.</li>+<li><code>input_lengths</code> — Lengths of input.</li>+</ul>+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def forward(+ self, input: torch.Tensor, input_lengths: torch.Tensor+ ) -> Tuple[torch.Tensor, torch.Tensor]:+ """Forward pass for training.++ Args:+ input: Input audio/text data.+ input_lengths: Lengths of input.+ """+ wavs = [wav[: input_lengths[i]] for i, wav in enumerate(input)]+ self.upstream.eval()+ with torch.no_grad():+ feats = self.upstream(wavs)+ feats = self.featurizer(wavs, feats)++ if self.args.tile_factor != 1:+ feats = self._tile_representations(feats)++ input_feats = pad_list(feats, 0.0)+ feats_lens = torch.tensor([f.shape[0] for f in feats], dtype=torch.long)++ # Saving CUDA Memory+ del feats++ return input_feats, feats_lens+</code></pre></details></div><div class="mblk"><code>.reload_pretrained_parameters()</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/frontends/s3prl.py#L157" target="_blank" class="gh-link">L157</a><div class="mdoc"><p>Reload pretrained parameters.</p>+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def reload_pretrained_parameters(self):+ """Reload pretrained parameters."""+ self.upstream.load_state_dict(self.pretrained_params)+ logging.info("Pretrained S3PRL frontend model parameters reloaded!")+</code></pre></details></div></div>+<div class="api-detail" id="e285"><span class="dtag badge-m">method</span><h2><code>S3prlFrontend.output_size()</code></h2><div class="dmod">funasr.frontends.s3prl.S3prlFrontend · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/frontends/s3prl.py#L127" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Output size.</p>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def output_size(self) -> int:+ """Output size."""+ return self.output_dim+</code></pre></details></div>+<div class="api-detail" id="e286"><span class="dtag badge-m">method</span><h2><code>S3prlFrontend.forward(input, input_lengths)</code></h2><div class="dmod">funasr.frontends.s3prl.S3prlFrontend · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/frontends/s3prl.py#L131" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Forward pass for training.</p>+<h4>Args:</h4>+<ul>+<li><code>input</code> — Input audio/text data.</li>+<li><code>input_lengths</code> — Lengths of input.</li>+</ul>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def forward(+ self, input: torch.Tensor, input_lengths: torch.Tensor+ ) -> Tuple[torch.Tensor, torch.Tensor]:+ """Forward pass for training.++ Args:+ input: Input audio/text data.+ input_lengths: Lengths of input.+ """+ wavs = [wav[: input_lengths[i]] for i, wav in enumerate(input)]+ self.upstream.eval()+ with torch.no_grad():+ feats = self.upstream(wavs)+ feats = self.featurizer(wavs, feats)++ if self.args.tile_factor != 1:+ feats = self._tile_representations(feats)++ input_feats = pad_list(feats, 0.0)+ feats_lens = torch.tensor([f.shape[0] for f in feats], dtype=torch.long)++ # Saving CUDA Memory+ del feats++ return input_feats, feats_lens+</code></pre></details></div>+<div class="api-detail" id="e287"><span class="dtag badge-m">method</span><h2><code>S3prlFrontend.reload_pretrained_parameters()</code></h2><div class="dmod">funasr.frontends.s3prl.S3prlFrontend · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/frontends/s3prl.py#L157" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Reload pretrained parameters.</p>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def reload_pretrained_parameters(self):+ """Reload pretrained parameters."""+ self.upstream.load_state_dict(self.pretrained_params)+ logging.info("Pretrained S3PRL frontend model parameters reloaded!")+</code></pre></details></div>+<div class="api-detail" id="e288"><span class="dtag badge-f">function</span><h2><code>load_cmvn(cmvn_file)</code></h2><div class="dmod">funasr.frontends.wav_frontend · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/frontends/wav_frontend.py#L15" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Load cmvn.</p>+<h4>Args:</h4>+<ul>+<li><code>cmvn_file</code> — TODO.</li>+</ul>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code>def load_cmvn(cmvn_file):+ """Load cmvn.++ Args:+ cmvn_file: TODO.+ """+ with open(cmvn_file, "r", encoding="utf-8") as f:+ lines = f.readlines()+ means_list = []+ vars_list = []+ for i in range(len(lines)):+ line_item = lines[i].split()+ if line_item[0] == "<AddShift>":+ line_item = lines[i + 1].split()+ if line_item[0] == "<LearnRateCoef>":+ add_shift_line = line_item[3 : (len(line_item) - 1)]+ means_list = list(add_shift_line)+ continue+ elif line_item[0] == "<Rescale>":+ line_item = lines[i + 1].split()+ if line_item[0] == "<LearnRateCoef>":+ rescale_line = line_item[3 : (len(line_item) - 1)]+ vars_list = list(rescale_line)+ continue+ means = np.array(means_list).astype(np.float32)+ vars = np.array(vars_list).astype(np.float32)+ cmvn = np.array([means, vars])+ cmvn = torch.as_tensor(cmvn, dtype=torch.float32)+ return cmvn+</code></pre></details></div>+<div class="api-detail" id="e289"><span class="dtag badge-f">function</span><h2><code>apply_cmvn(inputs, cmvn)</code></h2><div class="dmod">funasr.frontends.wav_frontend · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/frontends/wav_frontend.py#L46" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Apply CMVN with mvn data</p>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code>def apply_cmvn(inputs, cmvn): # noqa+ """+ Apply CMVN with mvn data+ """++ device = inputs.device+ dtype = inputs.dtype+ frame, dim = inputs.shape++ means = cmvn[0:1, :dim]+ vars = cmvn[1:2, :dim]+ inputs += means.to(device)+ inputs *= vars.to(device)++ return inputs.type(torch.float32)+</code></pre></details></div>+<div class="api-detail" id="e290"><span class="dtag badge-f">function</span><h2><code>apply_lfr(inputs, lfr_m, lfr_n)</code></h2><div class="dmod">funasr.frontends.wav_frontend · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/frontends/wav_frontend.py#L63" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Apply lfr.</p>+<h4>Args:</h4>+<ul>+<li><code>inputs</code> — TODO.</li>+<li><code>lfr_m</code> — TODO.</li>+<li><code>lfr_n</code> — TODO.</li>+</ul>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code>def apply_lfr(inputs, lfr_m, lfr_n):+ """Apply lfr.++ Args:+ inputs: TODO.+ lfr_m: TODO.+ lfr_n: TODO.+ """+ LFR_inputs = []+ T = inputs.shape[0]+ T_lfr = int(np.ceil(T / lfr_n))+ left_padding = inputs[0].repeat((lfr_m - 1) // 2, 1)+ inputs = torch.vstack((left_padding, inputs))+ T = T + (lfr_m - 1) // 2+ feat_dim = inputs.shape[-1]+ strides = (lfr_n * feat_dim, 1)+ sizes = (T_lfr, lfr_m * feat_dim)+ last_idx = (T - lfr_m) // lfr_n + 1+ num_padding = lfr_m - (T - last_idx * lfr_n)+ if num_padding > 0:+ num_padding = (2 * lfr_m - 2 * T + (T_lfr - 1 + last_idx) * lfr_n) / 2 * (T_lfr - last_idx)+ inputs = torch.vstack([inputs] + [inputs[-1:]] * int(num_padding))+ LFR_outputs = inputs.as_strided(sizes, strides)+ return LFR_outputs.clone().type(torch.float32)+</code></pre></details></div>+<div class="api-detail" id="e291"><span class="dtag badge-c">class</span><h2>WavFrontend</h2><div class="dmod">funasr.frontends.wav_frontend · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/frontends/wav_frontend.py#L91" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Conventional frontend structure for ASR.</p>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code>class WavFrontend(nn.Module):+ """Conventional frontend structure for ASR."""++ def __init__(+ self,+ cmvn_file: str = None,+ fs: int = 16000,+ window: str = "hamming",+ n_mels: int = 80,+ frame_length: int = 25,+ frame_shift: int = 10,+ filter_length_min: int = -1,+ filter_length_max: int = -1,+ lfr_m: int = 1,+ lfr_n: int = 1,+ dither: float = 1.0,+ snip_edges: bool = True,+ upsacle_samples: bool = True,+ **kwargs,+ ):+ """Initialize WavFrontend.++ Args:+ cmvn_file: TODO.+ fs: TODO.+ window: TODO.+ n_mels: TODO.+ frame_length: TODO.+ frame_shift: TODO.+ filter_length_min: TODO.+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/frontends/wav_frontend.py#L91" target="_blank" class="src-more">View full source on GitHub →</a></details><h3 class="mt">Methods</h3><div class="mblk"><code>.output_size()</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/frontends/wav_frontend.py#L145" target="_blank" class="gh-link">L145</a><div class="mdoc"><p>Output size.</p>+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def output_size(self) -> int:+ """Output size."""+ return self.n_mels * self.lfr_m+</code></pre></details></div><div class="mblk"><code>.forward(input, input_lengths, **kwargs)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/frontends/wav_frontend.py#L149" target="_blank" class="gh-link">L149</a><div class="mdoc"><p>Forward pass for training.</p>+<h4>Args:</h4>+<ul>+<li><code>input</code> — Input audio/text data.</li>+<li><code>input_lengths</code> — Lengths of input.</li>+<li><code>**kwargs</code> — Additional keyword arguments.</li>+</ul>+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def forward(+ self,+ input: torch.Tensor,+ input_lengths,+ **kwargs,+ ) -> Tuple[torch.Tensor, torch.Tensor]:+ """Forward pass for training.++ Args:+ input: Input audio/text data.+ input_lengths: Lengths of input.+ **kwargs: Additional keyword arguments.+ """+ batch_size = input.size(0)+ feats = []+ feats_lens = []+ for i in range(batch_size):+ waveform_length = input_lengths[i]+ waveform = input[i][:waveform_length]+ if self.upsacle_samples:+ waveform = waveform * (1 << 15)+ waveform = waveform.unsqueeze(0)+ mat = kaldi.fbank(+ waveform,+ num_mel_bins=self.n_mels,+ frame_length=min(self.frame_length,waveform_length/self.fs*1000),+ frame_shift=self.frame_shift,+ dither=self.dither,+ energy_floor=0.0,+ window_type=self.window,+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/frontends/wav_frontend.py#L149" target="_blank" class="src-more">View full source →</a></details></div><div class="mblk"><code>.forward_fbank(input, input_lengths)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/frontends/wav_frontend.py#L198" target="_blank" class="gh-link">L198</a><div class="mdoc"><p>Forward fbank.</p>+<h4>Args:</h4>+<ul>+<li><code>input</code> — Input audio/text data.</li>+<li><code>input_lengths</code> — Lengths of input.</li>+</ul>+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def forward_fbank(+ self, input: torch.Tensor, input_lengths: torch.Tensor+ ) -> Tuple[torch.Tensor, torch.Tensor]:+ """Forward fbank.++ Args:+ input: Input audio/text data.+ input_lengths: Lengths of input.+ """+ batch_size = input.size(0)+ feats = []+ feats_lens = []+ for i in range(batch_size):+ waveform_length = input_lengths[i]+ waveform = input[i][:waveform_length]+ waveform = waveform * (1 << 15)+ waveform = waveform.unsqueeze(0)+ mat = kaldi.fbank(+ waveform,+ num_mel_bins=self.n_mels,+ frame_length=self.frame_length,+ frame_shift=self.frame_shift,+ dither=self.dither,+ energy_floor=0.0,+ window_type=self.window,+ sample_frequency=self.fs,+ )++ feat_length = mat.size(0)+ feats.append(mat)+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/frontends/wav_frontend.py#L198" target="_blank" class="src-more">View full source →</a></details></div><div class="mblk"><code>.forward_lfr_cmvn(input, input_lengths)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/frontends/wav_frontend.py#L234" target="_blank" class="gh-link">L234</a><div class="mdoc"><p>Forward lfr cmvn.</p>+<h4>Args:</h4>+<ul>+<li><code>input</code> — Input audio/text data.</li>+<li><code>input_lengths</code> — Lengths of input.</li>+</ul>+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def forward_lfr_cmvn(+ self, input: torch.Tensor, input_lengths: torch.Tensor+ ) -> Tuple[torch.Tensor, torch.Tensor]:+ """Forward lfr cmvn.++ Args:+ input: Input audio/text data.+ input_lengths: Lengths of input.+ """+ batch_size = input.size(0)+ feats = []+ feats_lens = []+ for i in range(batch_size):+ mat = input[i, : input_lengths[i], :]+ if self.lfr_m != 1 or self.lfr_n != 1:+ mat = apply_lfr(mat, self.lfr_m, self.lfr_n)+ if self.cmvn is not None:+ mat = apply_cmvn(mat, self.cmvn)+ feat_length = mat.size(0)+ feats.append(mat)+ feats_lens.append(feat_length)++ feats_lens = torch.as_tensor(feats_lens)+ feats_pad = pad_sequence(feats, batch_first=True, padding_value=0.0)+ return feats_pad, feats_lens+</code></pre></details></div></div>+<div class="api-detail" id="e292"><span class="dtag badge-m">method</span><h2><code>WavFrontend.output_size()</code></h2><div class="dmod">funasr.frontends.wav_frontend.WavFrontend · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/frontends/wav_frontend.py#L145" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Output size.</p>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def output_size(self) -> int:+ """Output size."""+ return self.n_mels * self.lfr_m+</code></pre></details></div>+<div class="api-detail" id="e293"><span class="dtag badge-m">method</span><h2><code>WavFrontend.forward(input, input_lengths, **kwargs)</code></h2><div class="dmod">funasr.frontends.wav_frontend.WavFrontend · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/frontends/wav_frontend.py#L149" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Forward pass for training.</p>+<h4>Args:</h4>+<ul>+<li><code>input</code> — Input audio/text data.</li>+<li><code>input_lengths</code> — Lengths of input.</li>+<li><code>**kwargs</code> — Additional keyword arguments.</li>+</ul>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def forward(+ self,+ input: torch.Tensor,+ input_lengths,+ **kwargs,+ ) -> Tuple[torch.Tensor, torch.Tensor]:+ """Forward pass for training.++ Args:+ input: Input audio/text data.+ input_lengths: Lengths of input.+ **kwargs: Additional keyword arguments.+ """+ batch_size = input.size(0)+ feats = []+ feats_lens = []+ for i in range(batch_size):+ waveform_length = input_lengths[i]+ waveform = input[i][:waveform_length]+ if self.upsacle_samples:+ waveform = waveform * (1 << 15)+ waveform = waveform.unsqueeze(0)+ mat = kaldi.fbank(+ waveform,+ num_mel_bins=self.n_mels,+ frame_length=min(self.frame_length,waveform_length/self.fs*1000),+ frame_shift=self.frame_shift,+ dither=self.dither,+ energy_floor=0.0,+ window_type=self.window,+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/frontends/wav_frontend.py#L149" target="_blank" class="src-more">View full source on GitHub →</a></details></div>+<div class="api-detail" id="e294"><span class="dtag badge-m">method</span><h2><code>WavFrontend.forward_fbank(input, input_lengths)</code></h2><div class="dmod">funasr.frontends.wav_frontend.WavFrontend · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/frontends/wav_frontend.py#L198" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Forward fbank.</p>+<h4>Args:</h4>+<ul>+<li><code>input</code> — Input audio/text data.</li>+<li><code>input_lengths</code> — Lengths of input.</li>+</ul>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def forward_fbank(+ self, input: torch.Tensor, input_lengths: torch.Tensor+ ) -> Tuple[torch.Tensor, torch.Tensor]:+ """Forward fbank.++ Args:+ input: Input audio/text data.+ input_lengths: Lengths of input.+ """+ batch_size = input.size(0)+ feats = []+ feats_lens = []+ for i in range(batch_size):+ waveform_length = input_lengths[i]+ waveform = input[i][:waveform_length]+ waveform = waveform * (1 << 15)+ waveform = waveform.unsqueeze(0)+ mat = kaldi.fbank(+ waveform,+ num_mel_bins=self.n_mels,+ frame_length=self.frame_length,+ frame_shift=self.frame_shift,+ dither=self.dither,+ energy_floor=0.0,+ window_type=self.window,+ sample_frequency=self.fs,+ )++ feat_length = mat.size(0)+ feats.append(mat)+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/frontends/wav_frontend.py#L198" target="_blank" class="src-more">View full source on GitHub →</a></details></div>+<div class="api-detail" id="e295"><span class="dtag badge-m">method</span><h2><code>WavFrontend.forward_lfr_cmvn(input, input_lengths)</code></h2><div class="dmod">funasr.frontends.wav_frontend.WavFrontend · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/frontends/wav_frontend.py#L234" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Forward lfr cmvn.</p>+<h4>Args:</h4>+<ul>+<li><code>input</code> — Input audio/text data.</li>+<li><code>input_lengths</code> — Lengths of input.</li>+</ul>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def forward_lfr_cmvn(+ self, input: torch.Tensor, input_lengths: torch.Tensor+ ) -> Tuple[torch.Tensor, torch.Tensor]:+ """Forward lfr cmvn.++ Args:+ input: Input audio/text data.+ input_lengths: Lengths of input.+ """+ batch_size = input.size(0)+ feats = []+ feats_lens = []+ for i in range(batch_size):+ mat = input[i, : input_lengths[i], :]+ if self.lfr_m != 1 or self.lfr_n != 1:+ mat = apply_lfr(mat, self.lfr_m, self.lfr_n)+ if self.cmvn is not None:+ mat = apply_cmvn(mat, self.cmvn)+ feat_length = mat.size(0)+ feats.append(mat)+ feats_lens.append(feat_length)++ feats_lens = torch.as_tensor(feats_lens)+ feats_pad = pad_sequence(feats, batch_first=True, padding_value=0.0)+ return feats_pad, feats_lens+</code></pre></details></div>+<div class="api-detail" id="e296"><span class="dtag badge-c">class</span><h2>WavFrontendOnline</h2><div class="dmod">funasr.frontends.wav_frontend · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/frontends/wav_frontend.py#L262" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Conventional frontend structure for streaming ASR/VAD.</p>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code>class WavFrontendOnline(nn.Module):+ """Conventional frontend structure for streaming ASR/VAD."""++ def __init__(+ self,+ cmvn_file: str = None,+ fs: int = 16000,+ window: str = "hamming",+ n_mels: int = 80,+ frame_length: int = 25,+ frame_shift: int = 10,+ filter_length_min: int = -1,+ filter_length_max: int = -1,+ lfr_m: int = 1,+ lfr_n: int = 1,+ dither: float = 1.0,+ snip_edges: bool = True,+ upsacle_samples: bool = True,+ **kwargs,+ ):+ """Initialize WavFrontendOnline.++ Args:+ cmvn_file: TODO.+ fs: TODO.+ window: TODO.+ n_mels: TODO.+ frame_length: TODO.+ frame_shift: TODO.+ filter_length_min: TODO.+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/frontends/wav_frontend.py#L262" target="_blank" class="src-more">View full source on GitHub →</a></details><h3 class="mt">Methods</h3><div class="mblk"><code>.output_size()</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/frontends/wav_frontend.py#L324" target="_blank" class="gh-link">L324</a><div class="mdoc"><p>Output size.</p>+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def output_size(self) -> int:+ """Output size."""+ return self.n_mels * self.lfr_m+</code></pre></details></div><div class="mblk"><code>.apply_cmvn(inputs, cmvn)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/frontends/wav_frontend.py#L329" target="_blank" class="gh-link">L329</a><div class="mdoc"><p>Apply CMVN with mvn data</p>+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def apply_cmvn(inputs: torch.Tensor, cmvn: torch.Tensor) -> torch.Tensor:+ """+ Apply CMVN with mvn data+ """++ device = inputs.device+ dtype = inputs.dtype+ frame, dim = inputs.shape++ means = np.tile(cmvn[0:1, :dim], (frame, 1))+ vars = np.tile(cmvn[1:2, :dim], (frame, 1))+ inputs += torch.from_numpy(means).type(dtype).to(device)+ inputs *= torch.from_numpy(vars).type(dtype).to(device)++ return inputs.type(torch.float32)+</code></pre></details></div><div class="mblk"><code>.apply_lfr(inputs, lfr_m, lfr_n, is_final)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/frontends/wav_frontend.py#L346" target="_blank" class="gh-link">L346</a><div class="mdoc"><p>Apply lfr with data</p>+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def apply_lfr(+ inputs: torch.Tensor, lfr_m: int, lfr_n: int, is_final: bool = False+ ) -> Tuple[torch.Tensor, torch.Tensor, int]:+ """+ Apply lfr with data+ """++ LFR_inputs = []+ # inputs = torch.vstack((inputs_lfr_cache, inputs))+ T = inputs.shape[0] # include the right context+ T_lfr = int(+ np.ceil((T - (lfr_m - 1) // 2) / lfr_n)+ ) # minus the right context: (lfr_m - 1) // 2+ splice_idx = T_lfr+ feat_dim = inputs.shape[-1]+ ori_inputs = inputs+ strides = (lfr_n * feat_dim, 1)+ sizes = (T_lfr, lfr_m * feat_dim)+ last_idx = (T - lfr_m) // lfr_n + 1+ num_padding = lfr_m - (T - last_idx * lfr_n)+ if is_final:+ if num_padding > 0:+ num_padding = (2 * lfr_m - 2 * T + (T_lfr - 1 + last_idx) * lfr_n) / 2 * (T_lfr - last_idx)+ inputs = torch.vstack([inputs] + [inputs[-1:]] * int(num_padding))+ else:+ if num_padding > 0:+ sizes = (last_idx, lfr_m * feat_dim)+ splice_idx = last_idx+ splice_idx = min(T - 1, splice_idx * lfr_n)+ LFR_outputs = inputs[:splice_idx].as_strided(sizes, strides)+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/frontends/wav_frontend.py#L346" target="_blank" class="src-more">View full source →</a></details></div><div class="mblk"><code>.compute_frame_num(sample_length, frame_sample_length, frame_shift_sample_length)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/frontends/wav_frontend.py#L380" target="_blank" class="gh-link">L380</a><div class="mdoc"><p>Compute frame num.</p>+<h4>Args:</h4>+<ul>+<li><code>sample_length</code> — TODO.</li>+<li><code>frame_sample_length</code> — TODO.</li>+<li><code>frame_shift_sample_length</code> — TODO.</li>+</ul>+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def compute_frame_num(+ sample_length: int, frame_sample_length: int, frame_shift_sample_length: int+ ) -> int:+ """Compute frame num.++ Args:+ sample_length: TODO.+ frame_sample_length: TODO.+ frame_shift_sample_length: TODO.+ """+ frame_num = int((sample_length - frame_sample_length) / frame_shift_sample_length + 1)+ return frame_num if frame_num >= 1 and sample_length >= frame_sample_length else 0+</code></pre></details></div><div class="mblk"><code>.forward_fbank(input, input_lengths, cache, **kwargs)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/frontends/wav_frontend.py#L393" target="_blank" class="gh-link">L393</a><div class="mdoc"><p>Forward fbank.</p>+<h4>Args:</h4>+<ul>+<li><code>input</code> — Input audio/text data.</li>+<li><code>input_lengths</code> — Lengths of input.</li>+<li><code>cache</code> — State cache dict for streaming inference.</li>+<li><code>**kwargs</code> — Additional keyword arguments.</li>+</ul>+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def forward_fbank(+ self,+ input: torch.Tensor,+ input_lengths: torch.Tensor,+ cache: dict = None,+ **kwargs,+ ) -> Tuple[torch.Tensor, torch.Tensor, torch.Tensor]:+ """Forward fbank.++ Args:+ input: Input audio/text data.+ input_lengths: Lengths of input.+ cache: State cache dict for streaming inference.+ **kwargs: Additional keyword arguments.+ """+ if cache is None:+ cache = {}+ batch_size = input.size(0)++ input = torch.cat((cache["input_cache"], input), dim=1)+ frame_num = self.compute_frame_num(+ input.shape[-1], self.frame_sample_length, self.frame_shift_sample_length+ )+ # update self.in_cache+ cache["input_cache"] = input[+ :, -(input.shape[-1] - frame_num * self.frame_shift_sample_length) :+ ]+ waveforms = torch.empty(0)+ feats_pad = torch.empty(0)+ feats_lens = torch.empty(0)+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/frontends/wav_frontend.py#L393" target="_blank" class="src-more">View full source →</a></details></div><div class="mblk"><code>.forward_lfr_cmvn(input, input_lengths, is_final, cache, **kwargs)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/frontends/wav_frontend.py#L462" target="_blank" class="gh-link">L462</a><div class="mdoc"><p>Forward lfr cmvn.</p>+<h4>Args:</h4>+<ul>+<li><code>input</code> — Input audio/text data.</li>+<li><code>input_lengths</code> — Lengths of input.</li>+<li><code>is_final</code> — Whether this is the final chunk in streaming.</li>+<li><code>cache</code> — State cache dict for streaming inference.</li>+<li><code>**kwargs</code> — Additional keyword arguments.</li>+</ul>+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def forward_lfr_cmvn(+ self,+ input: torch.Tensor,+ input_lengths: torch.Tensor,+ is_final: bool = False,+ cache: dict = None,+ **kwargs,+ ):+ """Forward lfr cmvn.++ Args:+ input: Input audio/text data.+ input_lengths: Lengths of input.+ is_final: Whether this is the final chunk in streaming.+ cache: State cache dict for streaming inference.+ **kwargs: Additional keyword arguments.+ """+ if cache is None:+ cache = {}+ batch_size = input.size(0)+ feats = []+ feats_lens = []+ lfr_splice_frame_idxs = []+ for i in range(batch_size):+ mat = input[i, : input_lengths[i], :]+ if self.lfr_m != 1 or self.lfr_n != 1:+ # update self.lfr_splice_cache in self.apply_lfr+ # mat, self.lfr_splice_cache[i], lfr_splice_frame_idx = self.apply_lfr(mat, self.lfr_m, self.lfr_n, self.lfr_splice_cache[i],+ mat, cache["lfr_splice_cache"][i], lfr_splice_frame_idx = self.apply_lfr(+ mat, self.lfr_m, self.lfr_n, is_final+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/frontends/wav_frontend.py#L462" target="_blank" class="src-more">View full source →</a></details></div><div class="mblk"><code>.forward(input, input_lengths, **kwargs)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/frontends/wav_frontend.py#L505" target="_blank" class="gh-link">L505</a><div class="mdoc"><p>Forward pass for training.</p>+<h4>Args:</h4>+<ul>+<li><code>input</code> — Input audio/text data.</li>+<li><code>input_lengths</code> — Lengths of input.</li>+<li><code>**kwargs</code> — Additional keyword arguments.</li>+</ul>+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def forward(self, input: torch.Tensor, input_lengths: torch.Tensor, **kwargs):+ """Forward pass for training.++ Args:+ input: Input audio/text data.+ input_lengths: Lengths of input.+ **kwargs: Additional keyword arguments.+ """+ is_final = kwargs.get("is_final", False)+ cache = kwargs.get("cache", {})+ if len(cache) == 0:+ self.init_cache(cache)++ batch_size = input.shape[0]+ assert (+ batch_size == 1+ ), "we support to extract feature online only when the batch size is equal to 1 now"++ waveforms, feats, feats_lengths = self.forward_fbank(+ input, input_lengths, cache=cache+ ) # input shape: B T D++ if feats.shape[0]:++ cache["waveforms"] = torch.cat((cache["reserve_waveforms"], waveforms), dim=1)++ if not cache["lfr_splice_cache"]: # 初始化splice_cache+ for i in range(batch_size):+ cache["lfr_splice_cache"].append(+ feats[i][0, :].unsqueeze(dim=0).repeat((self.lfr_m - 1) // 2, 1)+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/frontends/wav_frontend.py#L505" target="_blank" class="src-more">View full source →</a></details></div><div class="mblk"><code>.init_cache(cache)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/frontends/wav_frontend.py#L594" target="_blank" class="gh-link">L594</a><div class="mdoc"><p>Init cache.</p>+<h4>Args:</h4>+<ul>+<li><code>cache</code> — State cache dict for streaming inference.</li>+</ul>+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def init_cache(self, cache: dict = None):+ """Init cache.++ Args:+ cache: State cache dict for streaming inference.+ """+ if cache is None:+ cache = {}+ cache["reserve_waveforms"] = torch.empty(0)+ cache["input_cache"] = torch.empty(0)+ cache["lfr_splice_cache"] = []+ cache["waveforms"] = None+ cache["fbanks"] = None+ cache["fbanks_lens"] = None+ return cache+</code></pre></details></div></div>+<div class="api-detail" id="e297"><span class="dtag badge-m">method</span><h2><code>WavFrontendOnline.output_size()</code></h2><div class="dmod">funasr.frontends.wav_frontend.WavFrontendOnline · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/frontends/wav_frontend.py#L324" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Output size.</p>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def output_size(self) -> int:+ """Output size."""+ return self.n_mels * self.lfr_m+</code></pre></details></div>+<div class="api-detail" id="e298"><span class="dtag badge-m">method</span><h2><code>WavFrontendOnline.apply_cmvn(inputs, cmvn)</code></h2><div class="dmod">funasr.frontends.wav_frontend.WavFrontendOnline · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/frontends/wav_frontend.py#L329" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Apply CMVN with mvn data</p>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def apply_cmvn(inputs: torch.Tensor, cmvn: torch.Tensor) -> torch.Tensor:+ """+ Apply CMVN with mvn data+ """++ device = inputs.device+ dtype = inputs.dtype+ frame, dim = inputs.shape++ means = np.tile(cmvn[0:1, :dim], (frame, 1))+ vars = np.tile(cmvn[1:2, :dim], (frame, 1))+ inputs += torch.from_numpy(means).type(dtype).to(device)+ inputs *= torch.from_numpy(vars).type(dtype).to(device)++ return inputs.type(torch.float32)+</code></pre></details></div>+<div class="api-detail" id="e299"><span class="dtag badge-m">method</span><h2><code>WavFrontendOnline.apply_lfr(inputs, lfr_m, lfr_n, is_final)</code></h2><div class="dmod">funasr.frontends.wav_frontend.WavFrontendOnline · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/frontends/wav_frontend.py#L346" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Apply lfr with data</p>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def apply_lfr(+ inputs: torch.Tensor, lfr_m: int, lfr_n: int, is_final: bool = False+ ) -> Tuple[torch.Tensor, torch.Tensor, int]:+ """+ Apply lfr with data+ """++ LFR_inputs = []+ # inputs = torch.vstack((inputs_lfr_cache, inputs))+ T = inputs.shape[0] # include the right context+ T_lfr = int(+ np.ceil((T - (lfr_m - 1) // 2) / lfr_n)+ ) # minus the right context: (lfr_m - 1) // 2+ splice_idx = T_lfr+ feat_dim = inputs.shape[-1]+ ori_inputs = inputs+ strides = (lfr_n * feat_dim, 1)+ sizes = (T_lfr, lfr_m * feat_dim)+ last_idx = (T - lfr_m) // lfr_n + 1+ num_padding = lfr_m - (T - last_idx * lfr_n)+ if is_final:+ if num_padding > 0:+ num_padding = (2 * lfr_m - 2 * T + (T_lfr - 1 + last_idx) * lfr_n) / 2 * (T_lfr - last_idx)+ inputs = torch.vstack([inputs] + [inputs[-1:]] * int(num_padding))+ else:+ if num_padding > 0:+ sizes = (last_idx, lfr_m * feat_dim)+ splice_idx = last_idx+ splice_idx = min(T - 1, splice_idx * lfr_n)+ LFR_outputs = inputs[:splice_idx].as_strided(sizes, strides)+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/frontends/wav_frontend.py#L346" target="_blank" class="src-more">View full source on GitHub →</a></details></div>+<div class="api-detail" id="e300"><span class="dtag badge-m">method</span><h2><code>WavFrontendOnline.compute_frame_num(sample_length, frame_sample_length, frame_shift_sample_length)</code></h2><div class="dmod">funasr.frontends.wav_frontend.WavFrontendOnline · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/frontends/wav_frontend.py#L380" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Compute frame num.</p>+<h4>Args:</h4>+<ul>+<li><code>sample_length</code> — TODO.</li>+<li><code>frame_sample_length</code> — TODO.</li>+<li><code>frame_shift_sample_length</code> — TODO.</li>+</ul>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def compute_frame_num(+ sample_length: int, frame_sample_length: int, frame_shift_sample_length: int+ ) -> int:+ """Compute frame num.++ Args:+ sample_length: TODO.+ frame_sample_length: TODO.+ frame_shift_sample_length: TODO.+ """+ frame_num = int((sample_length - frame_sample_length) / frame_shift_sample_length + 1)+ return frame_num if frame_num >= 1 and sample_length >= frame_sample_length else 0+</code></pre></details></div>+<div class="api-detail" id="e301"><span class="dtag badge-m">method</span><h2><code>WavFrontendOnline.forward_fbank(input, input_lengths, cache, **kwargs)</code></h2><div class="dmod">funasr.frontends.wav_frontend.WavFrontendOnline · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/frontends/wav_frontend.py#L393" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Forward fbank.</p>+<h4>Args:</h4>+<ul>+<li><code>input</code> — Input audio/text data.</li>+<li><code>input_lengths</code> — Lengths of input.</li>+<li><code>cache</code> — State cache dict for streaming inference.</li>+<li><code>**kwargs</code> — Additional keyword arguments.</li>+</ul>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def forward_fbank(+ self,+ input: torch.Tensor,+ input_lengths: torch.Tensor,+ cache: dict = None,+ **kwargs,+ ) -> Tuple[torch.Tensor, torch.Tensor, torch.Tensor]:+ """Forward fbank.++ Args:+ input: Input audio/text data.+ input_lengths: Lengths of input.+ cache: State cache dict for streaming inference.+ **kwargs: Additional keyword arguments.+ """+ if cache is None:+ cache = {}+ batch_size = input.size(0)++ input = torch.cat((cache["input_cache"], input), dim=1)+ frame_num = self.compute_frame_num(+ input.shape[-1], self.frame_sample_length, self.frame_shift_sample_length+ )+ # update self.in_cache+ cache["input_cache"] = input[+ :, -(input.shape[-1] - frame_num * self.frame_shift_sample_length) :+ ]+ waveforms = torch.empty(0)+ feats_pad = torch.empty(0)+ feats_lens = torch.empty(0)+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/frontends/wav_frontend.py#L393" target="_blank" class="src-more">View full source on GitHub →</a></details></div>+<div class="api-detail" id="e302"><span class="dtag badge-m">method</span><h2><code>WavFrontendOnline.forward_lfr_cmvn(input, input_lengths, is_final, cache, **kwargs)</code></h2><div class="dmod">funasr.frontends.wav_frontend.WavFrontendOnline · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/frontends/wav_frontend.py#L462" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Forward lfr cmvn.</p>+<h4>Args:</h4>+<ul>+<li><code>input</code> — Input audio/text data.</li>+<li><code>input_lengths</code> — Lengths of input.</li>+<li><code>is_final</code> — Whether this is the final chunk in streaming.</li>+<li><code>cache</code> — State cache dict for streaming inference.</li>+<li><code>**kwargs</code> — Additional keyword arguments.</li>+</ul>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def forward_lfr_cmvn(+ self,+ input: torch.Tensor,+ input_lengths: torch.Tensor,+ is_final: bool = False,+ cache: dict = None,+ **kwargs,+ ):+ """Forward lfr cmvn.++ Args:+ input: Input audio/text data.+ input_lengths: Lengths of input.+ is_final: Whether this is the final chunk in streaming.+ cache: State cache dict for streaming inference.+ **kwargs: Additional keyword arguments.+ """+ if cache is None:+ cache = {}+ batch_size = input.size(0)+ feats = []+ feats_lens = []+ lfr_splice_frame_idxs = []+ for i in range(batch_size):+ mat = input[i, : input_lengths[i], :]+ if self.lfr_m != 1 or self.lfr_n != 1:+ # update self.lfr_splice_cache in self.apply_lfr+ # mat, self.lfr_splice_cache[i], lfr_splice_frame_idx = self.apply_lfr(mat, self.lfr_m, self.lfr_n, self.lfr_splice_cache[i],+ mat, cache["lfr_splice_cache"][i], lfr_splice_frame_idx = self.apply_lfr(+ mat, self.lfr_m, self.lfr_n, is_final+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/frontends/wav_frontend.py#L462" target="_blank" class="src-more">View full source on GitHub →</a></details></div>+<div class="api-detail" id="e303"><span class="dtag badge-m">method</span><h2><code>WavFrontendOnline.forward(input, input_lengths, **kwargs)</code></h2><div class="dmod">funasr.frontends.wav_frontend.WavFrontendOnline · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/frontends/wav_frontend.py#L505" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Forward pass for training.</p>+<h4>Args:</h4>+<ul>+<li><code>input</code> — Input audio/text data.</li>+<li><code>input_lengths</code> — Lengths of input.</li>+<li><code>**kwargs</code> — Additional keyword arguments.</li>+</ul>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def forward(self, input: torch.Tensor, input_lengths: torch.Tensor, **kwargs):+ """Forward pass for training.++ Args:+ input: Input audio/text data.+ input_lengths: Lengths of input.+ **kwargs: Additional keyword arguments.+ """+ is_final = kwargs.get("is_final", False)+ cache = kwargs.get("cache", {})+ if len(cache) == 0:+ self.init_cache(cache)++ batch_size = input.shape[0]+ assert (+ batch_size == 1+ ), "we support to extract feature online only when the batch size is equal to 1 now"++ waveforms, feats, feats_lengths = self.forward_fbank(+ input, input_lengths, cache=cache+ ) # input shape: B T D++ if feats.shape[0]:++ cache["waveforms"] = torch.cat((cache["reserve_waveforms"], waveforms), dim=1)++ if not cache["lfr_splice_cache"]: # 初始化splice_cache+ for i in range(batch_size):+ cache["lfr_splice_cache"].append(+ feats[i][0, :].unsqueeze(dim=0).repeat((self.lfr_m - 1) // 2, 1)+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/frontends/wav_frontend.py#L505" target="_blank" class="src-more">View full source on GitHub →</a></details></div>+<div class="api-detail" id="e304"><span class="dtag badge-m">method</span><h2><code>WavFrontendOnline.init_cache(cache)</code></h2><div class="dmod">funasr.frontends.wav_frontend.WavFrontendOnline · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/frontends/wav_frontend.py#L594" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Init cache.</p>+<h4>Args:</h4>+<ul>+<li><code>cache</code> — State cache dict for streaming inference.</li>+</ul>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def init_cache(self, cache: dict = None):+ """Init cache.++ Args:+ cache: State cache dict for streaming inference.+ """+ if cache is None:+ cache = {}+ cache["reserve_waveforms"] = torch.empty(0)+ cache["input_cache"] = torch.empty(0)+ cache["lfr_splice_cache"] = []+ cache["waveforms"] = None+ cache["fbanks"] = None+ cache["fbanks_lens"] = None+ return cache+</code></pre></details></div>+<div class="api-detail" id="e305"><span class="dtag badge-c">class</span><h2>WavFrontendMel23</h2><div class="dmod">funasr.frontends.wav_frontend · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/frontends/wav_frontend.py#L611" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Conventional frontend structure for ASR.</p>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code>class WavFrontendMel23(nn.Module):+ """Conventional frontend structure for ASR."""++ def __init__(+ self,+ fs: int = 16000,+ frame_length: int = 25,+ frame_shift: int = 10,+ lfr_m: int = 1,+ lfr_n: int = 1,+ **kwargs,+ ):+ """Initialize WavFrontendMel23.++ Args:+ fs: TODO.+ frame_length: TODO.+ frame_shift: TODO.+ lfr_m: TODO.+ lfr_n: TODO.+ **kwargs: Additional keyword arguments.+ """+ super().__init__()+ self.fs = fs+ self.frame_length = frame_length+ self.frame_shift = frame_shift+ self.lfr_m = lfr_m+ self.lfr_n = lfr_n+ self.n_mels = 23++</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/frontends/wav_frontend.py#L611" target="_blank" class="src-more">View full source on GitHub →</a></details><h3 class="mt">Methods</h3><div class="mblk"><code>.output_size()</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/frontends/wav_frontend.py#L641" target="_blank" class="gh-link">L641</a><div class="mdoc"><p>Output size.</p>+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def output_size(self) -> int:+ """Output size."""+ return self.n_mels * (2 * self.lfr_m + 1)+</code></pre></details></div><div class="mblk"><code>.forward(input, input_lengths)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/frontends/wav_frontend.py#L645" target="_blank" class="gh-link">L645</a><div class="mdoc"><p>Forward pass for training.</p>+<h4>Args:</h4>+<ul>+<li><code>input</code> — Input audio/text data.</li>+<li><code>input_lengths</code> — Lengths of input.</li>+</ul>+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def forward(+ self, input: torch.Tensor, input_lengths: torch.Tensor+ ) -> Tuple[torch.Tensor, torch.Tensor]:+ """Forward pass for training.++ Args:+ input: Input audio/text data.+ input_lengths: Lengths of input.+ """+ batch_size = input.size(0)+ feats = []+ feats_lens = []+ for i in range(batch_size):+ waveform_length = input_lengths[i]+ waveform = input[i][:waveform_length]+ waveform = waveform.numpy()+ mat = eend_ola_feature.stft(waveform, self.frame_length, self.frame_shift)+ mat = eend_ola_feature.transform(mat)+ mat = eend_ola_feature.splice(mat, context_size=self.lfr_m)+ mat = mat[:: self.lfr_n]+ mat = torch.from_numpy(mat)+ feat_length = mat.size(0)+ feats.append(mat)+ feats_lens.append(feat_length)++ feats_lens = torch.as_tensor(feats_lens)+ feats_pad = pad_sequence(feats, batch_first=True, padding_value=0.0)+ return feats_pad, feats_lens+</code></pre></details></div></div>+<div class="api-detail" id="e306"><span class="dtag badge-m">method</span><h2><code>WavFrontendMel23.output_size()</code></h2><div class="dmod">funasr.frontends.wav_frontend.WavFrontendMel23 · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/frontends/wav_frontend.py#L641" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Output size.</p>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def output_size(self) -> int:+ """Output size."""+ return self.n_mels * (2 * self.lfr_m + 1)+</code></pre></details></div>+<div class="api-detail" id="e307"><span class="dtag badge-m">method</span><h2><code>WavFrontendMel23.forward(input, input_lengths)</code></h2><div class="dmod">funasr.frontends.wav_frontend.WavFrontendMel23 · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/frontends/wav_frontend.py#L645" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Forward pass for training.</p>+<h4>Args:</h4>+<ul>+<li><code>input</code> — Input audio/text data.</li>+<li><code>input_lengths</code> — Lengths of input.</li>+</ul>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def forward(+ self, input: torch.Tensor, input_lengths: torch.Tensor+ ) -> Tuple[torch.Tensor, torch.Tensor]:+ """Forward pass for training.++ Args:+ input: Input audio/text data.+ input_lengths: Lengths of input.+ """+ batch_size = input.size(0)+ feats = []+ feats_lens = []+ for i in range(batch_size):+ waveform_length = input_lengths[i]+ waveform = input[i][:waveform_length]+ waveform = waveform.numpy()+ mat = eend_ola_feature.stft(waveform, self.frame_length, self.frame_shift)+ mat = eend_ola_feature.transform(mat)+ mat = eend_ola_feature.splice(mat, context_size=self.lfr_m)+ mat = mat[:: self.lfr_n]+ mat = torch.from_numpy(mat)+ feat_length = mat.size(0)+ feats.append(mat)+ feats_lens.append(feat_length)++ feats_lens = torch.as_tensor(feats_lens)+ feats_pad = pad_sequence(feats, batch_first=True, padding_value=0.0)+ return feats_pad, feats_lens+</code></pre></details></div>+<div class="api-detail" id="e308"><span class="dtag badge-c">class</span><h2>WhisperFrontend</h2><div class="dmod">funasr.frontends.whisper_frontend · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/frontends/whisper_frontend.py#L11" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Speech Representation Using Encoder Outputs from OpenAI's Whisper Model:</p>+<ul>+<li><code>URL</code> — https://github.com/openai/whisper</li>+</ul>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code>class WhisperFrontend(nn.Module):+ """Speech Representation Using Encoder Outputs from OpenAI's Whisper Model:++ URL: https://github.com/openai/whisper+ """++ def __init__(+ self,+ fs: int = 16000,+ whisper_model: str = None,+ do_pad_trim: bool = True,+ n_mels: int = 80,+ permute: bool = False,+ **kwargs,+ ):+ """Initialize WhisperFrontend.++ Args:+ fs: TODO.+ whisper_model: Whisper Model instance.+ do_pad_trim: TODO.+ n_mels: TODO.+ permute: TODO.+ **kwargs: Additional keyword arguments.+ """+ super().__init__()+ assert fs == 16000+ self.fs = fs+ import whisper+ from whisper.audio import HOP_LENGTH, N_FFT, N_SAMPLES+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/frontends/whisper_frontend.py#L11" target="_blank" class="src-more">View full source on GitHub →</a></details><h3 class="mt">Methods</h3><div class="mblk"><code>.output_size()</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/frontends/whisper_frontend.py#L67" target="_blank" class="gh-link">L67</a><div class="mdoc"><p>Output size.</p>+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def output_size(self) -> int:+ """Output size."""+ return self.n_mels+</code></pre></details></div><div class="mblk"><code>.log_mel_spectrogram(audio, ilens)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/frontends/whisper_frontend.py#L71" target="_blank" class="gh-link">L71</a><div class="mdoc"><p>Log mel spectrogram.</p>+<h4>Args:</h4>+<ul>+<li><code>audio</code> — TODO.</li>+<li><code>ilens</code> — TODO.</li>+</ul>+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def log_mel_spectrogram(+ self,+ audio: torch.Tensor,+ ilens: torch.Tensor = None,+ ) -> torch.Tensor:+ """Log mel spectrogram.++ Args:+ audio: TODO.+ ilens: TODO.+ """+ window = torch.hann_window(self.win_length).to(audio.device)+ stft = torch.stft(audio, self.n_fft, self.hop_length, window=window, return_complex=True)++ # whisper deletes the last frame by default (Shih-Lun)+ magnitudes = stft[..., :-1].abs() ** 2+ if self.filters_path is not None:+ filters = self.mel_filters(audio.device, self.n_mels, self.filters_path)+ else:+ filters = self.mel_filters(audio.device, self.n_mels)+ mel_spec = filters @ magnitudes++ log_spec = torch.clamp(mel_spec, min=1e-10).log10()++ if ilens is not None:+ olens = ilens // self.hop_length+ else:+ olens = None++ log_spec = torch.maximum(+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/frontends/whisper_frontend.py#L71" target="_blank" class="src-more">View full source →</a></details></div><div class="mblk"><code>.forward(input, input_lengths, **kwargs)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/frontends/whisper_frontend.py#L108" target="_blank" class="gh-link">L108</a><div class="mdoc"><p>Forward pass for training.</p>+<h4>Args:</h4>+<ul>+<li><code>input</code> — Input audio/text data.</li>+<li><code>input_lengths</code> — Lengths of input.</li>+<li><code>**kwargs</code> — Additional keyword arguments.</li>+</ul>+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def forward(+ self,+ input: torch.Tensor,+ input_lengths: torch.Tensor,+ **kwargs,+ ) -> Tuple[torch.Tensor, torch.Tensor]:+ """Forward pass for training.++ Args:+ input: Input audio/text data.+ input_lengths: Lengths of input.+ **kwargs: Additional keyword arguments.+ """+ batch_size = input.size(0)+ feats = []+ feats_lens = []+ input = input.to(torch.float32)+ for i in range(batch_size):+ if self.do_pad_trim:+ feat = self.pad_or_trim(input[i], self.pad_samples)+ else:+ feat = input[i]+ feat, feat_len = self.log_mel_spectrogram(feat[None, :], input_lengths[0])+ feats.append(feat[0])+ feats_lens.append(feat_len)+ feats_lens = torch.as_tensor(feats_lens)++ if batch_size == 1:+ feats_pad = feats[0][None, :, :]+ else:+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/frontends/whisper_frontend.py#L108" target="_blank" class="src-more">View full source →</a></details></div></div>+<div class="api-detail" id="e309"><span class="dtag badge-m">method</span><h2><code>WhisperFrontend.output_size()</code></h2><div class="dmod">funasr.frontends.whisper_frontend.WhisperFrontend · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/frontends/whisper_frontend.py#L67" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Output size.</p>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def output_size(self) -> int:+ """Output size."""+ return self.n_mels+</code></pre></details></div>+<div class="api-detail" id="e310"><span class="dtag badge-m">method</span><h2><code>WhisperFrontend.log_mel_spectrogram(audio, ilens)</code></h2><div class="dmod">funasr.frontends.whisper_frontend.WhisperFrontend · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/frontends/whisper_frontend.py#L71" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Log mel spectrogram.</p>+<h4>Args:</h4>+<ul>+<li><code>audio</code> — TODO.</li>+<li><code>ilens</code> — TODO.</li>+</ul>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def log_mel_spectrogram(+ self,+ audio: torch.Tensor,+ ilens: torch.Tensor = None,+ ) -> torch.Tensor:+ """Log mel spectrogram.++ Args:+ audio: TODO.+ ilens: TODO.+ """+ window = torch.hann_window(self.win_length).to(audio.device)+ stft = torch.stft(audio, self.n_fft, self.hop_length, window=window, return_complex=True)++ # whisper deletes the last frame by default (Shih-Lun)+ magnitudes = stft[..., :-1].abs() ** 2+ if self.filters_path is not None:+ filters = self.mel_filters(audio.device, self.n_mels, self.filters_path)+ else:+ filters = self.mel_filters(audio.device, self.n_mels)+ mel_spec = filters @ magnitudes++ log_spec = torch.clamp(mel_spec, min=1e-10).log10()++ if ilens is not None:+ olens = ilens // self.hop_length+ else:+ olens = None++ log_spec = torch.maximum(+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/frontends/whisper_frontend.py#L71" target="_blank" class="src-more">View full source on GitHub →</a></details></div>+<div class="api-detail" id="e311"><span class="dtag badge-m">method</span><h2><code>WhisperFrontend.forward(input, input_lengths, **kwargs)</code></h2><div class="dmod">funasr.frontends.whisper_frontend.WhisperFrontend · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/frontends/whisper_frontend.py#L108" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Forward pass for training.</p>+<h4>Args:</h4>+<ul>+<li><code>input</code> — Input audio/text data.</li>+<li><code>input_lengths</code> — Lengths of input.</li>+<li><code>**kwargs</code> — Additional keyword arguments.</li>+</ul>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def forward(+ self,+ input: torch.Tensor,+ input_lengths: torch.Tensor,+ **kwargs,+ ) -> Tuple[torch.Tensor, torch.Tensor]:+ """Forward pass for training.++ Args:+ input: Input audio/text data.+ input_lengths: Lengths of input.+ **kwargs: Additional keyword arguments.+ """+ batch_size = input.size(0)+ feats = []+ feats_lens = []+ input = input.to(torch.float32)+ for i in range(batch_size):+ if self.do_pad_trim:+ feat = self.pad_or_trim(input[i], self.pad_samples)+ else:+ feat = input[i]+ feat, feat_len = self.log_mel_spectrogram(feat[None, :], input_lengths[0])+ feats.append(feat[0])+ feats_lens.append(feat_len)+ feats_lens = torch.as_tensor(feats_lens)++ if batch_size == 1:+ feats_pad = feats[0][None, :, :]+ else:+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/frontends/whisper_frontend.py#L108" target="_blank" class="src-more">View full source on GitHub →</a></details></div>+<div class="api-detail" id="e312"><span class="dtag badge-c">class</span><h2>SlidingWindow</h2><div class="dmod">funasr.frontends.windowing · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/frontends/windowing.py#L12" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Sliding Window.</p>+<p>Provides a sliding window over a batched continuous raw audio tensor.</p>+<p>Optionally, provides padding (Currently not implemented).</p>+<p>Combine this module with a pre-encoder compatible with raw audio data,</p>+<p>for example Sinc convolutions.</p>+<p>Known issues:</p>+<p>Output length is calculated incorrectly if audio shorter than win_length.</p>+<ul>+<li><code>WARNING</code> — trailing values are discarded - padding not implemented yet.</li>+</ul>+<p>There is currently no additional window function applied to input values.</p>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code>class SlidingWindow(nn.Module):+ """Sliding Window.+ Provides a sliding window over a batched continuous raw audio tensor.+ Optionally, provides padding (Currently not implemented).+ Combine this module with a pre-encoder compatible with raw audio data,+ for example Sinc convolutions.+ Known issues:+ Output length is calculated incorrectly if audio shorter than win_length.+ WARNING: trailing values are discarded - padding not implemented yet.+ There is currently no additional window function applied to input values.+ """++ def __init__(+ self,+ win_length: int = 400,+ hop_length: int = 160,+ channels: int = 1,+ padding: int = None,+ fs=None,+ ):+ """Initialize.+ Args:+ win_length: Length of frame.+ hop_length: Relative starting point of next frame.+ channels: Number of input channels.+ padding: Padding (placeholder, currently not implemented).+ fs: Sampling rate (placeholder for compatibility, not used).+ """+ super().__init__()+ self.fs = fs+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/frontends/windowing.py#L12" target="_blank" class="src-more">View full source on GitHub →</a></details><h3 class="mt">Methods</h3><div class="mblk"><code>.forward(input, input_lengths)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/frontends/windowing.py#L47" target="_blank" class="gh-link">L47</a><div class="mdoc"><p>Apply a sliding window on the input.</p>+<h4>Args:</h4>+<ul>+<li><code>input</code> — Input (B, T, C*D) or (B, T*C*D), with D=C=1.</li>+<li><code>input_lengths</code> — Input lengths within batch.</li>+</ul>+<h4>Returns:</h4>+<ul>+<li><code>Tensor</code> — Output with dimensions (B, T, C, D), with D=win_length.</li>+<li><code>Tensor</code> — Output lengths within batch.</li>+</ul>+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def forward(+ self, input: torch.Tensor, input_lengths: torch.Tensor+ ) -> Tuple[torch.Tensor, torch.Tensor]:+ """Apply a sliding window on the input.+ Args:+ input: Input (B, T, C*D) or (B, T*C*D), with D=C=1.+ input_lengths: Input lengths within batch.+ Returns:+ Tensor: Output with dimensions (B, T, C, D), with D=win_length.+ Tensor: Output lengths within batch.+ """+ input_size = input.size()+ B = input_size[0]+ T = input_size[1]+ C = self.channels+ D = self.win_length+ # (B, T, C) --> (T, B, C)+ continuous = input.view(B, T, C).permute(1, 0, 2)+ windowed = continuous.unfold(0, D, self.hop_length)+ # (T, B, C, D) --> (B, T, C, D)+ output = windowed.permute(1, 0, 2, 3).contiguous()+ # After unfold(), windowed lengths change:+ output_lengths = (input_lengths - self.win_length) // self.hop_length + 1+ return output, output_lengths+</code></pre></details></div><div class="mblk"><code>.output_size()</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/frontends/windowing.py#L72" target="_blank" class="gh-link">L72</a><div class="mdoc"><p>Return output length of feature dimension D, i.e. the window length.</p>+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def output_size(self) -> int:+ """Return output length of feature dimension D, i.e. the window length."""+ return self.win_length+</code></pre></details></div></div>+<div class="api-detail" id="e313"><span class="dtag badge-m">method</span><h2><code>SlidingWindow.forward(input, input_lengths)</code></h2><div class="dmod">funasr.frontends.windowing.SlidingWindow · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/frontends/windowing.py#L47" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Apply a sliding window on the input.</p>+<h4>Args:</h4>+<ul>+<li><code>input</code> — Input (B, T, C*D) or (B, T*C*D), with D=C=1.</li>+<li><code>input_lengths</code> — Input lengths within batch.</li>+</ul>+<h4>Returns:</h4>+<ul>+<li><code>Tensor</code> — Output with dimensions (B, T, C, D), with D=win_length.</li>+<li><code>Tensor</code> — Output lengths within batch.</li>+</ul>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def forward(+ self, input: torch.Tensor, input_lengths: torch.Tensor+ ) -> Tuple[torch.Tensor, torch.Tensor]:+ """Apply a sliding window on the input.+ Args:+ input: Input (B, T, C*D) or (B, T*C*D), with D=C=1.+ input_lengths: Input lengths within batch.+ Returns:+ Tensor: Output with dimensions (B, T, C, D), with D=win_length.+ Tensor: Output lengths within batch.+ """+ input_size = input.size()+ B = input_size[0]+ T = input_size[1]+ C = self.channels+ D = self.win_length+ # (B, T, C) --> (T, B, C)+ continuous = input.view(B, T, C).permute(1, 0, 2)+ windowed = continuous.unfold(0, D, self.hop_length)+ # (T, B, C, D) --> (B, T, C, D)+ output = windowed.permute(1, 0, 2, 3).contiguous()+ # After unfold(), windowed lengths change:+ output_lengths = (input_lengths - self.win_length) // self.hop_length + 1+ return output, output_lengths+</code></pre></details></div>+<div class="api-detail" id="e314"><span class="dtag badge-m">method</span><h2><code>SlidingWindow.output_size()</code></h2><div class="dmod">funasr.frontends.windowing.SlidingWindow · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/frontends/windowing.py#L72" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Return output length of feature dimension D, i.e. the window length.</p>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def output_size(self) -> int:+ """Return output length of feature dimension D, i.e. the window length."""+ return self.win_length+</code></pre></details></div>+<div class="api-detail" id="e315"><span class="dtag badge-f">function</span><h2><code>build_tokenizer(token_type, bpemodel, non_linguistic_symbols, remove_non_linguistic_symbols, space_symbol, delimiter, g2p_type)</code></h2><div class="dmod">funasr.tokenizer.build_tokenizer · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/tokenizer/build_tokenizer.py#L13" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>A helper function to instantiate Tokenizer</p>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code>def build_tokenizer(+ token_type: str,+ bpemodel: Union[Path, str, Iterable[str]] = None,+ non_linguistic_symbols: Union[Path, str, Iterable[str]] = None,+ remove_non_linguistic_symbols: bool = False,+ space_symbol: str = "<space>",+ delimiter: str = None,+ g2p_type: str = None,+) -> AbsTokenizer:+ """A helper function to instantiate Tokenizer"""+ if token_type == "bpe":+ if bpemodel is None:+ raise ValueError('bpemodel is required if token_type = "bpe"')++ if remove_non_linguistic_symbols:+ raise RuntimeError(+ "remove_non_linguistic_symbols is not implemented for token_type=bpe"+ )+ return SentencepiecesTokenizer(bpemodel)++ elif token_type == "word":+ if remove_non_linguistic_symbols and non_linguistic_symbols is not None:+ return WordTokenizer(+ delimiter=delimiter,+ non_linguistic_symbols=non_linguistic_symbols,+ remove_non_linguistic_symbols=True,+ )+ else:+ return WordTokenizer(delimiter=delimiter)++</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/tokenizer/build_tokenizer.py#L13" target="_blank" class="src-more">View full source on GitHub →</a></details></div>+<div class="api-detail" id="e316"><span class="dtag badge-c">class</span><h2>CharTokenizer</h2><div class="dmod">funasr.tokenizer.char_tokenizer · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/tokenizer/char_tokenizer.py#L13" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p class="muted">No documentation yet.</p></div><details class="src-block"><summary>📄 Source code</summary><pre><code>class CharTokenizer(BaseTokenizer):+ def __init__(+ self,+ non_linguistic_symbols: Union[Path, str, Iterable[str]] = None,+ space_symbol: str = "<space>",+ remove_non_linguistic_symbols: bool = False,+ split_with_space: bool = False,+ seg_dict: str = None,+ **kwargs,+ ):+ """Initialize CharTokenizer.++ Args:+ non_linguistic_symbols: TODO.+ space_symbol: TODO.+ remove_non_linguistic_symbols: TODO.+ split_with_space: TODO.+ seg_dict: TODO.+ **kwargs: Additional keyword arguments.+ """+ super().__init__(**kwargs)+ self.space_symbol = space_symbol+ if non_linguistic_symbols is None:+ self.non_linguistic_symbols = set()+ elif isinstance(non_linguistic_symbols, (Path, str)):+ non_linguistic_symbols = Path(non_linguistic_symbols)+ try:+ with non_linguistic_symbols.open("r", encoding="utf-8") as f:+ self.non_linguistic_symbols = set(line.rstrip() for line in f)+ except FileNotFoundError:+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/tokenizer/char_tokenizer.py#L13" target="_blank" class="src-more">View full source on GitHub →</a></details><h3 class="mt">Methods</h3><div class="mblk"><code>.text2tokens(line)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/tokenizer/char_tokenizer.py#L63" target="_blank" class="gh-link">L63</a><div class="mdoc"><p>Text2tokens.</p>+<h4>Args:</h4>+<ul>+<li><code>line</code> — TODO.</li>+</ul>+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def text2tokens(self, line: Union[str, list]) -> List[str]:+ # if self.split_with_space:+ """Text2tokens.++ Args:+ line: TODO.+ """+ if self.seg_dict is not None:+ tokens = line.strip().split(" ")+ tokens = seg_tokenize(tokens, self.seg_dict)+ else:+ tokens = []+ while len(line) != 0:+ for w in self.non_linguistic_symbols:+ if line.startswith(w):+ if not self.remove_non_linguistic_symbols:+ tokens.append(line[: len(w)])+ line = line[len(w) :]+ break+ else:+ t = line[0]+ if t == " ":+ # t = "<space>"+ line = line[1:]+ continue+ tokens.append(t)+ line = line[1:]+ return tokens+</code></pre></details></div><div class="mblk"><code>.tokens2text(tokens)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/tokenizer/char_tokenizer.py#L92" target="_blank" class="gh-link">L92</a><div class="mdoc"><p>Tokens2text.</p>+<h4>Args:</h4>+<ul>+<li><code>tokens</code> — TODO.</li>+</ul>+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def tokens2text(self, tokens: Iterable[str]) -> str:+ """Tokens2text.++ Args:+ tokens: TODO.+ """+ tokens = [t if t != self.space_symbol else " " for t in tokens]+ return "".join(tokens)+</code></pre></details></div></div>+<div class="api-detail" id="e317"><span class="dtag badge-m">method</span><h2><code>CharTokenizer.text2tokens(line)</code></h2><div class="dmod">funasr.tokenizer.char_tokenizer.CharTokenizer · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/tokenizer/char_tokenizer.py#L63" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Text2tokens.</p>+<h4>Args:</h4>+<ul>+<li><code>line</code> — TODO.</li>+</ul>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def text2tokens(self, line: Union[str, list]) -> List[str]:+ # if self.split_with_space:+ """Text2tokens.++ Args:+ line: TODO.+ """+ if self.seg_dict is not None:+ tokens = line.strip().split(" ")+ tokens = seg_tokenize(tokens, self.seg_dict)+ else:+ tokens = []+ while len(line) != 0:+ for w in self.non_linguistic_symbols:+ if line.startswith(w):+ if not self.remove_non_linguistic_symbols:+ tokens.append(line[: len(w)])+ line = line[len(w) :]+ break+ else:+ t = line[0]+ if t == " ":+ # t = "<space>"+ line = line[1:]+ continue+ tokens.append(t)+ line = line[1:]+ return tokens+</code></pre></details></div>+<div class="api-detail" id="e318"><span class="dtag badge-m">method</span><h2><code>CharTokenizer.tokens2text(tokens)</code></h2><div class="dmod">funasr.tokenizer.char_tokenizer.CharTokenizer · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/tokenizer/char_tokenizer.py#L92" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Tokens2text.</p>+<h4>Args:</h4>+<ul>+<li><code>tokens</code> — TODO.</li>+</ul>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def tokens2text(self, tokens: Iterable[str]) -> str:+ """Tokens2text.++ Args:+ tokens: TODO.+ """+ tokens = [t if t != self.space_symbol else " " for t in tokens]+ return "".join(tokens)+</code></pre></details></div>+<div class="api-detail" id="e319"><span class="dtag badge-f">function</span><h2><code>load_seg_dict(seg_dict_file)</code></h2><div class="dmod">funasr.tokenizer.char_tokenizer · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/tokenizer/char_tokenizer.py#L102" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Load seg dict.</p>+<h4>Args:</h4>+<ul>+<li><code>seg_dict_file</code> — TODO.</li>+</ul>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code>def load_seg_dict(seg_dict_file):+ """Load seg dict.++ Args:+ seg_dict_file: TODO.+ """+ seg_dict = {}+ assert isinstance(seg_dict_file, str)+ with open(seg_dict_file, "r", encoding="utf8") as f:+ lines = f.readlines()+ for line in lines:+ s = line.strip().split()+ key = s[0]+ value = s[1:]+ seg_dict[key] = " ".join(value)+ return seg_dict+</code></pre></details></div>+<div class="api-detail" id="e320"><span class="dtag badge-f">function</span><h2><code>seg_tokenize(txt, seg_dict)</code></h2><div class="dmod">funasr.tokenizer.char_tokenizer · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/tokenizer/char_tokenizer.py#L120" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Seg tokenize.</p>+<h4>Args:</h4>+<ul>+<li><code>txt</code> — TODO.</li>+<li><code>seg_dict</code> — TODO.</li>+</ul>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code>def seg_tokenize(txt, seg_dict):+ # pattern = re.compile(r'^[\u4E00-\u9FA50-9]+$')+ """Seg tokenize.++ Args:+ txt: TODO.+ seg_dict: TODO.+ """+ pattern = re.compile(r"([\u4E00-\u9FA5A-Za-z0-9])")+ out_txt = ""+ for word in txt:+ word = word.lower()+ if word in seg_dict:+ out_txt += seg_dict[word] + " "+ else:+ if pattern.match(word):+ for char in word:+ if char in seg_dict:+ out_txt += seg_dict[char] + " "+ else:+ out_txt += "<unk>" + " "+ else:+ out_txt += "<unk>" + " "+ return out_txt.strip().split()+</code></pre></details></div>+<div class="api-detail" id="e321"><span class="dtag badge-c">class</span><h2>TextCleaner</h2><div class="dmod">funasr.tokenizer.cleaner · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/tokenizer/cleaner.py#L13" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Text cleaner.</p>+<h4>Examples:</h4>+<p>>>> cleaner = TextCleaner("tacotron")</p>+<p>>>> cleaner("(Hello-World); & jr. & dr.")</p>+<p>'HELLO WORLD, AND JUNIOR AND DOCTOR'</p>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code>class TextCleaner:+ """Text cleaner.++ Examples:+ >>> cleaner = TextCleaner("tacotron")+ >>> cleaner("(Hello-World); & jr. & dr.")+ 'HELLO WORLD, AND JUNIOR AND DOCTOR'++ """++ def __init__(self, cleaner_types: Collection[str] = None):++ """Initialize TextCleaner.++ Args:+ cleaner_types: TODO.+ """+ if cleaner_types is None:+ self.cleaner_types = []+ elif isinstance(cleaner_types, str):+ self.cleaner_types = [cleaner_types]+ else:+ self.cleaner_types = list(cleaner_types)++ def __call__(self, text: str) -> str:+ """Internal: call .++ Args:+ text: Text tensor or string input.+ """+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/tokenizer/cleaner.py#L13" target="_blank" class="src-more">View full source on GitHub →</a></details></div>+<div class="api-detail" id="e322"><span class="dtag badge-f">function</span><h2><code>HuggingfaceTokenizer(init_param_path, **kwargs)</code></h2><div class="dmod">funasr.tokenizer.hf_tokenizer · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/tokenizer/hf_tokenizer.py#L5" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Huggingfacetokenizer.</p>+<h4>Args:</h4>+<ul>+<li><code>init_param_path</code> — TODO.</li>+<li><code>**kwargs</code> — Additional keyword arguments.</li>+</ul>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code>def HuggingfaceTokenizer(init_param_path, **kwargs):+ """Huggingfacetokenizer.++ Args:+ init_param_path: TODO.+ **kwargs: Additional keyword arguments.+ """+ try:+ from transformers import AutoTokenizer+ except Exception as e:+ raise ImportError(+ "HuggingfaceTokenizer requires 'transformers'. "+ "Please install it with: pip install -U transformers"+ ) from e++ tokenizer = AutoTokenizer.from_pretrained(init_param_path)++ return tokenizer+</code></pre></details></div>+<div class="api-detail" id="e323"><span class="dtag badge-c">class</span><h2>KoreanCleaner</h2><div class="dmod">funasr.tokenizer.korean_cleaner · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/tokenizer/korean_cleaner.py#L6" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p class="muted">No documentation yet.</p></div><details class="src-block"><summary>📄 Source code</summary><pre><code>class KoreanCleaner:+ @classmethod+ def _normalize_numbers(cls, text):+ """Internal: normalize numbers.++ Args:+ text: Text tensor or string input.+ """+ number_to_kor = {+ "0": "영",+ "1": "일",+ "2": "이",+ "3": "삼",+ "4": "사",+ "5": "오",+ "6": "육",+ "7": "칠",+ "8": "팔",+ "9": "구",+ }+ new_text = "".join(+ number_to_kor[char] if char in number_to_kor.keys() else char for char in text+ )+ return new_text++ @classmethod+ def _normalize_english_text(cls, text):+ """Internal: normalize english text.++ Args:+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/tokenizer/korean_cleaner.py#L6" target="_blank" class="src-more">View full source on GitHub →</a></details><h3 class="mt">Methods</h3><div class="mblk"><code>.normalize_text(cls, text)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/tokenizer/korean_cleaner.py#L75" target="_blank" class="gh-link">L75</a><div class="mdoc"><p>Normalize text.</p>+<h4>Args:</h4>+<ul>+<li><code>text</code> — Text tensor or string input.</li>+</ul>+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def normalize_text(cls, text):+ # stage 0 : text strip+ """Normalize text.++ Args:+ text: Text tensor or string input.+ """+ text = text.strip()++ # stage 1 : normalize numbers+ text = cls._normalize_numbers(text)++ # stage 2 : normalize english text+ text = cls._normalize_english_text(text)+ return text+</code></pre></details></div></div>+<div class="api-detail" id="e324"><span class="dtag badge-m">method</span><h2><code>KoreanCleaner.normalize_text(cls, text)</code></h2><div class="dmod">funasr.tokenizer.korean_cleaner.KoreanCleaner · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/tokenizer/korean_cleaner.py#L75" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Normalize text.</p>+<h4>Args:</h4>+<ul>+<li><code>text</code> — Text tensor or string input.</li>+</ul>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def normalize_text(cls, text):+ # stage 0 : text strip+ """Normalize text.++ Args:+ text: Text tensor or string input.+ """+ text = text.strip()++ # stage 1 : normalize numbers+ text = cls._normalize_numbers(text)++ # stage 2 : normalize english text+ text = cls._normalize_english_text(text)+ return text+</code></pre></details></div>+<div class="api-detail" id="e325"><span class="dtag badge-f">function</span><h2><code>split_by_space(text)</code></h2><div class="dmod">funasr.tokenizer.phoneme_tokenizer · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/tokenizer/phoneme_tokenizer.py#L45" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Split by space.</p>+<h4>Args:</h4>+<ul>+<li><code>text</code> — Text tensor or string input.</li>+</ul>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code>def split_by_space(text) -> List[str]:+ """Split by space.++ Args:+ text: Text tensor or string input.+ """+ if " " in text:+ text = text.replace(" ", " <space> ")+ return [c.replace("<space>", " ") for c in text.split(" ")]+ else:+ return text.split(" ")+</code></pre></details></div>+<div class="api-detail" id="e326"><span class="dtag badge-f">function</span><h2><code>pyopenjtalk_g2p(text)</code></h2><div class="dmod">funasr.tokenizer.phoneme_tokenizer · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/tokenizer/phoneme_tokenizer.py#L58" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Pyopenjtalk g2p.</p>+<h4>Args:</h4>+<ul>+<li><code>text</code> — Text tensor or string input.</li>+</ul>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code>def pyopenjtalk_g2p(text) -> List[str]:+ """Pyopenjtalk g2p.++ Args:+ text: Text tensor or string input.+ """+ import pyopenjtalk++ # phones is a str object separated by space+ phones = pyopenjtalk.g2p(text, kana=False)+ phones = phones.split(" ")+ return phones+</code></pre></details></div>+<div class="api-detail" id="e327"><span class="dtag badge-f">function</span><h2><code>pyopenjtalk_g2p_accent(text)</code></h2><div class="dmod">funasr.tokenizer.phoneme_tokenizer · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/tokenizer/phoneme_tokenizer.py#L72" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Pyopenjtalk g2p accent.</p>+<h4>Args:</h4>+<ul>+<li><code>text</code> — Text tensor or string input.</li>+</ul>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code>def pyopenjtalk_g2p_accent(text) -> List[str]:+ """Pyopenjtalk g2p accent.++ Args:+ text: Text tensor or string input.+ """+ import pyopenjtalk+ import re++ phones = []+ for labels in pyopenjtalk.run_frontend(text)[1]:+ p = re.findall(r"\-(.*?)\+.*?\/A:([0-9\-]+).*?\/F:.*?_([0-9]+)", labels)+ if len(p) == 1:+ phones += [p[0][0], p[0][2], p[0][1]]+ return phones+</code></pre></details></div>+<div class="api-detail" id="e328"><span class="dtag badge-f">function</span><h2><code>pyopenjtalk_g2p_accent_with_pause(text)</code></h2><div class="dmod">funasr.tokenizer.phoneme_tokenizer · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/tokenizer/phoneme_tokenizer.py#L89" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Pyopenjtalk g2p accent with pause.</p>+<h4>Args:</h4>+<ul>+<li><code>text</code> — Text tensor or string input.</li>+</ul>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code>def pyopenjtalk_g2p_accent_with_pause(text) -> List[str]:+ """Pyopenjtalk g2p accent with pause.++ Args:+ text: Text tensor or string input.+ """+ import pyopenjtalk+ import re++ phones = []+ for labels in pyopenjtalk.run_frontend(text)[1]:+ if labels.split("-")[1].split("+")[0] == "pau":+ phones += ["pau"]+ continue+ p = re.findall(r"\-(.*?)\+.*?\/A:([0-9\-]+).*?\/F:.*?_([0-9]+)", labels)+ if len(p) == 1:+ phones += [p[0][0], p[0][2], p[0][1]]+ return phones+</code></pre></details></div>+<div class="api-detail" id="e329"><span class="dtag badge-f">function</span><h2><code>pyopenjtalk_g2p_kana(text)</code></h2><div class="dmod">funasr.tokenizer.phoneme_tokenizer · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/tokenizer/phoneme_tokenizer.py#L109" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Pyopenjtalk g2p kana.</p>+<h4>Args:</h4>+<ul>+<li><code>text</code> — Text tensor or string input.</li>+</ul>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code>def pyopenjtalk_g2p_kana(text) -> List[str]:+ """Pyopenjtalk g2p kana.++ Args:+ text: Text tensor or string input.+ """+ import pyopenjtalk++ kanas = pyopenjtalk.g2p(text, kana=True)+ return list(kanas)+</code></pre></details></div>+<div class="api-detail" id="e330"><span class="dtag badge-f">function</span><h2><code>pyopenjtalk_g2p_prosody(text, drop_unvoiced_vowels)</code></h2><div class="dmod">funasr.tokenizer.phoneme_tokenizer · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/tokenizer/phoneme_tokenizer.py#L121" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Extract phoneme + prosoody symbol sequence from input full-context labels.</p>+<p>The algorithm is based on `Prosodic features control by symbols as input of</p>+<ul>+<li><code>sequence</code> — to-sequence acoustic modeling for neural TTS`_ with some r9y9's tweaks.</li>+</ul>+<h4>Args:</h4>+<ul>+<li><code>text (str)</code> — Input text.</li>+<li><code>drop_unvoiced_vowels (bool)</code> — whether to drop unvoiced vowels.</li>+</ul>+<h4>Returns:</h4>+<p>List[str]: List of phoneme + prosody symbols.</p>+<h4>Examples:</h4>+<p>>>> from funasr.tokenizer.phoneme_tokenizer import pyopenjtalk_g2p_prosody</p>+<p>>>> pyopenjtalk_g2p_prosody("こんにちは。")</p>+<p>['^', 'k', 'o', '[', 'N', 'n', 'i', 'ch', 'i', 'w', 'a', '$']</p>+<p>.. _`Prosodic features control by symbols as input of sequence-to-sequence acoustic</p>+<p>modeling for neural TTS`: https://doi.org/10.1587/transinf.2020EDP7104</p>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code>def pyopenjtalk_g2p_prosody(text: str, drop_unvoiced_vowels: bool = True) -> List[str]:+ """Extract phoneme + prosoody symbol sequence from input full-context labels.++ The algorithm is based on `Prosodic features control by symbols as input of+ sequence-to-sequence acoustic modeling for neural TTS`_ with some r9y9's tweaks.++ Args:+ text (str): Input text.+ drop_unvoiced_vowels (bool): whether to drop unvoiced vowels.++ Returns:+ List[str]: List of phoneme + prosody symbols.++ Examples:+ >>> from funasr.tokenizer.phoneme_tokenizer import pyopenjtalk_g2p_prosody+ >>> pyopenjtalk_g2p_prosody("こんにちは。")+ ['^', 'k', 'o', '[', 'N', 'n', 'i', 'ch', 'i', 'w', 'a', '$']++ .. _`Prosodic features control by symbols as input of sequence-to-sequence acoustic+ modeling for neural TTS`: https://doi.org/10.1587/transinf.2020EDP7104++ """+ import pyopenjtalk++ labels = pyopenjtalk.run_frontend(text)[1]+ N = len(labels)++ phones = []+ for n in range(N):+ lab_curr = labels[n]+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/tokenizer/phoneme_tokenizer.py#L121" target="_blank" class="src-more">View full source on GitHub →</a></details></div>+<div class="api-detail" id="e331"><span class="dtag badge-f">function</span><h2><code>pypinyin_g2p(text)</code></h2><div class="dmod">funasr.tokenizer.phoneme_tokenizer · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/tokenizer/phoneme_tokenizer.py#L213" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Pypinyin g2p.</p>+<h4>Args:</h4>+<ul>+<li><code>text</code> — Text tensor or string input.</li>+</ul>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code>def pypinyin_g2p(text) -> List[str]:+ """Pypinyin g2p.++ Args:+ text: Text tensor or string input.+ """+ from pypinyin import pinyin+ from pypinyin import Style++ phones = [phone[0] for phone in pinyin(text, style=Style.TONE3)]+ return phones+</code></pre></details></div>+<div class="api-detail" id="e332"><span class="dtag badge-f">function</span><h2><code>pypinyin_g2p_phone(text)</code></h2><div class="dmod">funasr.tokenizer.phoneme_tokenizer · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/tokenizer/phoneme_tokenizer.py#L226" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Pypinyin g2p phone.</p>+<h4>Args:</h4>+<ul>+<li><code>text</code> — Text tensor or string input.</li>+</ul>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code>def pypinyin_g2p_phone(text) -> List[str]:+ """Pypinyin g2p phone.++ Args:+ text: Text tensor or string input.+ """+ from pypinyin import pinyin+ from pypinyin import Style+ from pypinyin.style._utils import get_finals+ from pypinyin.style._utils import get_initials++ phones = [+ p+ for phone in pinyin(text, style=Style.TONE3)+ for p in [+ get_initials(phone[0], strict=True),+ get_finals(phone[0], strict=True),+ ]+ if len(p) != 0+ ]+ return phones+</code></pre></details></div>+<div class="api-detail" id="e333"><span class="dtag badge-c">class</span><h2>G2p_en</h2><div class="dmod">funasr.tokenizer.phoneme_tokenizer · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/tokenizer/phoneme_tokenizer.py#L249" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>On behalf of g2p_en.G2p.</p>+<p>g2p_en.G2p isn't pickalable and it can't be copied to the other processes</p>+<p>via multiprocessing module.</p>+<p>As a workaround, g2p_en.G2p is instantiated upon calling this class.</p>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code>class G2p_en:+ """On behalf of g2p_en.G2p.++ g2p_en.G2p isn't pickalable and it can't be copied to the other processes+ via multiprocessing module.+ As a workaround, g2p_en.G2p is instantiated upon calling this class.++ """++ def __init__(self, no_space: bool = False):+ """Initialize G2p_en.++ Args:+ no_space: TODO.+ """+ self.no_space = no_space+ self.g2p = None++ def __call__(self, text) -> List[str]:+ """Internal: call .++ Args:+ text: Text tensor or string input.+ """+ if self.g2p is None:+ self.g2p = g2p_en.G2p()++ phones = self.g2p(text)+ if self.no_space:+ # remove space which represents word serapater+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/tokenizer/phoneme_tokenizer.py#L249" target="_blank" class="src-more">View full source on GitHub →</a></details></div>+<div class="api-detail" id="e334"><span class="dtag badge-c">class</span><h2>G2pk</h2><div class="dmod">funasr.tokenizer.phoneme_tokenizer · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/tokenizer/phoneme_tokenizer.py#L283" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>On behalf of g2pk.G2p.</p>+<p>g2pk.G2p isn't pickalable and it can't be copied to the other processes</p>+<p>via multiprocessing module.</p>+<p>As a workaround, g2pk.G2p is instantiated upon calling this class.</p>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code>class G2pk:+ """On behalf of g2pk.G2p.++ g2pk.G2p isn't pickalable and it can't be copied to the other processes+ via multiprocessing module.+ As a workaround, g2pk.G2p is instantiated upon calling this class.++ """++ def __init__(self, descritive=False, group_vowels=False, to_syl=False, no_space=False):+ """Initialize G2pk.++ Args:+ descritive: TODO.+ group_vowels: TODO.+ to_syl: TODO.+ no_space: TODO.+ """+ self.descritive = descritive+ self.group_vowels = group_vowels+ self.to_syl = to_syl+ self.no_space = no_space+ self.g2p = None++ def __call__(self, text) -> List[str]:+ """Internal: call .++ Args:+ text: Text tensor or string input.+ """+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/tokenizer/phoneme_tokenizer.py#L283" target="_blank" class="src-more">View full source on GitHub →</a></details></div>+<div class="api-detail" id="e335"><span class="dtag badge-c">class</span><h2>Jaso</h2><div class="dmod">funasr.tokenizer.phoneme_tokenizer · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/tokenizer/phoneme_tokenizer.py#L332" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p class="muted">No documentation yet.</p></div><details class="src-block"><summary>📄 Source code</summary><pre><code>class Jaso:+ PUNC = "!'(),-.:;?"+ SPACE = " "++ JAMO_LEADS = "".join([chr(_) for _ in range(0x1100, 0x1113)])+ JAMO_VOWELS = "".join([chr(_) for _ in range(0x1161, 0x1176)])+ JAMO_TAILS = "".join([chr(_) for _ in range(0x11A8, 0x11C3)])++ VALID_CHARS = JAMO_LEADS + JAMO_VOWELS + JAMO_TAILS + PUNC + SPACE++ def __init__(self, space_symbol=" ", no_space=False):+ """Initialize Jaso.++ Args:+ space_symbol: TODO.+ no_space: TODO.+ """+ self.space_symbol = space_symbol+ self.no_space = no_space++ def _text_to_jaso(self, line: str) -> List[str]:+ """Internal: text to jaso.++ Args:+ line: TODO.+ """+ jasos = list(jamo.hangul_to_jamo(line))+ return jasos++ def _remove_non_korean_characters(self, tokens):+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/tokenizer/phoneme_tokenizer.py#L332" target="_blank" class="src-more">View full source on GitHub →</a></details></div>+<div class="api-detail" id="e336"><span class="dtag badge-c">class</span><h2>Phonemizer</h2><div class="dmod">funasr.tokenizer.phoneme_tokenizer · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/tokenizer/phoneme_tokenizer.py#L386" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Phonemizer module for various languages.</p>+<p>This is wrapper module of https://github.com/bootphon/phonemizer.</p>+<p>You can define various g2p modules by specifying options for phonemizer.</p>+<p>See available options:</p>+<p>https://github.com/bootphon/phonemizer/blob/master/phonemizer/phonemize.py#L32</p>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code>class Phonemizer:+ """Phonemizer module for various languages.++ This is wrapper module of https://github.com/bootphon/phonemizer.+ You can define various g2p modules by specifying options for phonemizer.++ See available options:+ https://github.com/bootphon/phonemizer/blob/master/phonemizer/phonemize.py#L32++ """++ def __init__(+ self,+ backend,+ word_separator: Optional[str] = None,+ syllable_separator: Optional[str] = None,+ phone_separator: Optional[str] = " ",+ strip=False,+ split_by_single_token: bool = False,+ **phonemizer_kwargs,+ ):+ # delayed import+ """Initialize Phonemizer.++ Args:+ backend: TODO.+ word_separator: TODO.+ syllable_separator: TODO.+ phone_separator: TODO.+ strip: TODO.+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/tokenizer/phoneme_tokenizer.py#L386" target="_blank" class="src-more">View full source on GitHub →</a></details></div>+<div class="api-detail" id="e337"><span class="dtag badge-c">class</span><h2>PhonemeTokenizer</h2><div class="dmod">funasr.tokenizer.phoneme_tokenizer · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/tokenizer/phoneme_tokenizer.py#L458" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p class="muted">No documentation yet.</p></div><details class="src-block"><summary>📄 Source code</summary><pre><code>class PhonemeTokenizer(AbsTokenizer):+ def __init__(+ self,+ g2p_type: Union[None, str],+ non_linguistic_symbols: Union[Path, str, Iterable[str]] = None,+ space_symbol: str = "<space>",+ remove_non_linguistic_symbols: bool = False,+ ):+ """Initialize PhonemeTokenizer.++ Args:+ g2p_type: TODO.+ non_linguistic_symbols: TODO.+ space_symbol: TODO.+ remove_non_linguistic_symbols: TODO.+ """+ if g2p_type is None:+ self.g2p = split_by_space+ elif g2p_type == "g2p_en":+ self.g2p = G2p_en(no_space=False)+ elif g2p_type == "g2p_en_no_space":+ self.g2p = G2p_en(no_space=True)+ elif g2p_type == "pyopenjtalk":+ self.g2p = pyopenjtalk_g2p+ elif g2p_type == "pyopenjtalk_kana":+ self.g2p = pyopenjtalk_g2p_kana+ elif g2p_type == "pyopenjtalk_accent":+ self.g2p = pyopenjtalk_g2p_accent+ elif g2p_type == "pyopenjtalk_accent_with_pause":+ self.g2p = pyopenjtalk_g2p_accent_with_pause+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/tokenizer/phoneme_tokenizer.py#L458" target="_blank" class="src-more">View full source on GitHub →</a></details><h3 class="mt">Methods</h3><div class="mblk"><code>.text2tokens(line)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/tokenizer/phoneme_tokenizer.py#L614" target="_blank" class="gh-link">L614</a><div class="mdoc"><p>Text2tokens.</p>+<h4>Args:</h4>+<ul>+<li><code>line</code> — TODO.</li>+</ul>+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def text2tokens(self, line: str) -> List[str]:+ """Text2tokens.++ Args:+ line: TODO.+ """+ tokens = []+ while len(line) != 0:+ for w in self.non_linguistic_symbols:+ if line.startswith(w):+ if not self.remove_non_linguistic_symbols:+ tokens.append(line[: len(w)])+ line = line[len(w) :]+ break+ else:+ t = line[0]+ tokens.append(t)+ line = line[1:]++ line = "".join(tokens)+ tokens = self.g2p(line)+ return tokens+</code></pre></details></div><div class="mblk"><code>.tokens2text(tokens)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/tokenizer/phoneme_tokenizer.py#L637" target="_blank" class="gh-link">L637</a><div class="mdoc"><p>Tokens2text.</p>+<h4>Args:</h4>+<ul>+<li><code>tokens</code> — TODO.</li>+</ul>+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def tokens2text(self, tokens: Iterable[str]) -> str:+ # phoneme type is not invertible+ """Tokens2text.++ Args:+ tokens: TODO.+ """+ return "".join(tokens)+</code></pre></details></div></div>+<div class="api-detail" id="e338"><span class="dtag badge-m">method</span><h2><code>PhonemeTokenizer.text2tokens(line)</code></h2><div class="dmod">funasr.tokenizer.phoneme_tokenizer.PhonemeTokenizer · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/tokenizer/phoneme_tokenizer.py#L614" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Text2tokens.</p>+<h4>Args:</h4>+<ul>+<li><code>line</code> — TODO.</li>+</ul>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def text2tokens(self, line: str) -> List[str]:+ """Text2tokens.++ Args:+ line: TODO.+ """+ tokens = []+ while len(line) != 0:+ for w in self.non_linguistic_symbols:+ if line.startswith(w):+ if not self.remove_non_linguistic_symbols:+ tokens.append(line[: len(w)])+ line = line[len(w) :]+ break+ else:+ t = line[0]+ tokens.append(t)+ line = line[1:]++ line = "".join(tokens)+ tokens = self.g2p(line)+ return tokens+</code></pre></details></div>+<div class="api-detail" id="e339"><span class="dtag badge-m">method</span><h2><code>PhonemeTokenizer.tokens2text(tokens)</code></h2><div class="dmod">funasr.tokenizer.phoneme_tokenizer.PhonemeTokenizer · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/tokenizer/phoneme_tokenizer.py#L637" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Tokens2text.</p>+<h4>Args:</h4>+<ul>+<li><code>tokens</code> — TODO.</li>+</ul>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def tokens2text(self, tokens: Iterable[str]) -> str:+ # phoneme type is not invertible+ """Tokens2text.++ Args:+ tokens: TODO.+ """+ return "".join(tokens)+</code></pre></details></div>+<div class="api-detail" id="e340"><span class="dtag badge-c">class</span><h2>SentencepiecesTokenizer</h2><div class="dmod">funasr.tokenizer.sentencepiece_tokenizer · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/tokenizer/sentencepiece_tokenizer.py#L13" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p class="muted">No documentation yet.</p></div><details class="src-block"><summary>📄 Source code</summary><pre><code>class SentencepiecesTokenizer(BaseTokenizer):+ def __init__(self, bpemodel: Union[Path, str], **kwargs):+ """Initialize SentencepiecesTokenizer.++ Args:+ bpemodel: TODO.+ **kwargs: Additional keyword arguments.+ """+ super().__init__(**kwargs)+ self.bpemodel = str(bpemodel)+ # NOTE(kamo):+ # Don't build SentencePieceProcessor in __init__()+ # because it's not picklable and it may cause following error,+ # "TypeError: can't pickle SwigPyObject objects",+ # when giving it as argument of "multiprocessing.Process()".+ self.sp = None+ self._build_sentence_piece_processor()++ def __repr__(self):+ """Internal: repr ."""+ return f'{self.__class__.__name__}(model="{self.bpemodel}")'++ def _build_sentence_piece_processor(self):+ # Build SentencePieceProcessor lazily.+ """Internal: build sentence piece processor."""+ if self.sp is None:+ self.sp = spm.SentencePieceProcessor()+ self.sp.load(self.bpemodel)++ def text2tokens(self, line: str) -> List[str]:+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/tokenizer/sentencepiece_tokenizer.py#L13" target="_blank" class="src-more">View full source on GitHub →</a></details><h3 class="mt">Methods</h3><div class="mblk"><code>.text2tokens(line)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/tokenizer/sentencepiece_tokenizer.py#L42" target="_blank" class="gh-link">L42</a><div class="mdoc"><p>Text2tokens.</p>+<h4>Args:</h4>+<ul>+<li><code>line</code> — TODO.</li>+</ul>+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def text2tokens(self, line: str) -> List[str]:+ """Text2tokens.++ Args:+ line: TODO.+ """+ self._build_sentence_piece_processor()+ return self.sp.EncodeAsPieces(line)+</code></pre></details></div><div class="mblk"><code>.tokens2text(tokens)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/tokenizer/sentencepiece_tokenizer.py#L51" target="_blank" class="gh-link">L51</a><div class="mdoc"><p>Tokens2text.</p>+<h4>Args:</h4>+<ul>+<li><code>tokens</code> — TODO.</li>+</ul>+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def tokens2text(self, tokens: Iterable[str]) -> str:+ """Tokens2text.++ Args:+ tokens: TODO.+ """+ self._build_sentence_piece_processor()+ return self.sp.DecodePieces(list(tokens))+</code></pre></details></div><div class="mblk"><code>.encode(line, **kwargs)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/tokenizer/sentencepiece_tokenizer.py#L60" target="_blank" class="gh-link">L60</a><div class="mdoc"><p>Encode.</p>+<h4>Args:</h4>+<ul>+<li><code>line</code> — TODO.</li>+<li><code>**kwargs</code> — Additional keyword arguments.</li>+</ul>+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def encode(self, line: str, **kwargs) -> List[int]:+ """Encode.++ Args:+ line: TODO.+ **kwargs: Additional keyword arguments.+ """+ self._build_sentence_piece_processor()+ return self.sp.EncodeAsIds(line)+</code></pre></details></div><div class="mblk"><code>.decode(line, **kwargs)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/tokenizer/sentencepiece_tokenizer.py#L70" target="_blank" class="gh-link">L70</a><div class="mdoc"><p>Decode.</p>+<h4>Args:</h4>+<ul>+<li><code>line</code> — TODO.</li>+<li><code>**kwargs</code> — Additional keyword arguments.</li>+</ul>+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def decode(self, line: List[int], **kwargs):+ """Decode.++ Args:+ line: TODO.+ **kwargs: Additional keyword arguments.+ """+ self._build_sentence_piece_processor()+ return self.sp.DecodeIds(line)+</code></pre></details></div><div class="mblk"><code>.get_vocab_size()</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/tokenizer/sentencepiece_tokenizer.py#L80" target="_blank" class="gh-link">L80</a><div class="mdoc"><p>Get vocab size.</p>+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def get_vocab_size(self):+ """Get vocab size."""+ return self.sp.GetPieceSize()+</code></pre></details></div><div class="mblk"><code>.ids2tokens(*args, **kwargs)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/tokenizer/sentencepiece_tokenizer.py#L84" target="_blank" class="gh-link">L84</a><div class="mdoc"><p>Ids2tokens.</p>+<h4>Args:</h4>+<ul>+<li><code>*args</code> — Variable positional arguments.</li>+<li><code>**kwargs</code> — Additional keyword arguments.</li>+</ul>+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def ids2tokens(self, *args, **kwargs):+ """Ids2tokens.++ Args:+ *args: Variable positional arguments.+ **kwargs: Additional keyword arguments.+ """+ return self.decode(*args, **kwargs)+</code></pre></details></div><div class="mblk"><code>.tokens2ids(*args, **kwargs)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/tokenizer/sentencepiece_tokenizer.py#L93" target="_blank" class="gh-link">L93</a><div class="mdoc"><p>Tokens2ids.</p>+<h4>Args:</h4>+<ul>+<li><code>*args</code> — Variable positional arguments.</li>+<li><code>**kwargs</code> — Additional keyword arguments.</li>+</ul>+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def tokens2ids(self, *args, **kwargs):+ """Tokens2ids.++ Args:+ *args: Variable positional arguments.+ **kwargs: Additional keyword arguments.+ """+ return self.encode(*args, **kwargs)+</code></pre></details></div></div>+<div class="api-detail" id="e341"><span class="dtag badge-m">method</span><h2><code>SentencepiecesTokenizer.text2tokens(line)</code></h2><div class="dmod">funasr.tokenizer.sentencepiece_tokenizer.SentencepiecesTokenizer · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/tokenizer/sentencepiece_tokenizer.py#L42" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Text2tokens.</p>+<h4>Args:</h4>+<ul>+<li><code>line</code> — TODO.</li>+</ul>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def text2tokens(self, line: str) -> List[str]:+ """Text2tokens.++ Args:+ line: TODO.+ """+ self._build_sentence_piece_processor()+ return self.sp.EncodeAsPieces(line)+</code></pre></details></div>+<div class="api-detail" id="e342"><span class="dtag badge-m">method</span><h2><code>SentencepiecesTokenizer.tokens2text(tokens)</code></h2><div class="dmod">funasr.tokenizer.sentencepiece_tokenizer.SentencepiecesTokenizer · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/tokenizer/sentencepiece_tokenizer.py#L51" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Tokens2text.</p>+<h4>Args:</h4>+<ul>+<li><code>tokens</code> — TODO.</li>+</ul>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def tokens2text(self, tokens: Iterable[str]) -> str:+ """Tokens2text.++ Args:+ tokens: TODO.+ """+ self._build_sentence_piece_processor()+ return self.sp.DecodePieces(list(tokens))+</code></pre></details></div>+<div class="api-detail" id="e343"><span class="dtag badge-m">method</span><h2><code>SentencepiecesTokenizer.encode(line, **kwargs)</code></h2><div class="dmod">funasr.tokenizer.sentencepiece_tokenizer.SentencepiecesTokenizer · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/tokenizer/sentencepiece_tokenizer.py#L60" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Encode.</p>+<h4>Args:</h4>+<ul>+<li><code>line</code> — TODO.</li>+<li><code>**kwargs</code> — Additional keyword arguments.</li>+</ul>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def encode(self, line: str, **kwargs) -> List[int]:+ """Encode.++ Args:+ line: TODO.+ **kwargs: Additional keyword arguments.+ """+ self._build_sentence_piece_processor()+ return self.sp.EncodeAsIds(line)+</code></pre></details></div>+<div class="api-detail" id="e344"><span class="dtag badge-m">method</span><h2><code>SentencepiecesTokenizer.decode(line, **kwargs)</code></h2><div class="dmod">funasr.tokenizer.sentencepiece_tokenizer.SentencepiecesTokenizer · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/tokenizer/sentencepiece_tokenizer.py#L70" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Decode.</p>+<h4>Args:</h4>+<ul>+<li><code>line</code> — TODO.</li>+<li><code>**kwargs</code> — Additional keyword arguments.</li>+</ul>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def decode(self, line: List[int], **kwargs):+ """Decode.++ Args:+ line: TODO.+ **kwargs: Additional keyword arguments.+ """+ self._build_sentence_piece_processor()+ return self.sp.DecodeIds(line)+</code></pre></details></div>+<div class="api-detail" id="e345"><span class="dtag badge-m">method</span><h2><code>SentencepiecesTokenizer.get_vocab_size()</code></h2><div class="dmod">funasr.tokenizer.sentencepiece_tokenizer.SentencepiecesTokenizer · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/tokenizer/sentencepiece_tokenizer.py#L80" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Get vocab size.</p>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def get_vocab_size(self):+ """Get vocab size."""+ return self.sp.GetPieceSize()+</code></pre></details></div>+<div class="api-detail" id="e346"><span class="dtag badge-m">method</span><h2><code>SentencepiecesTokenizer.ids2tokens(*args, **kwargs)</code></h2><div class="dmod">funasr.tokenizer.sentencepiece_tokenizer.SentencepiecesTokenizer · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/tokenizer/sentencepiece_tokenizer.py#L84" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Ids2tokens.</p>+<h4>Args:</h4>+<ul>+<li><code>*args</code> — Variable positional arguments.</li>+<li><code>**kwargs</code> — Additional keyword arguments.</li>+</ul>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def ids2tokens(self, *args, **kwargs):+ """Ids2tokens.++ Args:+ *args: Variable positional arguments.+ **kwargs: Additional keyword arguments.+ """+ return self.decode(*args, **kwargs)+</code></pre></details></div>+<div class="api-detail" id="e347"><span class="dtag badge-m">method</span><h2><code>SentencepiecesTokenizer.tokens2ids(*args, **kwargs)</code></h2><div class="dmod">funasr.tokenizer.sentencepiece_tokenizer.SentencepiecesTokenizer · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/tokenizer/sentencepiece_tokenizer.py#L93" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Tokens2ids.</p>+<h4>Args:</h4>+<ul>+<li><code>*args</code> — Variable positional arguments.</li>+<li><code>**kwargs</code> — Additional keyword arguments.</li>+</ul>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def tokens2ids(self, *args, **kwargs):+ """Tokens2ids.++ Args:+ *args: Variable positional arguments.+ **kwargs: Additional keyword arguments.+ """+ return self.encode(*args, **kwargs)+</code></pre></details></div>+<div class="api-detail" id="e348"><span class="dtag badge-c">class</span><h2>TokenIDConverter</h2><div class="dmod">funasr.tokenizer.token_id_converter · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/tokenizer/token_id_converter.py#L10" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p class="muted">No documentation yet.</p></div><details class="src-block"><summary>📄 Source code</summary><pre><code>class TokenIDConverter:+ def __init__(+ self,+ token_list: Union[Path, str, Iterable[str]],+ unk_symbol: str = "<unk>",+ ):++ """Initialize TokenIDConverter.++ Args:+ token_list: TODO.+ unk_symbol: TODO.+ """+ if isinstance(token_list, (Path, str)):+ token_list = Path(token_list)+ self.token_list_repr = str(token_list)+ self.token_list: List[str] = []++ with token_list.open("r", encoding="utf-8") as f:+ for idx, line in enumerate(f):+ line = line.rstrip()+ self.token_list.append(line)++ else:+ self.token_list: List[str] = list(token_list)+ self.token_list_repr = ""+ for i, t in enumerate(self.token_list):+ if i == 3:+ break+ self.token_list_repr += f"{t}, "+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/tokenizer/token_id_converter.py#L10" target="_blank" class="src-more">View full source on GitHub →</a></details><h3 class="mt">Methods</h3><div class="mblk"><code>.get_num_vocabulary_size()</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/tokenizer/token_id_converter.py#L53" target="_blank" class="gh-link">L53</a><div class="mdoc"><p>Get num vocabulary size.</p>+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def get_num_vocabulary_size(self) -> int:+ """Get num vocabulary size."""+ return len(self.token_list)+</code></pre></details></div><div class="mblk"><code>.ids2tokens(integers)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/tokenizer/token_id_converter.py#L57" target="_blank" class="gh-link">L57</a><div class="mdoc"><p>Ids2tokens.</p>+<h4>Args:</h4>+<ul>+<li><code>integers</code> — TODO.</li>+</ul>+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def ids2tokens(self, integers: Union[np.ndarray, Iterable[int]]) -> List[str]:+ """Ids2tokens.++ Args:+ integers: TODO.+ """+ if isinstance(integers, np.ndarray) and integers.ndim != 1:+ raise ValueError(f"Must be 1 dim ndarray, but got {integers.ndim}")+ return [self.token_list[i] for i in integers]+</code></pre></details></div><div class="mblk"><code>.tokens2ids(tokens)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/tokenizer/token_id_converter.py#L67" target="_blank" class="gh-link">L67</a><div class="mdoc"><p>Tokens2ids.</p>+<h4>Args:</h4>+<ul>+<li><code>tokens</code> — TODO.</li>+</ul>+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def tokens2ids(self, tokens: Iterable[str]) -> List[int]:+ """Tokens2ids.++ Args:+ tokens: TODO.+ """+ return [self.token2id.get(i, self.unk_id) for i in tokens]+</code></pre></details></div></div>+<div class="api-detail" id="e349"><span class="dtag badge-m">method</span><h2><code>TokenIDConverter.get_num_vocabulary_size()</code></h2><div class="dmod">funasr.tokenizer.token_id_converter.TokenIDConverter · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/tokenizer/token_id_converter.py#L53" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Get num vocabulary size.</p>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def get_num_vocabulary_size(self) -> int:+ """Get num vocabulary size."""+ return len(self.token_list)+</code></pre></details></div>+<div class="api-detail" id="e350"><span class="dtag badge-m">method</span><h2><code>TokenIDConverter.ids2tokens(integers)</code></h2><div class="dmod">funasr.tokenizer.token_id_converter.TokenIDConverter · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/tokenizer/token_id_converter.py#L57" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Ids2tokens.</p>+<h4>Args:</h4>+<ul>+<li><code>integers</code> — TODO.</li>+</ul>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def ids2tokens(self, integers: Union[np.ndarray, Iterable[int]]) -> List[str]:+ """Ids2tokens.++ Args:+ integers: TODO.+ """+ if isinstance(integers, np.ndarray) and integers.ndim != 1:+ raise ValueError(f"Must be 1 dim ndarray, but got {integers.ndim}")+ return [self.token_list[i] for i in integers]+</code></pre></details></div>+<div class="api-detail" id="e351"><span class="dtag badge-m">method</span><h2><code>TokenIDConverter.tokens2ids(tokens)</code></h2><div class="dmod">funasr.tokenizer.token_id_converter.TokenIDConverter · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/tokenizer/token_id_converter.py#L67" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Tokens2ids.</p>+<h4>Args:</h4>+<ul>+<li><code>tokens</code> — TODO.</li>+</ul>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def tokens2ids(self, tokens: Iterable[str]) -> List[int]:+ """Tokens2ids.++ Args:+ tokens: TODO.+ """+ return [self.token2id.get(i, self.unk_id) for i in tokens]+</code></pre></details></div>+<div class="api-detail" id="e352"><span class="dtag badge-f">function</span><h2><code>WhisperTokenizer(**kwargs)</code></h2><div class="dmod">funasr.tokenizer.whisper_tokenizer · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/tokenizer/whisper_tokenizer.py#L5" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Whispertokenizer.</p>+<h4>Args:</h4>+<ul>+<li><code>**kwargs</code> — Additional keyword arguments.</li>+</ul>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code>def WhisperTokenizer(**kwargs):+ """Whispertokenizer.++ Args:+ **kwargs: Additional keyword arguments.+ """+ try:+ from whisper.tokenizer import get_tokenizer+ except:+ print("Notice: If you want to use whisper, please `pip install -U openai-whisper`")++ language = kwargs.get("language", None)+ task = kwargs.get("task", "transcribe")+ is_multilingual = kwargs.get("is_multilingual", True)+ num_languages = kwargs.get("num_languages", 99)+ tokenizer = get_tokenizer(+ multilingual=is_multilingual,+ num_languages=num_languages,+ language=language,+ task=task,+ )++ return tokenizer+</code></pre></details></div>+<div class="api-detail" id="e353"><span class="dtag badge-f">function</span><h2><code>SenseVoiceTokenizer(**kwargs)</code></h2><div class="dmod">funasr.tokenizer.whisper_tokenizer · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/tokenizer/whisper_tokenizer.py#L31" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Sensevoicetokenizer.</p>+<h4>Args:</h4>+<ul>+<li><code>**kwargs</code> — Additional keyword arguments.</li>+</ul>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code>def SenseVoiceTokenizer(**kwargs):+ """Sensevoicetokenizer.++ Args:+ **kwargs: Additional keyword arguments.+ """+ from funasr.models.sense_voice.whisper_lib.tokenizer import get_tokenizer++ language = kwargs.get("language", None)+ task = kwargs.get("task", None)+ is_multilingual = kwargs.get("is_multilingual", True)+ num_languages = kwargs.get("num_languages", 8749)+ vocab_path = kwargs.get("vocab_path", None)+ tokenizer = get_tokenizer(+ multilingual=is_multilingual,+ num_languages=num_languages,+ language=language,+ task=task,+ vocab_path=vocab_path,+ )++ return tokenizer+</code></pre></details></div>+<div class="api-detail" id="e354"><span class="dtag badge-c">class</span><h2>WordTokenizer</h2><div class="dmod">funasr.tokenizer.word_tokenizer · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/tokenizer/word_tokenizer.py#L11" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p class="muted">No documentation yet.</p></div><details class="src-block"><summary>📄 Source code</summary><pre><code>class WordTokenizer(AbsTokenizer):+ def __init__(+ self,+ delimiter: str = None,+ non_linguistic_symbols: Union[Path, str, Iterable[str]] = None,+ remove_non_linguistic_symbols: bool = False,+ ):+ """Initialize WordTokenizer.++ Args:+ delimiter: TODO.+ non_linguistic_symbols: TODO.+ remove_non_linguistic_symbols: TODO.+ """+ self.delimiter = delimiter++ if not remove_non_linguistic_symbols and non_linguistic_symbols is not None:+ warnings.warn(+ "non_linguistic_symbols is only used " "when remove_non_linguistic_symbols = True"+ )++ if non_linguistic_symbols is None:+ self.non_linguistic_symbols = set()+ elif isinstance(non_linguistic_symbols, (Path, str)):+ non_linguistic_symbols = Path(non_linguistic_symbols)+ try:+ with non_linguistic_symbols.open("r", encoding="utf-8") as f:+ self.non_linguistic_symbols = set(line.rstrip() for line in f)+ except FileNotFoundError:+ warnings.warn(f"{non_linguistic_symbols} doesn't exist.")+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/tokenizer/word_tokenizer.py#L11" target="_blank" class="src-more">View full source on GitHub →</a></details><h3 class="mt">Methods</h3><div class="mblk"><code>.text2tokens(line)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/tokenizer/word_tokenizer.py#L50" target="_blank" class="gh-link">L50</a><div class="mdoc"><p>Text2tokens.</p>+<h4>Args:</h4>+<ul>+<li><code>line</code> — TODO.</li>+</ul>+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def text2tokens(self, line: str) -> List[str]:+ """Text2tokens.++ Args:+ line: TODO.+ """+ tokens = []+ for t in line.split(self.delimiter):+ if self.remove_non_linguistic_symbols and t in self.non_linguistic_symbols:+ continue+ tokens.append(t)+ return tokens+</code></pre></details></div><div class="mblk"><code>.tokens2text(tokens)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/tokenizer/word_tokenizer.py#L63" target="_blank" class="gh-link">L63</a><div class="mdoc"><p>Tokens2text.</p>+<h4>Args:</h4>+<ul>+<li><code>tokens</code> — TODO.</li>+</ul>+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def tokens2text(self, tokens: Iterable[str]) -> str:+ """Tokens2text.++ Args:+ tokens: TODO.+ """+ if self.delimiter is None:+ delimiter = " "+ else:+ delimiter = self.delimiter+ return delimiter.join(tokens)+</code></pre></details></div></div>+<div class="api-detail" id="e355"><span class="dtag badge-m">method</span><h2><code>WordTokenizer.text2tokens(line)</code></h2><div class="dmod">funasr.tokenizer.word_tokenizer.WordTokenizer · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/tokenizer/word_tokenizer.py#L50" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Text2tokens.</p>+<h4>Args:</h4>+<ul>+<li><code>line</code> — TODO.</li>+</ul>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def text2tokens(self, line: str) -> List[str]:+ """Text2tokens.++ Args:+ line: TODO.+ """+ tokens = []+ for t in line.split(self.delimiter):+ if self.remove_non_linguistic_symbols and t in self.non_linguistic_symbols:+ continue+ tokens.append(t)+ return tokens+</code></pre></details></div>+<div class="api-detail" id="e356"><span class="dtag badge-m">method</span><h2><code>WordTokenizer.tokens2text(tokens)</code></h2><div class="dmod">funasr.tokenizer.word_tokenizer.WordTokenizer · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/tokenizer/word_tokenizer.py#L63" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Tokens2text.</p>+<h4>Args:</h4>+<ul>+<li><code>tokens</code> — TODO.</li>+</ul>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def tokens2text(self, tokens: Iterable[str]) -> str:+ """Tokens2text.++ Args:+ tokens: TODO.+ """+ if self.delimiter is None:+ delimiter = " "+ else:+ delimiter = self.delimiter+ return delimiter.join(tokens)+</code></pre></details></div>+<div class="api-detail" id="e357"><span class="dtag badge-c">class</span><h2>thread_wrapper</h2><div class="dmod">funasr.utils.compute_det_ctc · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/utils/compute_det_ctc.py#L14" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p class="muted">No documentation yet.</p></div><details class="src-block"><summary>📄 Source code</summary><pre><code>class thread_wrapper(threading.Thread):+ def __init__(self, func, args=()):+ """Initialize thread_wrapper.++ Args:+ func: TODO.+ args: TODO.+ """+ super(thread_wrapper, self).__init__()+ self.func = func+ self.args = args+ self.result = []++ def run(self):+ """Run."""+ self.result = self.func(*self.args)++ def get_result(self):+ """Get result."""+ try:+ return self.result+ except Exception:+ return None+</code></pre></details><h3 class="mt">Methods</h3><div class="mblk"><code>.run()</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/utils/compute_det_ctc.py#L27" target="_blank" class="gh-link">L27</a><div class="mdoc"><p>Run.</p>+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def run(self):+ """Run."""+ self.result = self.func(*self.args)+</code></pre></details></div><div class="mblk"><code>.get_result()</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/utils/compute_det_ctc.py#L31" target="_blank" class="gh-link">L31</a><div class="mdoc"><p>Get result.</p>+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def get_result(self):+ """Get result."""+ try:+ return self.result+ except Exception:+ return None+</code></pre></details></div></div>+<div class="api-detail" id="e358"><span class="dtag badge-m">method</span><h2><code>thread_wrapper.run()</code></h2><div class="dmod">funasr.utils.compute_det_ctc.thread_wrapper · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/utils/compute_det_ctc.py#L27" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Run.</p>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def run(self):+ """Run."""+ self.result = self.func(*self.args)+</code></pre></details></div>+<div class="api-detail" id="e359"><span class="dtag badge-m">method</span><h2><code>thread_wrapper.get_result()</code></h2><div class="dmod">funasr.utils.compute_det_ctc.thread_wrapper · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/utils/compute_det_ctc.py#L31" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Get result.</p>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def get_result(self):+ """Get result."""+ try:+ return self.result+ except Exception:+ return None+</code></pre></details></div>+<div class="api-detail" id="e360"><span class="dtag badge-f">function</span><h2><code>space_mixed_label(input_str)</code></h2><div class="dmod">funasr.utils.compute_det_ctc · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/utils/compute_det_ctc.py#L39" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Space mixed label.</p>+<h4>Args:</h4>+<ul>+<li><code>input_str</code> — TODO.</li>+</ul>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code>def space_mixed_label(input_str):+ """Space mixed label.++ Args:+ input_str: TODO.+ """+ splits = split_mixed_label(input_str)+ space_str = ''.join(f'{sub} ' for sub in splits)+ return space_str.strip()+</code></pre></details></div>+<div class="api-detail" id="e361"><span class="dtag badge-f">function</span><h2><code>read_lists(list_file)</code></h2><div class="dmod">funasr.utils.compute_det_ctc · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/utils/compute_det_ctc.py#L50" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Read lists.</p>+<h4>Args:</h4>+<ul>+<li><code>list_file</code> — TODO.</li>+</ul>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code>def read_lists(list_file):+ """Read lists.++ Args:+ list_file: TODO.+ """+ lists = []+ with open(list_file, 'r', encoding='utf8') as fin:+ for line in fin:+ if line.strip() != '':+ lists.append(line.strip())+ return lists+</code></pre></details></div>+<div class="api-detail" id="e362"><span class="dtag badge-f">function</span><h2><code>make_pair(wav_lists, trans_lists)</code></h2><div class="dmod">funasr.utils.compute_det_ctc · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/utils/compute_det_ctc.py#L64" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Make pair.</p>+<h4>Args:</h4>+<ul>+<li><code>wav_lists</code> — TODO.</li>+<li><code>trans_lists</code> — TODO.</li>+</ul>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code>def make_pair(wav_lists, trans_lists):+ """Make pair.++ Args:+ wav_lists: TODO.+ trans_lists: TODO.+ """+ logging.info('make pair for wav-trans list')++ trans_table = {}+ for line in trans_lists:+ arr = line.strip().replace('\t', ' ').split()+ if len(arr) < 2:+ logging.debug('invalid line in trans file: {}'.format(+ line.strip()))+ continue++ trans_table[arr[0]] = line.replace(arr[0],'').strip()++ lists = []+ for line in wav_lists:+ arr = line.strip().replace('\t', ' ').split()+ if len(arr) == 2 and arr[0] in trans_table:+ lists.append(+ dict(key=arr[0],+ txt=trans_table[arr[0]],+ wav=arr[1],+ sample_rate=16000))+ else:+ logging.debug("can't find corresponding trans for key: {}".format(+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/utils/compute_det_ctc.py#L64" target="_blank" class="src-more">View full source on GitHub →</a></details></div>+<div class="api-detail" id="e363"><span class="dtag badge-f">function</span><h2><code>count_duration(tid, data_lists)</code></h2><div class="dmod">funasr.utils.compute_det_ctc · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/utils/compute_det_ctc.py#L100" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Count duration.</p>+<h4>Args:</h4>+<ul>+<li><code>tid</code> — TODO.</li>+<li><code>data_lists</code> — TODO.</li>+</ul>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code>def count_duration(tid, data_lists):+ """Count duration.++ Args:+ tid: TODO.+ data_lists: TODO.+ """+ results = []++ for obj in data_lists:+ assert 'key' in obj+ assert 'wav' in obj+ assert 'txt' in obj+ key = obj['key']+ wav_file = obj['wav']+ txt = obj['txt']++ try:+ rate, waveform = kaldiio.load_mat(wav_file)+ waveform = torch.tensor(waveform, dtype=torch.float32)+ waveform = waveform.unsqueeze(0)+ frames = len(waveform[0])+ duration = frames / float(rate)+ except:+ logging.info(f'load file failed: {wav_file}')+ duration = 0.0++ obj['duration'] = duration+ results.append(obj)++</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/utils/compute_det_ctc.py#L100" target="_blank" class="src-more">View full source on GitHub →</a></details></div>+<div class="api-detail" id="e364"><span class="dtag badge-f">function</span><h2><code>load_data_and_score(keywords_list, data_file, trans_file, score_file)</code></h2><div class="dmod">funasr.utils.compute_det_ctc · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/utils/compute_det_ctc.py#L133" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Load data and score.</p>+<h4>Args:</h4>+<ul>+<li><code>keywords_list</code> — TODO.</li>+<li><code>data_file</code> — TODO.</li>+<li><code>trans_file</code> — TODO.</li>+<li><code>score_file</code> — TODO.</li>+</ul>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code>def load_data_and_score(keywords_list, data_file, trans_file, score_file):+ # score_table: {uttid: [keywordlist]}+ """Load data and score.++ Args:+ keywords_list: TODO.+ data_file: TODO.+ trans_file: TODO.+ score_file: TODO.+ """+ score_table = {}+ with open(score_file, 'r', encoding='utf8') as fin:+ # read score file and store in table+ for line in fin:+ arr = line.strip().split()+ key = arr[0]+ is_detected = arr[1]+ if is_detected == 'detected':+ if key not in score_table:+ score_table.update(+ {key: {+ 'kw': space_mixed_label(arr[2]),+ 'confi': float(arr[3])+ }})+ else:+ if key not in score_table:+ score_table.update({key: {'kw': 'unknown', 'confi': -1.0}})++ wav_lists = read_lists(data_file)+ trans_lists = read_lists(trans_file)+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/utils/compute_det_ctc.py#L133" target="_blank" class="src-more">View full source on GitHub →</a></details></div>+<div class="api-detail" id="e365"><span class="dtag badge-c">class</span><h2>DatadirWriter</h2><div class="dmod">funasr.utils.datadir_writer · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/utils/datadir_writer.py#L6" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Writer class to create kaldi like data directory.</p>+<h4>Examples:</h4>+<p>>>> with DatadirWriter("output") as writer:</p>+<p>... # output/sub.txt is created here</p>+<p>... subwriter = writer["sub.txt"]</p>+<p>... # Write "uttidA some/where/a.wav"</p>+<p>... subwriter["uttidA"] = "some/where/a.wav"</p>+<p>... subwriter["uttidB"] = "some/where/b.wav"</p>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code>class DatadirWriter:+ """Writer class to create kaldi like data directory.++ Examples:+ >>> with DatadirWriter("output") as writer:+ ... # output/sub.txt is created here+ ... subwriter = writer["sub.txt"]+ ... # Write "uttidA some/where/a.wav"+ ... subwriter["uttidA"] = "some/where/a.wav"+ ... subwriter["uttidB"] = "some/where/b.wav"++ """++ def __init__(self, p: Union[Path, str]):+ """Initialize DatadirWriter.++ Args:+ p: TODO.+ """+ self.path = Path(p)+ self.chilidren = {}+ self.fd = None+ self.has_children = False+ self.keys = set()++ def __enter__(self):+ """Internal: enter ."""+ return self++ def __getitem__(self, key: str) -> "DatadirWriter":+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/utils/datadir_writer.py#L6" target="_blank" class="src-more">View full source on GitHub →</a></details><h3 class="mt">Methods</h3><div class="mblk"><code>.close()</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/utils/datadir_writer.py#L82" target="_blank" class="gh-link">L82</a><div class="mdoc"><p>Close.</p>+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def close(self):+ """Close."""+ if self.has_children:+ prev_child = None+ for child in self.chilidren.values():+ child.close()+ if prev_child is not None and prev_child.keys != child.keys:+ warnings.warn(+ f"Ids are mismatching between " f"{prev_child.path} and {child.path}"+ )+ prev_child = child++ elif self.fd is not None:+ self.fd.close()+</code></pre></details></div></div>+<div class="api-detail" id="e366"><span class="dtag badge-m">method</span><h2><code>DatadirWriter.close()</code></h2><div class="dmod">funasr.utils.datadir_writer.DatadirWriter · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/utils/datadir_writer.py#L82" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Close.</p>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def close(self):+ """Close."""+ if self.has_children:+ prev_child = None+ for child in self.chilidren.values():+ child.close()+ if prev_child is not None and prev_child.keys != child.keys:+ warnings.warn(+ f"Ids are mismatching between " f"{prev_child.path} and {child.path}"+ )+ prev_child = child++ elif self.fd is not None:+ self.fd.close()+</code></pre></details></div>+<div class="api-detail" id="e367"><span class="dtag badge-f">function</span><h2><code>load_module_from_path(file_path)</code></h2><div class="dmod">funasr.utils.dynamic_import · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/utils/dynamic_import.py#L9" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>从给定的文件路径动态加载模块。</p>+<p>:param file_path: 模块文件的绝对路径。</p>+<p>:return: 加载的模块</p>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code>def load_module_from_path(file_path):+ """+ 从给定的文件路径动态加载模块。++ :param file_path: 模块文件的绝对路径。+ :return: 加载的模块+ """+ module_name = file_path.split("/")[-1].replace(".py", "")+ spec = importlib.util.spec_from_file_location(module_name, file_path)+ module = importlib.util.module_from_spec(spec)+ spec.loader.exec_module(module)+ return module+</code></pre></details></div>+<div class="api-detail" id="e368"><span class="dtag badge-f">function</span><h2><code>import_module_from_path(file_path)</code></h2><div class="dmod">funasr.utils.dynamic_import · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/utils/dynamic_import.py#L23" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Import module from path.</p>+<h4>Args:</h4>+<ul>+<li><code>file_path</code> — TODO.</li>+</ul>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code>def import_module_from_path(file_path: str):++ """Import module from path.++ Args:+ file_path: TODO.+ """+ if file_path.startswith("http"):+ from funasr.download.file import download_from_url++ file_path = download_from_url(file_path)++ file_dir = os.path.dirname(file_path)+ # file_name = os.path.basename(file_path)+ module_name = file_path.split("/")[-1].replace(".py", "")+ if len(file_dir) < 1:+ file_dir = "./"+ sys.path.append(file_dir)+ try:+ importlib.import_module(module_name)+ print(f"Loading remote code successfully: {file_path}")+ except Exception as e:+ print(f"Loading remote code failed: {file_path}, {e}")+</code></pre></details></div>+<div class="api-detail" id="e369"><span class="dtag badge-f">function</span><h2><code>export(model, data_in, quantize, opset_version, type, **kwargs)</code></h2><div class="dmod">funasr.utils.export_utils · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/utils/export_utils.py#L10" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Export.</p>+<h4>Args:</h4>+<ul>+<li><code>model</code> — Model instance or model name.</li>+<li><code>data_in</code> — Input data (audio samples, file paths, or text).</li>+<li><code>quantize</code> — TODO.</li>+<li><code>opset_version</code> — TODO.</li>+<li><code>type</code> — TODO.</li>+<li><code>**kwargs</code> — Additional keyword arguments.</li>+</ul>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code>def export(+ model, data_in=None, quantize: bool = False, opset_version: int = 14, type="onnx", **kwargs+):+ """Export.++ Args:+ model: Model instance or model name.+ data_in: Input data (audio samples, file paths, or text).+ quantize: TODO.+ opset_version: TODO.+ type: TODO.+ **kwargs: Additional keyword arguments.+ """+ model_scripts = model.export(**kwargs)+ export_dir = kwargs.get("output_dir", os.path.dirname(kwargs.get("init_param")))+ os.makedirs(export_dir, exist_ok=True)++ if not isinstance(model_scripts, (list, tuple)):+ model_scripts = (model_scripts,)+ for m in model_scripts:+ m.eval()+ if type == "onnx":+ _onnx(+ m,+ data_in=data_in,+ quantize=quantize,+ opset_version=opset_version,+ export_dir=export_dir,+ **kwargs,+ )+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/utils/export_utils.py#L10" target="_blank" class="src-more">View full source on GitHub →</a></details></div>+<div class="api-detail" id="e370"><span class="dtag badge-f">function</span><h2><code>install_requirements(requirements_path)</code></h2><div class="dmod">funasr.utils.install_model_requirements · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/utils/install_model_requirements.py#L5" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Install requirements.</p>+<h4>Args:</h4>+<ul>+<li><code>requirements_path</code> — TODO.</li>+</ul>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code>def install_requirements(requirements_path):+ """Install requirements.++ Args:+ requirements_path: TODO.+ """+ try:+ result = pip_install_r(requirements_path)+ # check status+ if result.returncode == 0:+ print("install model requirements successfully")+ return True+ else:+ print("fail to install model requirements! ")+ print("error", result.stderr)+ return False+ except Exception as e:+ result = pip_install_r(requirements_path)+ # check status+ if result.returncode == 0:+ print("install model requirements successfully")+ return True+ else:+ print("fail to install model requirements! ")+ print("error", result.stderr)+ return False+</code></pre></details></div>+<div class="api-detail" id="e371"><span class="dtag badge-f">function</span><h2><code>pip_install_r(requirements_path)</code></h2><div class="dmod">funasr.utils.install_model_requirements · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/utils/install_model_requirements.py#L33" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Pip install r.</p>+<h4>Args:</h4>+<ul>+<li><code>requirements_path</code> — TODO.</li>+</ul>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code>def pip_install_r(requirements_path):+ """Pip install r.++ Args:+ requirements_path: TODO.+ """+ cmd = []+ if shutil.which("pip") is not None:+ cmd = ["pip"]+ elif shutil.which("uv") is not None:+ cmd = ["uv", "pip"]+ else:+ raise RuntimeError("pip not found, failed to install model requirements")+ cmd += ["install", "-r", requirements_path]+ return subprocess.run(+ cmd,+ stdout=subprocess.PIPE,+ stderr=subprocess.PIPE,+ text=True,+ )+</code></pre></details></div>+<div class="api-detail" id="e372"><span class="dtag badge-f">function</span><h2><code>split_mixed_label(input_str)</code></h2><div class="dmod">funasr.utils.kws_utils · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/utils/kws_utils.py#L13" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Split mixed label.</p>+<h4>Args:</h4>+<ul>+<li><code>input_str</code> — TODO.</li>+</ul>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code>def split_mixed_label(input_str):+ """Split mixed label.++ Args:+ input_str: TODO.+ """+ tokens = []+ s = input_str.lower()+ while len(s) > 0:+ match = re.match(r'[A-Za-z!?,<>()\']+', s)+ if match is not None:+ word = match.group(0)+ else:+ word = s[0:1]+ tokens.append(word)+ s = s.replace(word, '', 1).strip(' ')+ return tokens+</code></pre></details></div>+<div class="api-detail" id="e373"><span class="dtag badge-f">function</span><h2><code>query_token_set(txt, symbol_table, lexicon_table)</code></h2><div class="dmod">funasr.utils.kws_utils · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/utils/kws_utils.py#L32" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Query token set.</p>+<h4>Args:</h4>+<ul>+<li><code>txt</code> — TODO.</li>+<li><code>symbol_table</code> — TODO.</li>+<li><code>lexicon_table</code> — TODO.</li>+</ul>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code>def query_token_set(txt, symbol_table, lexicon_table):+ """Query token set.++ Args:+ txt: TODO.+ symbol_table: TODO.+ lexicon_table: TODO.+ """+ tokens_str = tuple()+ tokens_idx = tuple()++ if txt in symbol_table:+ tokens_str = tokens_str + (txt, )+ tokens_idx = tokens_idx + (symbol_table[txt], )+ return tokens_str, tokens_idx++ parts = split_mixed_label(txt)+ for part in parts:+ if part == '!sil' or part == '(sil)' or part == '<sil>':+ tokens_str = tokens_str + ('!sil', )+ elif part == '<blank>' or part == '<blank>':+ tokens_str = tokens_str + ('<blank>', )+ elif part == '(noise)' or part == 'noise)' or part == '(noise' or part == '<noise>':+ tokens_str = tokens_str + ('<unk>', )+ elif part in symbol_table:+ tokens_str = tokens_str + (part, )+ elif part in lexicon_table:+ for ch in lexicon_table[part]:+ tokens_str = tokens_str + (ch, )+ else:+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/utils/kws_utils.py#L32" target="_blank" class="src-more">View full source on GitHub →</a></details></div>+<div class="api-detail" id="e374"><span class="dtag badge-c">class</span><h2>KwsCtcPrefixDecoder</h2><div class="dmod">funasr.utils.kws_utils · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/utils/kws_utils.py#L90" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Decoder interface wrapper for CTCPrefixDecode.</p>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code>class KwsCtcPrefixDecoder():+ """Decoder interface wrapper for CTCPrefixDecode."""++ def __init__(+ self,+ ctc: torch.nn.Module,+ keywords: str,+ token_list: list,+ seg_dict: dict,+ ):+ """Initialize class.++ Args:+ ctc (torch.nn.Module): The CTC implementation.+ For example, :class:`espnet.nets.pytorch_backend.ctc.CTC`++ """+ self.ctc = ctc+ self.token_list = token_list++ token_table = {}+ for token in token_list:+ token_table[token] = token_list.index(token)++ self.keywords_idxset = {0}+ self.keywords_token = {}+ self.keywords_str = keywords+ keywords_list = self.keywords_str.strip().replace(' ', '').split(',')+ for keyword in keywords_list:+ strs, indexs = query_token_set(keyword, token_table, seg_dict)+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/utils/kws_utils.py#L90" target="_blank" class="src-more">View full source on GitHub →</a></details><h3 class="mt">Methods</h3><div class="mblk"><code>.beam_search(logits, logits_lengths, keywords_tokenset, score_beam_size, path_beam_size)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/utils/kws_utils.py#L125" target="_blank" class="gh-link">L125</a><div class="mdoc"><p>CTC prefix beam search inner implementation</p>+<h4>Args:</h4>+<ul>+<li><code>logits (torch.Tensor)</code> — (1, max_len, vocab_size)</li>+<li><code>logits_lengths (torch.Tensor)</code> — (1, )</li>+<li><code>keywords_tokenset (set)</code> — token set for filtering score</li>+<li><code>score_beam_size (int)</code> — beam size for score</li>+<li><code>path_beam_size (int)</code> — beam size for path</li>+</ul>+<h4>Returns:</h4>+<p>List[List[int]]: nbest results</p>+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def beam_search(+ self,+ logits: torch.Tensor,+ logits_lengths: torch.Tensor,+ keywords_tokenset: set = None,+ score_beam_size: int = 3,+ path_beam_size: int = 20,+ ) -> Tuple[List[List[int]], torch.Tensor]:+ """ CTC prefix beam search inner implementation++ Args:+ logits (torch.Tensor): (1, max_len, vocab_size)+ logits_lengths (torch.Tensor): (1, )+ keywords_tokenset (set): token set for filtering score+ score_beam_size (int): beam size for score+ path_beam_size (int): beam size for path++ Returns:+ List[List[int]]: nbest results+ """++ maxlen = logits.size(0)+ ctc_probs = logits+ cur_hyps = [(tuple(), (1.0, 0.0, []))]++ # CTC beam search step by step+ for t in range(0, maxlen):+ probs = ctc_probs[t] # (vocab_size,)+ # key: prefix, value (pb, pnb), default value(-inf, -inf)+ next_hyps = defaultdict(lambda: (0.0, 0.0, []))+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/utils/kws_utils.py#L125" target="_blank" class="src-more">View full source →</a></details></div><div class="mblk"><code>.is_sublist(main_list, check_list)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/utils/kws_utils.py#L232" target="_blank" class="gh-link">L232</a><div class="mdoc"><p>Is sublist.</p>+<h4>Args:</h4>+<ul>+<li><code>main_list</code> — TODO.</li>+<li><code>check_list</code> — TODO.</li>+</ul>+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def is_sublist(self, main_list, check_list):+ """Is sublist.++ Args:+ main_list: TODO.+ check_list: TODO.+ """+ if len(main_list) < len(check_list):+ return -1++ if len(main_list) == len(check_list):+ return 0 if main_list == check_list else -1++ for i in range(len(main_list) - len(check_list) + 1):+ if main_list[i] == check_list[0]:+ for j in range(len(check_list)):+ if main_list[i + j] != check_list[j]:+ break+ else:+ return i+ else:+ return -1+</code></pre></details></div><div class="mblk"><code>.decode(x)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/utils/kws_utils.py#L295" target="_blank" class="gh-link">L295</a><div class="mdoc"><p>Get an initial state for decoding.</p>+<h4>Args:</h4>+<ul>+<li><code>x (torch.Tensor)</code> — The encoded feature tensor</li>+<li><code>Returns</code> — decode result</li>+</ul>+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def decode(self, x: torch.Tensor):+ """Get an initial state for decoding.++ Args:+ x (torch.Tensor): The encoded feature tensor++ Returns: decode result++ """++ raw_logp = self.ctc.softmax(x.unsqueeze(0)).detach().squeeze(0).cpu()+ xlen = torch.tensor([raw_logp.size(1)])++ return self._decode_inside(raw_logp, xlen)+</code></pre></details></div></div>+<div class="api-detail" id="e375"><span class="dtag badge-m">method</span><h2><code>KwsCtcPrefixDecoder.beam_search(logits, logits_lengths, keywords_tokenset, score_beam_size, path_beam_size)</code></h2><div class="dmod">funasr.utils.kws_utils.KwsCtcPrefixDecoder · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/utils/kws_utils.py#L125" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>CTC prefix beam search inner implementation</p>+<h4>Args:</h4>+<ul>+<li><code>logits (torch.Tensor)</code> — (1, max_len, vocab_size)</li>+<li><code>logits_lengths (torch.Tensor)</code> — (1, )</li>+<li><code>keywords_tokenset (set)</code> — token set for filtering score</li>+<li><code>score_beam_size (int)</code> — beam size for score</li>+<li><code>path_beam_size (int)</code> — beam size for path</li>+</ul>+<h4>Returns:</h4>+<p>List[List[int]]: nbest results</p>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def beam_search(+ self,+ logits: torch.Tensor,+ logits_lengths: torch.Tensor,+ keywords_tokenset: set = None,+ score_beam_size: int = 3,+ path_beam_size: int = 20,+ ) -> Tuple[List[List[int]], torch.Tensor]:+ """ CTC prefix beam search inner implementation++ Args:+ logits (torch.Tensor): (1, max_len, vocab_size)+ logits_lengths (torch.Tensor): (1, )+ keywords_tokenset (set): token set for filtering score+ score_beam_size (int): beam size for score+ path_beam_size (int): beam size for path++ Returns:+ List[List[int]]: nbest results+ """++ maxlen = logits.size(0)+ ctc_probs = logits+ cur_hyps = [(tuple(), (1.0, 0.0, []))]++ # CTC beam search step by step+ for t in range(0, maxlen):+ probs = ctc_probs[t] # (vocab_size,)+ # key: prefix, value (pb, pnb), default value(-inf, -inf)+ next_hyps = defaultdict(lambda: (0.0, 0.0, []))+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/utils/kws_utils.py#L125" target="_blank" class="src-more">View full source on GitHub →</a></details></div>+<div class="api-detail" id="e376"><span class="dtag badge-m">method</span><h2><code>KwsCtcPrefixDecoder.is_sublist(main_list, check_list)</code></h2><div class="dmod">funasr.utils.kws_utils.KwsCtcPrefixDecoder · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/utils/kws_utils.py#L232" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Is sublist.</p>+<h4>Args:</h4>+<ul>+<li><code>main_list</code> — TODO.</li>+<li><code>check_list</code> — TODO.</li>+</ul>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def is_sublist(self, main_list, check_list):+ """Is sublist.++ Args:+ main_list: TODO.+ check_list: TODO.+ """+ if len(main_list) < len(check_list):+ return -1++ if len(main_list) == len(check_list):+ return 0 if main_list == check_list else -1++ for i in range(len(main_list) - len(check_list) + 1):+ if main_list[i] == check_list[0]:+ for j in range(len(check_list)):+ if main_list[i + j] != check_list[j]:+ break+ else:+ return i+ else:+ return -1+</code></pre></details></div>+<div class="api-detail" id="e377"><span class="dtag badge-m">method</span><h2><code>KwsCtcPrefixDecoder.decode(x)</code></h2><div class="dmod">funasr.utils.kws_utils.KwsCtcPrefixDecoder · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/utils/kws_utils.py#L295" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Get an initial state for decoding.</p>+<h4>Args:</h4>+<ul>+<li><code>x (torch.Tensor)</code> — The encoded feature tensor</li>+<li><code>Returns</code> — decode result</li>+</ul>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def decode(self, x: torch.Tensor):+ """Get an initial state for decoding.++ Args:+ x (torch.Tensor): The encoded feature tensor++ Returns: decode result++ """++ raw_logp = self.ctc.softmax(x.unsqueeze(0)).detach().squeeze(0).cpu()+ xlen = torch.tensor([raw_logp.size(1)])++ return self._decode_inside(raw_logp, xlen)+</code></pre></details></div>+<div class="api-detail" id="e378"><span class="dtag badge-f">function</span><h2><code>is_ffmpeg_installed()</code></h2><div class="dmod">funasr.utils.load_utils · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/utils/load_utils.py#L27" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Is ffmpeg installed.</p>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code>def is_ffmpeg_installed():+ """Is ffmpeg installed."""+ try:+ output = subprocess.check_output(["ffmpeg", "-version"], stderr=subprocess.STDOUT)+ return "ffmpeg version" in output.decode("utf-8")+ except (subprocess.CalledProcessError, FileNotFoundError):+ return False+</code></pre></details></div>+<div class="api-detail" id="e379"><span class="dtag badge-f">function</span><h2><code>load_audio_text_image_video(data_or_path_or_list, fs, audio_fs, data_type, tokenizer, **kwargs)</code></h2><div class="dmod">funasr.utils.load_utils · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/utils/load_utils.py#L48" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Load audio/text/image/video data from various input formats.</p>+<h4>Args:</h4>+<ul>+<li><code>data_or_path_or_list</code> — File path, URL, numpy array, torch Tensor, bytes, or list.</li>+<li><code>fs (int)</code> — Target sample rate (default 16000).</li>+<li><code>audio_fs (int)</code> — Source audio sample rate.</li>+<li><code>data_type (str)</code> — Input type ("sound", "text", "fbank").</li>+</ul>+<h4>Returns:</h4>+<p>torch.Tensor or list: Loaded and resampled audio tensor(s).</p>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code>def load_audio_text_image_video(+ data_or_path_or_list,+ fs: int = 16000,+ audio_fs: int = 16000,+ data_type="sound",+ tokenizer=None,+ **kwargs,+):+ """Load audio/text/image/video data from various input formats.++ Args:+ data_or_path_or_list: File path, URL, numpy array, torch Tensor, bytes, or list.+ fs (int): Target sample rate (default 16000).+ audio_fs (int): Source audio sample rate.+ data_type (str): Input type ("sound", "text", "fbank").++ Returns:+ torch.Tensor or list: Loaded and resampled audio tensor(s).+ """+ if isinstance(data_or_path_or_list, (list, tuple)):+ if data_type is not None and isinstance(data_type, (list, tuple)):+ data_types = [data_type] * len(data_or_path_or_list)+ data_or_path_or_list_ret = [[] for d in data_type]+ for i, (data_type_i, data_or_path_or_list_i) in enumerate(+ zip(data_types, data_or_path_or_list)+ ):+ for j, (data_type_j, data_or_path_or_list_j) in enumerate(+ zip(data_type_i, data_or_path_or_list_i)+ ):+ data_or_path_or_list_j = load_audio_text_image_video(+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/utils/load_utils.py#L48" target="_blank" class="src-more">View full source on GitHub →</a></details></div>+<div class="api-detail" id="e380"><span class="dtag badge-f">function</span><h2><code>load_bytes(input)</code></h2><div class="dmod">funasr.utils.load_utils · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/utils/load_utils.py#L187" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Convert audio bytes to numpy array.</p>+<h4>Args:</h4>+<ul>+<li><code>input (bytes)</code> — Raw audio bytes.</li>+</ul>+<h4>Returns:</h4>+<ul>+<li><code>numpy.ndarray</code> — Decoded audio samples.</li>+</ul>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code>def load_bytes(input):+ """Convert audio bytes to numpy array.++ Args:+ input (bytes): Raw audio bytes.++ Returns:+ numpy.ndarray: Decoded audio samples.+ """+ # Only run the (expensive) frame-rate validation when the payload is an+ # actual audio container (WAV, MP3, OGG, …). Raw PCM buffers have no+ # recognisable header and would cause pydub to spend ~200 ms before+ # raising an exception that is then silently swallowed anyway.+ if _is_audio_container(input):+ try:+ input = validate_frame_rate(input)+ except Exception:+ pass+ middle_data = np.frombuffer(input, dtype=np.int16)+ middle_data = np.asarray(middle_data)+ if middle_data.dtype.kind not in "iu":+ raise TypeError("'middle_data' must be an array of integers")+ dtype = np.dtype("float32")+ if dtype.kind != "f":+ raise TypeError("'dtype' must be a floating point type")++ i = np.iinfo(middle_data.dtype)+ abs_max = 2 ** (i.bits - 1)+ offset = i.min + abs_max+ array = np.frombuffer((middle_data.astype(dtype) - offset) / abs_max, dtype=np.float32)+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/utils/load_utils.py#L187" target="_blank" class="src-more">View full source on GitHub →</a></details></div>+<div class="api-detail" id="e381"><span class="dtag badge-f">function</span><h2><code>validate_frame_rate(input, fs)</code></h2><div class="dmod">funasr.utils.load_utils · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/utils/load_utils.py#L220" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Validate frame rate.</p>+<h4>Args:</h4>+<ul>+<li><code>input</code> — Input audio/text data.</li>+<li><code>fs</code> — TODO.</li>+</ul>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code>def validate_frame_rate(+ input,+ fs: int = 16000,+):++ # 将文件读取为字节流+ """Validate frame rate.++ Args:+ input: Input audio/text data.+ fs: TODO.+ """+ byte_data = BytesIO(input)++ # 使用 pydub 加载音频+ try:+ audio = AudioSegment.from_file(byte_data)+ except:+ raise RuntimeError(+ "You are decoding the pcm data, please install pydub first. via `pip install pydub`."+ )++ # 确保采样率为 16000 Hz+ if audio.frame_rate != fs:+ audio = audio.set_frame_rate(fs)++ # 将重新采样后的音频导出为字节流+ output = BytesIO()+ audio.export(output, format="wav")+ output.seek(0)+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/utils/load_utils.py#L220" target="_blank" class="src-more">View full source on GitHub →</a></details></div>+<div class="api-detail" id="e382"><span class="dtag badge-f">function</span><h2><code>extract_fbank(data, data_len, data_type, frontend, **kwargs)</code></h2><div class="dmod">funasr.utils.load_utils · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/utils/load_utils.py#L257" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Extract filter-bank features from audio data.</p>+<h4>Args:</h4>+<ul>+<li><code>data</code> — Audio samples (list of numpy arrays or tensors).</li>+<li><code>data_len</code> — Lengths of each sample.</li>+<li><code>data_type (str)</code> — Input type ("sound", "fbank").</li>+<li><code>frontend</code> — Frontend instance for feature extraction.</li>+</ul>+<h4>Returns:</h4>+<ul>+<li><code>tuple</code> — (features_tensor, feature_lengths, feature_times)</li>+</ul>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code>def extract_fbank(data, data_len=None, data_type: str = "sound", frontend=None, **kwargs):+ """Extract filter-bank features from audio data.++ Args:+ data: Audio samples (list of numpy arrays or tensors).+ data_len: Lengths of each sample.+ data_type (str): Input type ("sound", "fbank").+ frontend: Frontend instance for feature extraction.++ Returns:+ tuple: (features_tensor, feature_lengths, feature_times)+ """+ if isinstance(data, np.ndarray):+ data = torch.from_numpy(data)+ if len(data.shape) < 2:+ data = data[None, :] # data: [batch, N]+ elif data.shape[0] > 1:+ data = data.mean(dim=0, keepdim=True) # convert stereo/multi-channel to mono+ data_len = [data.shape[1]] if data_len is None else data_len+ elif isinstance(data, torch.Tensor):+ if len(data.shape) < 2:+ data = data[None, :] # data: [batch, N]+ elif data.shape[0] > 1:+ data = data.mean(dim=0, keepdim=True) # convert stereo/multi-channel to mono+ data_len = [data.shape[1]] if data_len is None else data_len+ elif isinstance(data, (list, tuple)):+ data_list, data_len = [], []+ for data_i in data:+ if isinstance(data_i, np.ndarray):+ data_i = torch.from_numpy(data_i)+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/utils/load_utils.py#L257" target="_blank" class="src-more">View full source on GitHub →</a></details></div>+<div class="api-detail" id="e383"><span class="dtag badge-f">function</span><h2><code>statistic_model_parameters(model, prefix)</code></h2><div class="dmod">funasr.utils.misc · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/utils/misc.py#L10" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Statistic model parameters.</p>+<h4>Args:</h4>+<ul>+<li><code>model</code> — Model instance or model name.</li>+<li><code>prefix</code> — TODO.</li>+</ul>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code>def statistic_model_parameters(model, prefix=None):+ """Statistic model parameters.++ Args:+ model: Model instance or model name.+ prefix: TODO.+ """+ var_dict = model.state_dict()+ numel = 0+ for i, key in enumerate(+ sorted(list([x for x in var_dict.keys() if "num_batches_tracked" not in x]))+ ):+ if prefix is None or key.startswith(prefix):+ numel += var_dict[key].numel()+ return numel+</code></pre></details></div>+<div class="api-detail" id="e384"><span class="dtag badge-f">function</span><h2><code>int2vec(x, vec_dim, dtype)</code></h2><div class="dmod">funasr.utils.misc · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/utils/misc.py#L27" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Int2vec.</p>+<h4>Args:</h4>+<ul>+<li><code>x</code> — TODO.</li>+<li><code>vec_dim</code> — Size/dimension parameter.</li>+<li><code>dtype</code> — TODO.</li>+</ul>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code>def int2vec(x, vec_dim=8, dtype=np.int32):+ """Int2vec.++ Args:+ x: TODO.+ vec_dim: Size/dimension parameter.+ dtype: TODO.+ """+ b = ("{:0" + str(vec_dim) + "b}").format(x)+ # little-endian order: lower bit first+ return (np.array(list(b)[::-1]) == "1").astype(dtype)+</code></pre></details></div>+<div class="api-detail" id="e385"><span class="dtag badge-f">function</span><h2><code>seq2arr(seq, vec_dim)</code></h2><div class="dmod">funasr.utils.misc · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/utils/misc.py#L40" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Seq2arr.</p>+<h4>Args:</h4>+<ul>+<li><code>seq</code> — TODO.</li>+<li><code>vec_dim</code> — Size/dimension parameter.</li>+</ul>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code>def seq2arr(seq, vec_dim=8):+ """Seq2arr.++ Args:+ seq: TODO.+ vec_dim: Size/dimension parameter.+ """+ return np.row_stack([int2vec(int(x), vec_dim) for x in seq])+</code></pre></details></div>+<div class="api-detail" id="e386"><span class="dtag badge-f">function</span><h2><code>load_scp_as_dict(scp_path, value_type, kv_sep)</code></h2><div class="dmod">funasr.utils.misc · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/utils/misc.py#L50" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Load scp as dict.</p>+<h4>Args:</h4>+<ul>+<li><code>scp_path</code> — TODO.</li>+<li><code>value_type</code> — TODO.</li>+<li><code>kv_sep</code> — TODO.</li>+</ul>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code>def load_scp_as_dict(scp_path, value_type="str", kv_sep=" "):+ """Load scp as dict.++ Args:+ scp_path: TODO.+ value_type: TODO.+ kv_sep: TODO.+ """+ with io.open(scp_path, "r", encoding="utf-8") as f:+ ret_dict = OrderedDict()+ for one_line in f.readlines():+ one_line = one_line.strip()+ pos = one_line.find(kv_sep)+ key, value = one_line[:pos], one_line[pos + 1 :]+ if value_type == "list":+ value = value.split(" ")+ ret_dict[key] = value+ return ret_dict+</code></pre></details></div>+<div class="api-detail" id="e387"><span class="dtag badge-f">function</span><h2><code>load_scp_as_list(scp_path, value_type, kv_sep)</code></h2><div class="dmod">funasr.utils.misc · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/utils/misc.py#L70" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Load scp as list.</p>+<h4>Args:</h4>+<ul>+<li><code>scp_path</code> — TODO.</li>+<li><code>value_type</code> — TODO.</li>+<li><code>kv_sep</code> — TODO.</li>+</ul>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code>def load_scp_as_list(scp_path, value_type="str", kv_sep=" "):+ """Load scp as list.++ Args:+ scp_path: TODO.+ value_type: TODO.+ kv_sep: TODO.+ """+ with io.open(scp_path, "r", encoding="utf8") as f:+ ret_dict = []+ for one_line in f.readlines():+ one_line = one_line.strip()+ pos = one_line.find(kv_sep)+ key, value = one_line[:pos], one_line[pos + 1 :]+ if value_type == "list":+ value = value.split(" ")+ ret_dict.append((key, value))+ return ret_dict+</code></pre></details></div>+<div class="api-detail" id="e388"><span class="dtag badge-f">function</span><h2><code>deep_update(original, update)</code></h2><div class="dmod">funasr.utils.misc · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/utils/misc.py#L90" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Recursively merge update dict into original dict (in-place).</p>+<p>For nested dicts, merges recursively. For other types, overwrites.</p>+<h4>Args:</h4>+<ul>+<li><code>original (dict)</code> — Target dict to be updated in-place.</li>+<li><code>update (dict)</code> — Source dict with new values.</li>+</ul>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code>def deep_update(original, update):++ """Recursively merge update dict into original dict (in-place).++ For nested dicts, merges recursively. For other types, overwrites.++ Args:+ original (dict): Target dict to be updated in-place.+ update (dict): Source dict with new values.+ """+ for key, value in update.items():+ if isinstance(value, dict) and key in original:+ if len(value) == 0:+ original[key] = value+ deep_update(original[key], value)+ else:+ original[key] = value+</code></pre></details></div>+<div class="api-detail" id="e389"><span class="dtag badge-f">function</span><h2><code>prepare_model_dir(**kwargs)</code></h2><div class="dmod">funasr.utils.misc · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/utils/misc.py#L109" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Prepare model dir.</p>+<h4>Args:</h4>+<ul>+<li><code>**kwargs</code> — Additional keyword arguments.</li>+</ul>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code>def prepare_model_dir(**kwargs):++ """Prepare model dir.++ Args:+ **kwargs: Additional keyword arguments.+ """+ os.makedirs(kwargs.get("output_dir", "./"), exist_ok=True)++ yaml_file = os.path.join(kwargs.get("output_dir", "./"), "config.yaml")+ OmegaConf.save(config=kwargs, f=yaml_file)+ logging.info(f"kwargs: {kwargs}")+ logging.info("config.yaml is saved to: %s", yaml_file)++ model_path = kwargs.get("model_path", None)+ if model_path is not None:+ config_json = os.path.join(model_path, "configuration.json")+ if os.path.exists(config_json):+ shutil.copy(+ config_json, os.path.join(kwargs.get("output_dir", "./"), "configuration.json")+ )+</code></pre></details></div>+<div class="api-detail" id="e390"><span class="dtag badge-f">function</span><h2><code>extract_filename_without_extension(file_path)</code></h2><div class="dmod">funasr.utils.misc · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/utils/misc.py#L132" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>从给定的文件路径中提取文件名(不包含路径和扩展名)</p>+<p>:param file_path: 完整的文件路径</p>+<p>:return: 文件名(不含路径和扩展名)</p>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code>def extract_filename_without_extension(file_path):+ """+ 从给定的文件路径中提取文件名(不包含路径和扩展名)+ :param file_path: 完整的文件路径+ :return: 文件名(不含路径和扩展名)+ """+ # 首先,使用os.path.basename获取路径中的文件名部分(含扩展名)+ filename_with_extension = os.path.basename(file_path)+ # 然后,使用os.path.splitext分离文件名和扩展名+ filename, extension = os.path.splitext(filename_with_extension)+ # 返回不包含扩展名的文件名+ return filename+</code></pre></details></div>+<div class="api-detail" id="e391"><span class="dtag badge-f">function</span><h2><code>smart_remove(path)</code></h2><div class="dmod">funasr.utils.misc · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/utils/misc.py#L146" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Intelligently removes files, empty directories, and non-empty directories recursively.</p>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code>def smart_remove(path):+ """Intelligently removes files, empty directories, and non-empty directories recursively."""+ # Check if the provided path exists+ if not os.path.exists(path):+ print(f"{path} does not exist.")+ return++ # If the path is a file, delete it+ if os.path.isfile(path):+ os.remove(path)+ print(f"File {path} has been deleted.")+ # If the path is a directory+ elif os.path.isdir(path):+ try:+ # Attempt to remove an empty directory+ os.rmdir(path)+ print(f"Empty directory {path} has been deleted.")+ except OSError:+ # If the directory is not empty, remove it along with all its contents+ shutil.rmtree(path)+ print(f"Non-empty directory {path} has been recursively deleted.")+</code></pre></details></div>+<div class="api-detail" id="e392"><span class="dtag badge-f">function</span><h2><code>isChinese(ch)</code></h2><div class="dmod">funasr.utils.postprocess_utils · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/utils/postprocess_utils.py#L8" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Ischinese.</p>+<h4>Args:</h4>+<ul>+<li><code>ch</code> — TODO.</li>+</ul>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code>def isChinese(ch: str):+ """Ischinese.++ Args:+ ch: TODO.+ """+ if "\u4e00" <= ch <= "\u9fff" or "\u0030" <= ch <= "\u0039" or ch == "@":+ return True+ return False+</code></pre></details></div>+<div class="api-detail" id="e393"><span class="dtag badge-f">function</span><h2><code>isAllChinese(word)</code></h2><div class="dmod">funasr.utils.postprocess_utils · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/utils/postprocess_utils.py#L19" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Isallchinese.</p>+<h4>Args:</h4>+<ul>+<li><code>word</code> — TODO.</li>+</ul>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code>def isAllChinese(word: Union[List[Any], str]):+ """Isallchinese.++ Args:+ word: TODO.+ """+ word_lists = []+ for i in word:+ cur = i.replace(" ", "")+ cur = cur.replace("</s>", "")+ cur = cur.replace("<s>", "")+ cur = cur.replace("<unk>", "")+ cur = cur.replace("<OOV>", "")+ word_lists.append(cur)++ if len(word_lists) == 0:+ return False++ for ch in word_lists:+ if isChinese(ch) is False:+ return False+ return True+</code></pre></details></div>+<div class="api-detail" id="e394"><span class="dtag badge-f">function</span><h2><code>isAllAlpha(word)</code></h2><div class="dmod">funasr.utils.postprocess_utils · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/utils/postprocess_utils.py#L43" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Isallalpha.</p>+<h4>Args:</h4>+<ul>+<li><code>word</code> — TODO.</li>+</ul>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code>def isAllAlpha(word: Union[List[Any], str]):+ """Isallalpha.++ Args:+ word: TODO.+ """+ word_lists = []+ for i in word:+ cur = i.replace(" ", "")+ cur = cur.replace("</s>", "")+ cur = cur.replace("<s>", "")+ cur = cur.replace("<unk>", "")+ cur = cur.replace("<OOV>", "")+ word_lists.append(cur)++ if len(word_lists) == 0:+ return False++ for ch in word_lists:+ if ch.isalpha() is False and ch != "'":+ return False+ elif ch.isalpha() is True and isChinese(ch) is True:+ return False++ return True+</code></pre></details></div>+<div class="api-detail" id="e395"><span class="dtag badge-f">function</span><h2><code>abbr_dispose(words, time_stamp)</code></h2><div class="dmod">funasr.utils.postprocess_utils · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/utils/postprocess_utils.py#L71" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Abbr dispose.</p>+<h4>Args:</h4>+<ul>+<li><code>words</code> — TODO.</li>+<li><code>time_stamp</code> — TODO.</li>+</ul>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code>def abbr_dispose(words: List[Any], time_stamp: List[List] = None) -> List[Any]:+ """Abbr dispose.++ Args:+ words: TODO.+ time_stamp: TODO.+ """+ words_size = len(words)+ word_lists = []+ abbr_begin = []+ abbr_end = []+ last_num = -1+ ts_lists = []+ ts_nums = []+ ts_index = 0+ for num in range(words_size):+ if num <= last_num:+ continue++ if len(words[num]) == 1 and words[num].encode("utf-8").isalpha():+ if (+ num + 1 < words_size+ and words[num + 1] == " "+ and num + 2 < words_size+ and len(words[num + 2]) == 1+ and words[num + 2].encode("utf-8").isalpha()+ ):+ # found the begin of abbr+ abbr_begin.append(num)+ num += 2+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/utils/postprocess_utils.py#L71" target="_blank" class="src-more">View full source on GitHub →</a></details></div>+<div class="api-detail" id="e396"><span class="dtag badge-f">function</span><h2><code>sentence_postprocess(words, time_stamp)</code></h2><div class="dmod">funasr.utils.postprocess_utils · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/utils/postprocess_utils.py#L165" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Sentence postprocess.</p>+<h4>Args:</h4>+<ul>+<li><code>words</code> — TODO.</li>+<li><code>time_stamp</code> — TODO.</li>+</ul>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code>def sentence_postprocess(words: List[Any], time_stamp: List[List] = None):+ """Sentence postprocess.++ Args:+ words: TODO.+ time_stamp: TODO.+ """+ middle_lists = []+ word_lists = []+ word_item = ""+ ts_lists = []++ # wash words lists+ for i in words:+ word = ""+ if isinstance(i, str):+ word = i+ else:+ word = i.decode("utf-8")++ if word in ["<s>", "</s>", "<unk>", "<OOV>"]:+ continue+ else:+ middle_lists.append(word)++ # all chinese characters+ if isAllChinese(middle_lists):+ for i, ch in enumerate(middle_lists):+ word_lists.append(ch.replace(" ", ""))+ if time_stamp is not None:+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/utils/postprocess_utils.py#L165" target="_blank" class="src-more">View full source on GitHub →</a></details></div>+<div class="api-detail" id="e397"><span class="dtag badge-f">function</span><h2><code>sentence_postprocess_sentencepiece(words)</code></h2><div class="dmod">funasr.utils.postprocess_utils · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/utils/postprocess_utils.py#L281" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Sentence postprocess sentencepiece.</p>+<h4>Args:</h4>+<ul>+<li><code>words</code> — TODO.</li>+</ul>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code>def sentence_postprocess_sentencepiece(words):+ """Sentence postprocess sentencepiece.++ Args:+ words: TODO.+ """+ middle_lists = []+ word_lists = []+ word_item = ""++ # wash words lists+ for i in words:+ word = ""+ if isinstance(i, str):+ word = i+ else:+ word = i.decode("utf-8")++ if word in ["<s>", "</s>", "<unk>", "<OOV>"]:+ continue+ else:+ middle_lists.append(word)++ # all alpha characters+ for i, ch in enumerate(middle_lists):+ word = ""+ if "\u2581" in ch and i == 0:+ word_item = ""+ word = ch.replace("\u2581", "")+ word_item += word+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/utils/postprocess_utils.py#L281" target="_blank" class="src-more">View full source on GitHub →</a></details></div>+<div class="api-detail" id="e398"><span class="dtag badge-f">function</span><h2><code>format_str_v2(s)</code></h2><div class="dmod">funasr.utils.postprocess_utils · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/utils/postprocess_utils.py#L411" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Format str v2.</p>+<h4>Args:</h4>+<ul>+<li><code>s</code> — TODO.</li>+</ul>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code>def format_str_v2(s):+ """Format str v2.++ Args:+ s: TODO.+ """+ sptk_dict = {}+ for sptk in emoji_dict:+ sptk_dict[sptk] = s.count(sptk)+ s = s.replace(sptk, "")+ emo = "<|NEUTRAL|>"+ for e in emo_dict:+ if sptk_dict[e] > sptk_dict[emo]:+ emo = e+ for e in event_dict:+ if sptk_dict[e] > 0:+ s = event_dict[e] + s+ s = s + emo_dict[emo]++ for emoji in emo_set.union(event_set):+ s = s.replace(" " + emoji, emoji)+ s = s.replace(emoji + " ", emoji)+ return s.strip()+</code></pre></details></div>+<div class="api-detail" id="e399"><span class="dtag badge-f">function</span><h2><code>rich_transcription_postprocess(s)</code></h2><div class="dmod">funasr.utils.postprocess_utils · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/utils/postprocess_utils.py#L436" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Rich transcription postprocess.</p>+<h4>Args:</h4>+<ul>+<li><code>s</code> — TODO.</li>+</ul>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code>def rich_transcription_postprocess(s):+ """Rich transcription postprocess.++ Args:+ s: TODO.+ """+ def get_emo(s):+ """Get emo.++ Args:+ s: TODO.+ """+ return s[-1] if s[-1] in emo_set else None++ def get_event(s):+ """Get event.++ Args:+ s: TODO.+ """+ return s[0] if s[0] in event_set else None++ s = s.replace("<|nospeech|><|Event_UNK|>", "❓")+ for lang in lang_dict:+ s = s.replace(lang, "<|lang|>")+ s_list = [format_str_v2(s_i).strip(" ") for s_i in s.split("<|lang|>")]+ new_s = " " + s_list[0]+ cur_ent_event = get_event(new_s)+ for i in range(1, len(s_list)):+ if len(s_list[i]) == 0:+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/utils/postprocess_utils.py#L436" target="_blank" class="src-more">View full source on GitHub →</a></details></div>+<div class="api-detail" id="e400"><span class="dtag badge-f">function</span><h2><code>check_audio_list(audio)</code></h2><div class="dmod">funasr.utils.speaker_utils · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/utils/speaker_utils.py#L18" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Check audio list.</p>+<h4>Args:</h4>+<ul>+<li><code>audio</code> — TODO.</li>+</ul>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code>def check_audio_list(audio: list):+ """Check audio list.++ Args:+ audio: TODO.+ """+ audio_dur = 0+ for i in range(len(audio)):+ seg = audio[i]+ assert seg[1] >= seg[0], "modelscope error: Wrong time stamps."+ assert isinstance(seg[2], np.ndarray), "modelscope error: Wrong data type."+ assert (+ int(seg[1] * 16000) - int(seg[0] * 16000) == seg[2].shape[0]+ ), "modelscope error: audio data in list is inconsistent with time length."+ if i > 0:+ assert seg[0] >= audio[i - 1][1], "modelscope error: Wrong time stamps."+ audio_dur += seg[1] - seg[0]+ return audio_dur+</code></pre></details></div>+<div class="api-detail" id="e401"><span class="dtag badge-f">function</span><h2><code>sv_preprocess(inputs)</code></h2><div class="dmod">funasr.utils.speaker_utils · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/utils/speaker_utils.py#L39" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Sv preprocess.</p>+<h4>Args:</h4>+<ul>+<li><code>inputs</code> — TODO.</li>+</ul>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code>def sv_preprocess(inputs: Union[np.ndarray, list]):+ """Sv preprocess.++ Args:+ inputs: TODO.+ """+ output = []+ for i in range(len(inputs)):+ if isinstance(inputs[i], str):+ file_bytes = File.read(inputs[i])+ data, fs = sf.load(io.BytesIO(file_bytes), dtype="float32")+ if len(data.shape) == 2:+ data = data[:, 0]+ data = torch.from_numpy(data).unsqueeze(0)+ data = data.squeeze(0)+ elif isinstance(inputs[i], np.ndarray):+ assert len(inputs[i].shape) == 1, "modelscope error: Input array should be [N, T]"+ data = inputs[i]+ if data.dtype in ["int16", "int32", "int64"]:+ data = (data / (1 << 15)).astype("float32")+ else:+ data = data.astype("float32")+ data = torch.from_numpy(data)+ else:+ raise ValueError(+ "modelscope error: The input type is restricted to audio address and nump array."+ )+ output.append(data)+ return output+</code></pre></details></div>+<div class="api-detail" id="e402"><span class="dtag badge-f">function</span><h2><code>sv_chunk(vad_segments, fs)</code></h2><div class="dmod">funasr.utils.speaker_utils · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/utils/speaker_utils.py#L70" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Sv chunk.</p>+<h4>Args:</h4>+<ul>+<li><code>vad_segments</code> — TODO.</li>+<li><code>fs</code> — TODO.</li>+</ul>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code>def sv_chunk(vad_segments: list, fs=16000) -> list:+ """Sv chunk.++ Args:+ vad_segments: TODO.+ fs: TODO.+ """+ config = {+ "seg_dur": 1.5,+ "seg_shift": 0.75,+ }++ def seg_chunk(seg_data):+ """Seg chunk.++ Args:+ seg_data: TODO.+ """+ seg_st = seg_data[0]+ data = seg_data[2]+ chunk_len = int(config["seg_dur"] * fs)+ chunk_shift = int(config["seg_shift"] * fs)+ last_chunk_ed = 0+ seg_res = []+ for chunk_st in range(0, data.shape[0], chunk_shift):+ chunk_ed = min(chunk_st + chunk_len, data.shape[0])+ if chunk_ed <= last_chunk_ed:+ break+ last_chunk_ed = chunk_ed+ chunk_st = max(0, chunk_ed - chunk_len)+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/utils/speaker_utils.py#L70" target="_blank" class="src-more">View full source on GitHub →</a></details></div>+<div class="api-detail" id="e403"><span class="dtag badge-f">function</span><h2><code>extract_feature(audio)</code></h2><div class="dmod">funasr.utils.speaker_utils · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/utils/speaker_utils.py#L113" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Extract feature.</p>+<h4>Args:</h4>+<ul>+<li><code>audio</code> — TODO.</li>+</ul>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code>def extract_feature(audio):+ """Extract feature.++ Args:+ audio: TODO.+ """+ features = []+ for au in audio:+ feature = Kaldi.fbank(au.unsqueeze(0), num_mel_bins=80)+ feature = feature - feature.mean(dim=0, keepdim=True)+ features.append(feature.unsqueeze(0))+ features = torch.cat(features)+ return features+</code></pre></details></div>+<div class="api-detail" id="e404"><span class="dtag badge-f">function</span><h2><code>postprocess(segments, vad_segments, labels, embeddings)</code></h2><div class="dmod">funasr.utils.speaker_utils · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/utils/speaker_utils.py#L128" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Postprocess.</p>+<h4>Args:</h4>+<ul>+<li><code>segments</code> — TODO.</li>+<li><code>vad_segments</code> — TODO.</li>+<li><code>labels</code> — TODO.</li>+<li><code>embeddings</code> — TODO.</li>+</ul>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code>def postprocess(+ segments: list, vad_segments: list, labels: np.ndarray, embeddings: np.ndarray+) -> list:+ """Postprocess.++ Args:+ segments: TODO.+ vad_segments: TODO.+ labels: TODO.+ embeddings: TODO.+ """+ assert len(segments) == len(labels)+ labels = correct_labels(labels)+ distribute_res = []+ for i in range(len(segments)):+ distribute_res.append([segments[i][0], segments[i][1], labels[i]])+ # merge the same speakers chronologically+ distribute_res = merge_seque(distribute_res)++ # accquire speaker center+ spk_embs = []+ for i in range(labels.max() + 1):+ spk_emb = embeddings[labels == i].mean(0)+ spk_embs.append(spk_emb)+ spk_embs = np.stack(spk_embs)++ def is_overlapped(t1, t2):+ """Is overlapped.++ Args:+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/utils/speaker_utils.py#L128" target="_blank" class="src-more">View full source on GitHub →</a></details></div>+<div class="api-detail" id="e405"><span class="dtag badge-f">function</span><h2><code>correct_labels(labels)</code></h2><div class="dmod">funasr.utils.speaker_utils · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/utils/speaker_utils.py#L178" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Correct labels.</p>+<h4>Args:</h4>+<ul>+<li><code>labels</code> — TODO.</li>+</ul>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code>def correct_labels(labels):+ """Correct labels.++ Args:+ labels: TODO.+ """+ labels_id = 0+ id2id = {}+ new_labels = []+ for i in labels:+ if i not in id2id:+ id2id[i] = labels_id+ labels_id += 1+ new_labels.append(id2id[i])+ return np.array(new_labels)+</code></pre></details></div>+<div class="api-detail" id="e406"><span class="dtag badge-f">function</span><h2><code>merge_seque(distribute_res)</code></h2><div class="dmod">funasr.utils.speaker_utils · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/utils/speaker_utils.py#L195" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Merge seque.</p>+<h4>Args:</h4>+<ul>+<li><code>distribute_res</code> — TODO.</li>+</ul>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code>def merge_seque(distribute_res):+ """Merge seque.++ Args:+ distribute_res: TODO.+ """+ res = [distribute_res[0]]+ for i in range(1, len(distribute_res)):+ if distribute_res[i][2] != res[-1][2] or distribute_res[i][0] > res[-1][1]:+ res.append(distribute_res[i])+ else:+ res[-1][1] = distribute_res[i][1]+ return res+</code></pre></details></div>+<div class="api-detail" id="e407"><span class="dtag badge-f">function</span><h2><code>smooth(res, mindur)</code></h2><div class="dmod">funasr.utils.speaker_utils · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/utils/speaker_utils.py#L210" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Smooth.</p>+<h4>Args:</h4>+<ul>+<li><code>res</code> — TODO.</li>+<li><code>mindur</code> — TODO.</li>+</ul>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code>def smooth(res, mindur=1):+ # short segments are assigned to nearest speakers.+ """Smooth.++ Args:+ res: TODO.+ mindur: TODO.+ """+ for i in range(len(res)):+ res[i][0] = round(res[i][0], 2)+ res[i][1] = round(res[i][1], 2)+ if res[i][1] - res[i][0] < mindur:+ if i == 0:+ res[i][2] = res[i + 1][2]+ elif i == len(res) - 1:+ res[i][2] = res[i - 1][2]+ elif res[i][0] - res[i - 1][1] <= res[i + 1][0] - res[i][1]:+ res[i][2] = res[i - 1][2]+ else:+ res[i][2] = res[i + 1][2]+ # merge the speakers+ res = merge_seque(res)++ return res+</code></pre></details></div>+<div class="api-detail" id="e408"><span class="dtag badge-f">function</span><h2><code>distribute_spk(sentence_list, sd_time_list)</code></h2><div class="dmod">funasr.utils.speaker_utils · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/utils/speaker_utils.py#L236" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Distribute spk.</p>+<h4>Args:</h4>+<ul>+<li><code>sentence_list</code> — TODO.</li>+<li><code>sd_time_list</code> — TODO.</li>+</ul>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code>def distribute_spk(sentence_list, sd_time_list):+ """Distribute spk.++ Args:+ sentence_list: TODO.+ sd_time_list: TODO.+ """+ sd_sentence_list = []+ for d in sentence_list:+ sentence_start = d["ts_list"][0][0]+ sentence_end = d["ts_list"][-1][1]+ sentence_spk = 0+ max_overlap = 0+ for sd_time in sd_time_list:+ spk_st, spk_ed, spk = sd_time+ spk_st = spk_st * 1000+ spk_ed = spk_ed * 1000+ overlap = max(min(sentence_end, spk_ed) - max(sentence_start, spk_st), 0)+ if overlap > max_overlap:+ max_overlap = overlap+ sentence_spk = spk+ d["spk"] = sentence_spk+ sd_sentence_list.append(d)+ return sd_sentence_list+</code></pre></details></div>+<div class="api-detail" id="e409"><span class="dtag badge-f">function</span><h2><code>cif_wo_hidden(alphas, threshold)</code></h2><div class="dmod">funasr.utils.timestamp_tools · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/utils/timestamp_tools.py#L11" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Cif wo hidden.</p>+<h4>Args:</h4>+<ul>+<li><code>alphas</code> — TODO.</li>+<li><code>threshold</code> — TODO.</li>+</ul>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code>def cif_wo_hidden(alphas, threshold):+ """Cif wo hidden.++ Args:+ alphas: TODO.+ threshold: TODO.+ """+ batch_size, len_time = alphas.size()+ # loop varss+ integrate = torch.zeros([batch_size], device=alphas.device)+ # intermediate vars along time+ list_fires = []+ for t in range(len_time):+ alpha = alphas[:, t]+ integrate += alpha+ list_fires.append(integrate)+ fire_place = integrate >= threshold+ integrate = torch.where(+ fire_place,+ integrate - torch.ones([batch_size], device=alphas.device) * threshold,+ integrate,+ )+ fires = torch.stack(list_fires, 1)+ return fires+</code></pre></details></div>+<div class="api-detail" id="e410"><span class="dtag badge-f">function</span><h2><code>ts_prediction_lfr6_standard(us_alphas, us_peaks, char_list, vad_offset, force_time_shift, sil_in_str, upsample_rate)</code></h2><div class="dmod">funasr.utils.timestamp_tools · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/utils/timestamp_tools.py#L37" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Ts prediction lfr6 standard.</p>+<h4>Args:</h4>+<ul>+<li><code>us_alphas</code> — TODO.</li>+<li><code>us_peaks</code> — TODO.</li>+<li><code>char_list</code> — TODO.</li>+<li><code>vad_offset</code> — TODO.</li>+<li><code>force_time_shift</code> — TODO.</li>+<li><code>sil_in_str</code> — TODO.</li>+<li><code>upsample_rate</code> — TODO.</li>+</ul>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code>def ts_prediction_lfr6_standard(+ us_alphas, us_peaks, char_list, vad_offset=0.0, force_time_shift=-1.5, sil_in_str=True, upsample_rate=3,+):+ """Ts prediction lfr6 standard.++ Args:+ us_alphas: TODO.+ us_peaks: TODO.+ char_list: TODO.+ vad_offset: TODO.+ force_time_shift: TODO.+ sil_in_str: TODO.+ upsample_rate: TODO.+ """+ if not len(char_list):+ return "", []+ START_END_THRESHOLD = 5+ MAX_TOKEN_DURATION = 12 # 3 times upsampled+ TIME_RATE=10.0 * 6 / 1000 / upsample_rate+ if len(us_alphas.shape) == 2:+ alphas, peaks = us_alphas[0], us_peaks[0] # support inference batch_size=1 only+ else:+ alphas, peaks = us_alphas, us_peaks+ if char_list[-1] == "</s>":+ char_list = char_list[:-1]+ fire_place = (+ torch.where(peaks >= 1.0 - 1e-4)[0].cpu().numpy() + force_time_shift+ ) # total offset+ if len(fire_place) != len(char_list) + 1:+ alphas /= alphas.sum() / (len(char_list) + 1)+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/utils/timestamp_tools.py#L37" target="_blank" class="src-more">View full source on GitHub →</a></details></div>+<div class="api-detail" id="e411"><span class="dtag badge-f">function</span><h2><code>timestamp_sentence(punc_id_list, timestamp_postprocessed, text_postprocessed, return_raw_text)</code></h2><div class="dmod">funasr.utils.timestamp_tools · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/utils/timestamp_tools.py#L125" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Split recognized text into sentences using punctuation, with timestamps.</p>+<h4>Args:</h4>+<ul>+<li><code>punc_id_list (Tensor/list)</code> — Punctuation IDs from CT-Transformer.</li>+<li><code>Values</code> — 1=none, 2=comma, 3=period, 4=question.</li>+<li><code>timestamp_postprocessed (list)</code> — Per-character timestamps [[start_ms, end_ms], ...].</li>+<li><code>text_postprocessed (str)</code> — Space-separated recognized text.</li>+<li><code>return_raw_text (bool)</code> — Include raw_text in output.</li>+</ul>+<h4>Returns:</h4>+<p>list[dict]: Sentences with keys: text, start, end, timestamp, [raw_text].</p>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code>def timestamp_sentence(+ punc_id_list, timestamp_postprocessed, text_postprocessed, return_raw_text=False+):++ """Split recognized text into sentences using punctuation, with timestamps.++ Args:+ punc_id_list (Tensor/list): Punctuation IDs from CT-Transformer.+ Values: 1=none, 2=comma, 3=period, 4=question.+ timestamp_postprocessed (list): Per-character timestamps [[start_ms, end_ms], ...].+ text_postprocessed (str): Space-separated recognized text.+ return_raw_text (bool): Include raw_text in output.++ Returns:+ list[dict]: Sentences with keys: text, start, end, timestamp, [raw_text].+ """+ punc_list = [",", "。", "?", "、"]+ res = []+ if text_postprocessed is None:+ return res+ if timestamp_postprocessed is None:+ return res+ if len(timestamp_postprocessed) == 0:+ return res+ if len(text_postprocessed) == 0:+ return res++ if punc_id_list is None or len(punc_id_list) == 0:+ res.append(+ {+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/utils/timestamp_tools.py#L125" target="_blank" class="src-more">View full source on GitHub →</a></details></div>+<div class="api-detail" id="e412"><span class="dtag badge-f">function</span><h2><code>timestamp_sentence_en(punc_id_list, timestamp_postprocessed, text_postprocessed, return_raw_text)</code></h2><div class="dmod">funasr.utils.timestamp_tools · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/utils/timestamp_tools.py#L223" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Timestamp sentence en.</p>+<h4>Args:</h4>+<ul>+<li><code>punc_id_list</code> — TODO.</li>+<li><code>timestamp_postprocessed</code> — TODO.</li>+<li><code>text_postprocessed</code> — TODO.</li>+<li><code>return_raw_text</code> — TODO.</li>+</ul>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code>def timestamp_sentence_en(+ punc_id_list, timestamp_postprocessed, text_postprocessed, return_raw_text=False+):+ """Timestamp sentence en.++ Args:+ punc_id_list: TODO.+ timestamp_postprocessed: TODO.+ text_postprocessed: TODO.+ return_raw_text: TODO.+ """+ punc_list = [",", ".", "?", ","]+ res = []+ if text_postprocessed is None:+ return res+ if timestamp_postprocessed is None:+ return res+ if len(timestamp_postprocessed) == 0:+ return res+ if len(text_postprocessed) == 0:+ return res++ if punc_id_list is None or len(punc_id_list) == 0:+ res.append(+ {+ "text": text_postprocessed.split(),+ "start": timestamp_postprocessed[0][0],+ "end": timestamp_postprocessed[-1][1],+ "timestamp": timestamp_postprocessed,+ }+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/utils/timestamp_tools.py#L223" target="_blank" class="src-more">View full source on GitHub →</a></details></div>+<div class="api-detail" id="e413"><span class="dtag badge-c">class</span><h2>MakePadMask</h2><div class="dmod">funasr.utils.torch_function · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/utils/torch_function.py#L9" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p class="muted">No documentation yet.</p></div><details class="src-block"><summary>📄 Source code</summary><pre><code>class MakePadMask(nn.Module):+ def __init__(self, max_seq_len=512, flip=True):+ """Initialize MakePadMask.++ Args:+ max_seq_len: TODO.+ flip: TODO.+ """+ super().__init__()+ if flip:+ self.mask_pad = torch.Tensor(1 - np.tri(max_seq_len)).type(torch.bool)+ else:+ self.mask_pad = torch.Tensor(np.tri(max_seq_len)).type(torch.bool)++ def forward(self, lengths, xs=None, length_dim=-1, maxlen=None):+ """Make mask tensor containing indices of padded part.+ This implementation creates the same mask tensor with original make_pad_mask,+ which can be converted into onnx format.+ Dimension length of xs should be 2 or 3.+ """+ if length_dim == 0:+ raise ValueError("length_dim cannot be 0: {}".format(length_dim))++ if xs is not None and len(xs.shape) == 3:+ if length_dim == 1:+ lengths = lengths.unsqueeze(1).expand(*xs.transpose(1, 2).shape[:2])+ else:+ lengths = lengths.unsqueeze(1).expand(*xs.shape[:2])++ if maxlen is not None:+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/utils/torch_function.py#L9" target="_blank" class="src-more">View full source on GitHub →</a></details><h3 class="mt">Methods</h3><div class="mblk"><code>.forward(lengths, xs, length_dim, maxlen)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/utils/torch_function.py#L23" target="_blank" class="gh-link">L23</a><div class="mdoc"><p>Make mask tensor containing indices of padded part.</p>+<p>This implementation creates the same mask tensor with original make_pad_mask,</p>+<p>which can be converted into onnx format.</p>+<p>Dimension length of xs should be 2 or 3.</p>+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def forward(self, lengths, xs=None, length_dim=-1, maxlen=None):+ """Make mask tensor containing indices of padded part.+ This implementation creates the same mask tensor with original make_pad_mask,+ which can be converted into onnx format.+ Dimension length of xs should be 2 or 3.+ """+ if length_dim == 0:+ raise ValueError("length_dim cannot be 0: {}".format(length_dim))++ if xs is not None and len(xs.shape) == 3:+ if length_dim == 1:+ lengths = lengths.unsqueeze(1).expand(*xs.transpose(1, 2).shape[:2])+ else:+ lengths = lengths.unsqueeze(1).expand(*xs.shape[:2])++ if maxlen is not None:+ m = maxlen+ elif xs is not None:+ m = xs.shape[-1]+ else:+ m = torch.max(lengths)++ mask = self.mask_pad[lengths - 1][..., :m].type(torch.float32)++ if length_dim == 1:+ return mask.transpose(1, 2)+ else:+ return mask+</code></pre></details></div></div>+<div class="api-detail" id="e414"><span class="dtag badge-m">method</span><h2><code>MakePadMask.forward(lengths, xs, length_dim, maxlen)</code></h2><div class="dmod">funasr.utils.torch_function.MakePadMask · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/utils/torch_function.py#L23" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Make mask tensor containing indices of padded part.</p>+<p>This implementation creates the same mask tensor with original make_pad_mask,</p>+<p>which can be converted into onnx format.</p>+<p>Dimension length of xs should be 2 or 3.</p>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def forward(self, lengths, xs=None, length_dim=-1, maxlen=None):+ """Make mask tensor containing indices of padded part.+ This implementation creates the same mask tensor with original make_pad_mask,+ which can be converted into onnx format.+ Dimension length of xs should be 2 or 3.+ """+ if length_dim == 0:+ raise ValueError("length_dim cannot be 0: {}".format(length_dim))++ if xs is not None and len(xs.shape) == 3:+ if length_dim == 1:+ lengths = lengths.unsqueeze(1).expand(*xs.transpose(1, 2).shape[:2])+ else:+ lengths = lengths.unsqueeze(1).expand(*xs.shape[:2])++ if maxlen is not None:+ m = maxlen+ elif xs is not None:+ m = xs.shape[-1]+ else:+ m = torch.max(lengths)++ mask = self.mask_pad[lengths - 1][..., :m].type(torch.float32)++ if length_dim == 1:+ return mask.transpose(1, 2)+ else:+ return mask+</code></pre></details></div>+<div class="api-detail" id="e415"><span class="dtag badge-c">class</span><h2>sequence_mask</h2><div class="dmod">funasr.utils.torch_function · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/utils/torch_function.py#L53" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p class="muted">No documentation yet.</p></div><details class="src-block"><summary>📄 Source code</summary><pre><code>class sequence_mask(nn.Module):+ def __init__(self, max_seq_len=512, flip=True):+ """Initialize sequence_mask.++ Args:+ max_seq_len: TODO.+ flip: TODO.+ """+ super().__init__()++ def forward(self, lengths, max_seq_len=None, dtype=torch.float32, device=None):+ """Forward pass for training.++ Args:+ lengths: TODO.+ max_seq_len: TODO.+ dtype: TODO.+ device: Target device ("cuda:0", "cpu", etc.).+ """+ if max_seq_len is None:+ max_seq_len = lengths.max()+ row_vector = torch.arange(0, max_seq_len, 1).to(lengths.device)+ matrix = torch.unsqueeze(lengths, dim=-1)+ mask = row_vector < matrix++ return mask.type(dtype).to(device) if device is not None else mask.type(dtype)+</code></pre></details><h3 class="mt">Methods</h3><div class="mblk"><code>.forward(lengths, max_seq_len, dtype, device)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/utils/torch_function.py#L63" target="_blank" class="gh-link">L63</a><div class="mdoc"><p>Forward pass for training.</p>+<h4>Args:</h4>+<ul>+<li><code>lengths</code> — TODO.</li>+<li><code>max_seq_len</code> — TODO.</li>+<li><code>dtype</code> — TODO.</li>+<li><code>device</code> — Target device ("cuda:0", "cpu", etc.).</li>+</ul>+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def forward(self, lengths, max_seq_len=None, dtype=torch.float32, device=None):+ """Forward pass for training.++ Args:+ lengths: TODO.+ max_seq_len: TODO.+ dtype: TODO.+ device: Target device ("cuda:0", "cpu", etc.).+ """+ if max_seq_len is None:+ max_seq_len = lengths.max()+ row_vector = torch.arange(0, max_seq_len, 1).to(lengths.device)+ matrix = torch.unsqueeze(lengths, dim=-1)+ mask = row_vector < matrix++ return mask.type(dtype).to(device) if device is not None else mask.type(dtype)+</code></pre></details></div></div>+<div class="api-detail" id="e416"><span class="dtag badge-m">method</span><h2><code>sequence_mask.forward(lengths, max_seq_len, dtype, device)</code></h2><div class="dmod">funasr.utils.torch_function.sequence_mask · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/utils/torch_function.py#L63" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Forward pass for training.</p>+<h4>Args:</h4>+<ul>+<li><code>lengths</code> — TODO.</li>+<li><code>max_seq_len</code> — TODO.</li>+<li><code>dtype</code> — TODO.</li>+<li><code>device</code> — Target device ("cuda:0", "cpu", etc.).</li>+</ul>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def forward(self, lengths, max_seq_len=None, dtype=torch.float32, device=None):+ """Forward pass for training.++ Args:+ lengths: TODO.+ max_seq_len: TODO.+ dtype: TODO.+ device: Target device ("cuda:0", "cpu", etc.).+ """+ if max_seq_len is None:+ max_seq_len = lengths.max()+ row_vector = torch.arange(0, max_seq_len, 1).to(lengths.device)+ matrix = torch.unsqueeze(lengths, dim=-1)+ mask = row_vector < matrix++ return mask.type(dtype).to(device) if device is not None else mask.type(dtype)+</code></pre></details></div>+<div class="api-detail" id="e417"><span class="dtag badge-f">function</span><h2><code>normalize(input, p, dim, out)</code></h2><div class="dmod">funasr.utils.torch_function · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/utils/torch_function.py#L81" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Normalize.</p>+<h4>Args:</h4>+<ul>+<li><code>input</code> — Input audio/text data.</li>+<li><code>p</code> — TODO.</li>+<li><code>dim</code> — TODO.</li>+<li><code>out</code> — TODO.</li>+</ul>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code>def normalize(+ input: torch.Tensor, p: float = 2.0, dim: int = 1, out: Optional[torch.Tensor] = None+) -> torch.Tensor:+ """Normalize.++ Args:+ input: Input audio/text data.+ p: TODO.+ dim: TODO.+ out: TODO.+ """+ if out is None:+ denom = input.norm(p, dim, keepdim=True).expand_as(input)+ return input / denom+ else:+ denom = input.norm(p, dim, keepdim=True).expand_as(input)+ return torch.div(input, denom, out=out)+</code></pre></details></div>+<div class="api-detail" id="e418"><span class="dtag badge-f">function</span><h2><code>subsequent_mask(size)</code></h2><div class="dmod">funasr.utils.torch_function · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/utils/torch_function.py#L100" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Subsequent mask.</p>+<h4>Args:</h4>+<ul>+<li><code>size</code> — TODO.</li>+</ul>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code>def subsequent_mask(size: torch.Tensor):+ """Subsequent mask.++ Args:+ size: TODO.+ """+ return torch.ones(size, size).tril()+</code></pre></details></div>+<div class="api-detail" id="e419"><span class="dtag badge-f">function</span><h2><code>MakePadMask_test()</code></h2><div class="dmod">funasr.utils.torch_function · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/utils/torch_function.py#L109" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Makepadmask test.</p>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code>def MakePadMask_test():+ """Makepadmask test."""+ feats_length = torch.tensor([10]).type(torch.long)+ mask_fn = MakePadMask()+ mask = mask_fn(feats_length)+ print(mask)+</code></pre></details></div>+<div class="api-detail" id="e420"><span class="dtag badge-f">function</span><h2><code>str2bool(value)</code></h2><div class="dmod">funasr.utils.type_utils · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/utils/type_utils.py#L9" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Str2bool.</p>+<h4>Args:</h4>+<ul>+<li><code>value</code> — TODO.</li>+</ul>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code>def str2bool(value: str) -> bool:+ """Str2bool.++ Args:+ value: TODO.+ """+ return bool(strtobool(value))+</code></pre></details></div>+<div class="api-detail" id="e421"><span class="dtag badge-f">function</span><h2><code>remove_parenthesis(value)</code></h2><div class="dmod">funasr.utils.type_utils · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/utils/type_utils.py#L18" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Remove parenthesis.</p>+<h4>Args:</h4>+<ul>+<li><code>value</code> — TODO.</li>+</ul>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code>def remove_parenthesis(value: str):+ """Remove parenthesis.++ Args:+ value: TODO.+ """+ value = value.strip()+ if value.startswith("(") and value.endswith(")"):+ value = value[1:-1]+ elif value.startswith("[") and value.endswith("]"):+ value = value[1:-1]+ return value+</code></pre></details></div>+<div class="api-detail" id="e422"><span class="dtag badge-f">function</span><h2><code>remove_quotes(value)</code></h2><div class="dmod">funasr.utils.type_utils · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/utils/type_utils.py#L32" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Remove quotes.</p>+<h4>Args:</h4>+<ul>+<li><code>value</code> — TODO.</li>+</ul>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code>def remove_quotes(value: str):+ """Remove quotes.++ Args:+ value: TODO.+ """+ value = value.strip()+ if value.startswith('"') and value.endswith('"'):+ value = value[1:-1]+ elif value.startswith("'") and value.endswith("'"):+ value = value[1:-1]+ return value+</code></pre></details></div>+<div class="api-detail" id="e423"><span class="dtag badge-f">function</span><h2><code>int_or_none(value)</code></h2><div class="dmod">funasr.utils.type_utils · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/utils/type_utils.py#L46" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>int_or_none.</p>+<h4>Examples:</h4>+<p>>>> import argparse</p>+<p>>>> parser = argparse.ArgumentParser()</p>+<p>>>> _ = parser.add_argument('--foo', type=int_or_none)</p>+<p>>>> parser.parse_args(['--foo', '456'])</p>+<p>Namespace(foo=456)</p>+<p>>>> parser.parse_args(['--foo', 'none'])</p>+<p>Namespace(foo=None)</p>+<p>>>> parser.parse_args(['--foo', 'null'])</p>+<p>Namespace(foo=None)</p>+<p>>>> parser.parse_args(['--foo', 'nil'])</p>+<p>Namespace(foo=None)</p>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code>def int_or_none(value: str) -> Optional[int]:+ """int_or_none.++ Examples:+ >>> import argparse+ >>> parser = argparse.ArgumentParser()+ >>> _ = parser.add_argument('--foo', type=int_or_none)+ >>> parser.parse_args(['--foo', '456'])+ Namespace(foo=456)+ >>> parser.parse_args(['--foo', 'none'])+ Namespace(foo=None)+ >>> parser.parse_args(['--foo', 'null'])+ Namespace(foo=None)+ >>> parser.parse_args(['--foo', 'nil'])+ Namespace(foo=None)++ """+ if value.strip().lower() in ("none", "null", "nil"):+ return None+ return int(value)+</code></pre></details></div>+<div class="api-detail" id="e424"><span class="dtag badge-f">function</span><h2><code>float_or_none(value)</code></h2><div class="dmod">funasr.utils.type_utils · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/utils/type_utils.py#L68" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>float_or_none.</p>+<h4>Examples:</h4>+<p>>>> import argparse</p>+<p>>>> parser = argparse.ArgumentParser()</p>+<p>>>> _ = parser.add_argument('--foo', type=float_or_none)</p>+<p>>>> parser.parse_args(['--foo', '4.5'])</p>+<p>Namespace(foo=4.5)</p>+<p>>>> parser.parse_args(['--foo', 'none'])</p>+<p>Namespace(foo=None)</p>+<p>>>> parser.parse_args(['--foo', 'null'])</p>+<p>Namespace(foo=None)</p>+<p>>>> parser.parse_args(['--foo', 'nil'])</p>+<p>Namespace(foo=None)</p>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code>def float_or_none(value: str) -> Optional[float]:+ """float_or_none.++ Examples:+ >>> import argparse+ >>> parser = argparse.ArgumentParser()+ >>> _ = parser.add_argument('--foo', type=float_or_none)+ >>> parser.parse_args(['--foo', '4.5'])+ Namespace(foo=4.5)+ >>> parser.parse_args(['--foo', 'none'])+ Namespace(foo=None)+ >>> parser.parse_args(['--foo', 'null'])+ Namespace(foo=None)+ >>> parser.parse_args(['--foo', 'nil'])+ Namespace(foo=None)++ """+ if value.strip().lower() in ("none", "null", "nil"):+ return None+ return float(value)+</code></pre></details></div>+<div class="api-detail" id="e425"><span class="dtag badge-f">function</span><h2><code>humanfriendly_parse_size_or_none(value)</code></h2><div class="dmod">funasr.utils.type_utils · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/utils/type_utils.py#L90" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Humanfriendly parse size or none.</p>+<h4>Args:</h4>+<ul>+<li><code>value</code> — TODO.</li>+</ul>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code>def humanfriendly_parse_size_or_none(value) -> Optional[float]:+ """Humanfriendly parse size or none.++ Args:+ value: TODO.+ """+ if value.strip().lower() in ("none", "null", "nil"):+ return None+ return humanfriendly.parse_size(value)+</code></pre></details></div>+<div class="api-detail" id="e426"><span class="dtag badge-f">function</span><h2><code>str_or_int(value)</code></h2><div class="dmod">funasr.utils.type_utils · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/utils/type_utils.py#L101" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Str or int.</p>+<h4>Args:</h4>+<ul>+<li><code>value</code> — TODO.</li>+</ul>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code>def str_or_int(value: str) -> Union[str, int]:+ """Str or int.++ Args:+ value: TODO.+ """+ try:+ return int(value)+ except ValueError:+ return value+</code></pre></details></div>+<div class="api-detail" id="e427"><span class="dtag badge-f">function</span><h2><code>str_or_none(value)</code></h2><div class="dmod">funasr.utils.type_utils · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/utils/type_utils.py#L113" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>str_or_none.</p>+<h4>Examples:</h4>+<p>>>> import argparse</p>+<p>>>> parser = argparse.ArgumentParser()</p>+<p>>>> _ = parser.add_argument('--foo', type=str_or_none)</p>+<p>>>> parser.parse_args(['--foo', 'aaa'])</p>+<p>Namespace(foo='aaa')</p>+<p>>>> parser.parse_args(['--foo', 'none'])</p>+<p>Namespace(foo=None)</p>+<p>>>> parser.parse_args(['--foo', 'null'])</p>+<p>Namespace(foo=None)</p>+<p>>>> parser.parse_args(['--foo', 'nil'])</p>+<p>Namespace(foo=None)</p>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code>def str_or_none(value: str) -> Optional[str]:+ """str_or_none.++ Examples:+ >>> import argparse+ >>> parser = argparse.ArgumentParser()+ >>> _ = parser.add_argument('--foo', type=str_or_none)+ >>> parser.parse_args(['--foo', 'aaa'])+ Namespace(foo='aaa')+ >>> parser.parse_args(['--foo', 'none'])+ Namespace(foo=None)+ >>> parser.parse_args(['--foo', 'null'])+ Namespace(foo=None)+ >>> parser.parse_args(['--foo', 'nil'])+ Namespace(foo=None)++ """+ if value.strip().lower() in ("none", "null", "nil"):+ return None+ return value+</code></pre></details></div>+<div class="api-detail" id="e428"><span class="dtag badge-f">function</span><h2><code>str2pair_str(value)</code></h2><div class="dmod">funasr.utils.type_utils · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/utils/type_utils.py#L135" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>str2pair_str.</p>+<h4>Examples:</h4>+<p>>>> import argparse</p>+<p>>>> str2pair_str('abc,def ')</p>+<p>('abc', 'def')</p>+<p>>>> parser = argparse.ArgumentParser()</p>+<p>>>> _ = parser.add_argument('--foo', type=str2pair_str)</p>+<p>>>> parser.parse_args(['--foo', 'abc,def'])</p>+<p>Namespace(foo=('abc', 'def'))</p>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code>def str2pair_str(value: str) -> Tuple[str, str]:+ """str2pair_str.++ Examples:+ >>> import argparse+ >>> str2pair_str('abc,def ')+ ('abc', 'def')+ >>> parser = argparse.ArgumentParser()+ >>> _ = parser.add_argument('--foo', type=str2pair_str)+ >>> parser.parse_args(['--foo', 'abc,def'])+ Namespace(foo=('abc', 'def'))++ """+ value = remove_parenthesis(value)+ a, b = value.split(",")++ # Workaround for configargparse issues:+ # If the list values are given from yaml file,+ # the value givent to type() is shaped as python-list,+ # e.g. ['a', 'b', 'c'],+ # so we need to remove double quotes from it.+ return remove_quotes(a), remove_quotes(b)+</code></pre></details></div>+<div class="api-detail" id="e429"><span class="dtag badge-f">function</span><h2><code>str2triple_str(value)</code></h2><div class="dmod">funasr.utils.type_utils · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/utils/type_utils.py#L159" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>str2triple_str.</p>+<h4>Examples:</h4>+<p>>>> str2triple_str('abc,def ,ghi')</p>+<p>('abc', 'def', 'ghi')</p>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code>def str2triple_str(value: str) -> Tuple[str, str, str]:+ """str2triple_str.++ Examples:+ >>> str2triple_str('abc,def ,ghi')+ ('abc', 'def', 'ghi')+ """+ value = remove_parenthesis(value)+ a, b, c = value.split(",")++ # Workaround for configargparse issues:+ # If the list values are given from yaml file,+ # the value givent to type() is shaped as python-list,+ # e.g. ['a', 'b', 'c'],+ # so we need to remove quotes from it.+ return remove_quotes(a), remove_quotes(b), remove_quotes(c)+</code></pre></details></div>+<div class="api-detail" id="e430"><span class="dtag badge-f">function</span><h2><code>slice_padding_fbank(speech, speech_lengths, vad_segments)</code></h2><div class="dmod">funasr.utils.vad_utils · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/utils/vad_utils.py#L5" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Slice padding fbank.</p>+<h4>Args:</h4>+<ul>+<li><code>speech</code> — Speech audio tensor, shape (batch, time).</li>+<li><code>speech_lengths</code> — Length of each speech sample.</li>+<li><code>vad_segments</code> — TODO.</li>+</ul>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code>def slice_padding_fbank(speech, speech_lengths, vad_segments):+ """Slice padding fbank.++ Args:+ speech: Speech audio tensor, shape (batch, time).+ speech_lengths: Length of each speech sample.+ vad_segments: TODO.+ """+ speech_list = []+ speech_lengths_list = []+ for i, segment in enumerate(vad_segments):++ bed_idx = int(segment[0][0] * 16)+ end_idx = min(int(segment[0][1] * 16), speech_lengths[0])+ speech_i = speech[0, bed_idx:end_idx]+ speech_lengths_i = end_idx - bed_idx+ speech_list.append(speech_i)+ speech_lengths_list.append(speech_lengths_i)+ feats_pad = pad_sequence(speech_list, batch_first=True, padding_value=0.0)+ speech_lengths_pad = torch.Tensor(speech_lengths_list).int()+ return feats_pad, speech_lengths_pad+</code></pre></details></div>+<div class="api-detail" id="e431"><span class="dtag badge-f">function</span><h2><code>slice_padding_audio_samples(speech, speech_lengths, vad_segments)</code></h2><div class="dmod">funasr.utils.vad_utils · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/utils/vad_utils.py#L28" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Slice audio into VAD segments with proper padding.</p>+<h4>Args:</h4>+<ul>+<li><code>speech (Tensor)</code> — Full audio tensor.</li>+<li><code>speech_lengths (int)</code> — Total audio length.</li>+<li><code>vad_segments (list)</code> — List of (segment_info, original_index) tuples,</li>+<li class="sub">where segment_info is [start_ms, end_ms].</li>+</ul>+<h4>Returns:</h4>+<ul>+<li><code>tuple</code> — (speech_list, speech_lengths_list) - lists of numpy arrays and their lengths.</li>+</ul>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code>def slice_padding_audio_samples(speech, speech_lengths, vad_segments):++ """Slice audio into VAD segments with proper padding.++ Args:+ speech (Tensor): Full audio tensor.+ speech_lengths (int): Total audio length.+ vad_segments (list): List of (segment_info, original_index) tuples,+ where segment_info is [start_ms, end_ms].++ Returns:+ tuple: (speech_list, speech_lengths_list) - lists of numpy arrays and their lengths.+ """+ speech_list = []+ speech_lengths_list = []+ for i, segment in enumerate(vad_segments):+ bed_idx = int(segment[0][0] * 16)+ end_idx = min(int(segment[0][1] * 16), speech_lengths)+ speech_i = speech[bed_idx:end_idx]+ speech_lengths_i = end_idx - bed_idx+ speech_list.append(speech_i)+ speech_lengths_list.append(speech_lengths_i)++ return speech_list, speech_lengths_list+</code></pre></details></div>+<div class="api-detail" id="e432"><span class="dtag badge-f">function</span><h2><code>merge_vad(vad_result, max_length, min_length)</code></h2><div class="dmod">funasr.utils.vad_utils · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/utils/vad_utils.py#L54" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Merge short VAD segments to reduce fragmentation.</p>+<h4>Args:</h4>+<ul>+<li><code>vad_result (list)</code> — VAD segments [[start_ms, end_ms], ...].</li>+<li><code>max_length (int)</code> — Maximum merged segment length in ms (default 15000).</li>+<li><code>min_length (int)</code> — Minimum segment length; shorter ones get merged (default 0).</li>+</ul>+<h4>Returns:</h4>+<ul>+<li><code>list</code> — Merged VAD segments [[start_ms, end_ms], ...].</li>+</ul>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code>def merge_vad(vad_result, max_length=15000, min_length=0):++ """Merge short VAD segments to reduce fragmentation.++ Args:+ vad_result (list): VAD segments [[start_ms, end_ms], ...].+ max_length (int): Maximum merged segment length in ms (default 15000).+ min_length (int): Minimum segment length; shorter ones get merged (default 0).++ Returns:+ list: Merged VAD segments [[start_ms, end_ms], ...].+ """+ new_result = []+ if len(vad_result) <= 1:+ return vad_result+ time_step = [t[0] for t in vad_result] + [t[1] for t in vad_result]+ time_step = sorted(list(set(time_step)))+ if len(time_step) == 0:+ return []+ bg = 0+ for i in range(len(time_step) - 1):+ time = time_step[i]+ if time_step[i + 1] - bg < max_length:+ continue+ if time - bg > min_length:+ new_result.append([bg, time])+ # if time - bg < max_length * 1.5:+ # new_result.append([bg, time])+ # else:+ # split_num = int(time - bg) // max_length + 1+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/utils/vad_utils.py#L54" target="_blank" class="src-more">View full source on GitHub →</a></details></div>+<div class="api-detail" id="e433"><span class="dtag badge-f">function</span><h2><code>get_pypi_version(package_name)</code></h2><div class="dmod">funasr.utils.version_checker · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/utils/version_checker.py#L5" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Get pypi version.</p>+<h4>Args:</h4>+<ul>+<li><code>package_name</code> — TODO.</li>+</ul>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code>def get_pypi_version(package_name):+ """Get pypi version.++ Args:+ package_name: TODO.+ """+ import requests++ url = f"https://pypi.org/pypi/{package_name}/json"+ response = requests.get(url)+ if response.status_code == 200:+ data = response.json()+ return version.parse(data["info"]["version"])+ else:+ raise Exception("Failed to retrieve version information from PyPI.")+</code></pre></details></div>+<div class="api-detail" id="e434"><span class="dtag badge-f">function</span><h2><code>check_for_update(disable)</code></h2><div class="dmod">funasr.utils.version_checker · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/utils/version_checker.py#L22" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Check for update.</p>+<h4>Args:</h4>+<ul>+<li><code>disable</code> — TODO.</li>+</ul>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code>def check_for_update(disable=False):+ """Check for update.++ Args:+ disable: TODO.+ """+ current_version = version.parse(__version__)+ print(f"funasr version: {current_version}.")+ if disable:+ return++ print(+ "Check update of funasr, and it would cost few times. You may disable it by set `disable_update=True` in AutoModel"+ )+ pypi_version = get_pypi_version("funasr")++ if current_version < pypi_version:+ print(f"New version is available: {pypi_version}.")+ print('Please use the command "pip install -U funasr" to upgrade.')+ else:+ print(f"You are using the latest version of funasr-{current_version}")+</code></pre></details></div>+<div class="api-detail" id="e435"><span class="dtag badge-f">function</span><h2><code>main_hydra(kwargs)</code></h2><div class="dmod">funasr.bin.compute_audio_cmvn · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/bin/compute_audio_cmvn.py#L15" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Main hydra.</p>+<h4>Args:</h4>+<ul>+<li><code>kwargs</code> — Additional keyword arguments.</li>+</ul>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code>def main_hydra(kwargs: DictConfig):+ """Main hydra.++ Args:+ kwargs: Additional keyword arguments.+ """+ if kwargs.get("debug", False):+ import pdb++ pdb.set_trace()++ assert "model" in kwargs+ if "model_conf" not in kwargs:+ logging.info("download models from model hub: {}".format(kwargs.get("hub", "ms")))+ kwargs = download_model(is_training=kwargs.get("is_training", True), **kwargs)++ main(**kwargs)+</code></pre></details></div>+<div class="api-detail" id="e436"><span class="dtag badge-f">function</span><h2><code>main(**kwargs)</code></h2><div class="dmod">funasr.bin.compute_audio_cmvn · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/bin/compute_audio_cmvn.py#L34" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Main.</p>+<h4>Args:</h4>+<ul>+<li><code>**kwargs</code> — Additional keyword arguments.</li>+</ul>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code>def main(**kwargs):+ """Main.++ Args:+ **kwargs: Additional keyword arguments.+ """+ print(kwargs)+ # set random seed+ # tables.print()+ set_all_random_seed(kwargs.get("seed", 0))+ torch.backends.cudnn.enabled = kwargs.get("cudnn_enabled", torch.backends.cudnn.enabled)+ torch.backends.cudnn.benchmark = kwargs.get("cudnn_benchmark", torch.backends.cudnn.benchmark)+ torch.backends.cudnn.deterministic = kwargs.get("cudnn_deterministic", True)++ tokenizer = kwargs.get("tokenizer", None)++ # build frontend if frontend is none None+ frontend = kwargs.get("frontend", None)+ if frontend is not None:+ frontend_class = tables.frontend_classes.get(frontend)+ frontend = frontend_class(**kwargs["frontend_conf"])+ kwargs["frontend"] = frontend+ kwargs["input_size"] = frontend.output_size()++ # dataset+ dataset_class = tables.dataset_classes.get(kwargs.get("dataset", "AudioDataset"))+ dataset_train = dataset_class(+ kwargs.get("train_data_set_list"),+ frontend=frontend,+ tokenizer=None,+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/bin/compute_audio_cmvn.py#L34" target="_blank" class="src-more">View full source on GitHub →</a></details></div>+<div class="api-detail" id="e437"><span class="dtag badge-f">function</span><h2><code>main_hydra(cfg)</code></h2><div class="dmod">funasr.bin.export · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/bin/export.py#L10" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Main hydra.</p>+<h4>Args:</h4>+<ul>+<li><code>cfg</code> — Configuration overrides.</li>+</ul>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code>def main_hydra(cfg: DictConfig):+ """Main hydra.++ Args:+ cfg: Configuration overrides.+ """+ def to_plain_list(cfg_item):+ """To plain list.++ Args:+ cfg_item: TODO.+ """+ if isinstance(cfg_item, ListConfig):+ return OmegaConf.to_container(cfg_item, resolve=True)+ elif isinstance(cfg_item, DictConfig):+ return {k: to_plain_list(v) for k, v in cfg_item.items()}+ else:+ return cfg_item++ kwargs = to_plain_list(cfg)++ if kwargs.get("debug", False):+ import pdb++ pdb.set_trace()++ if "device" not in kwargs:+ kwargs["device"] = "cpu"+ model = AutoModel(**kwargs)++</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/bin/export.py#L10" target="_blank" class="src-more">View full source on GitHub →</a></details></div>+<div class="api-detail" id="e438"><span class="dtag badge-f">function</span><h2><code>main_hydra(cfg)</code></h2><div class="dmod">funasr.bin.inference · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/bin/inference.py#L9" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Main hydra.</p>+<h4>Args:</h4>+<ul>+<li><code>cfg</code> — Configuration overrides.</li>+</ul>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code>def main_hydra(cfg: DictConfig):+ """Main hydra.++ Args:+ cfg: Configuration overrides.+ """+ def to_plain_list(cfg_item):+ """To plain list.++ Args:+ cfg_item: TODO.+ """+ if isinstance(cfg_item, ListConfig):+ return OmegaConf.to_container(cfg_item, resolve=True)+ elif isinstance(cfg_item, DictConfig):+ return {k: to_plain_list(v) for k, v in cfg_item.items()}+ else:+ return cfg_item++ kwargs = to_plain_list(cfg)++ if kwargs.get("debug", False):+ import pdb++ pdb.set_trace()+ model = AutoModel(**kwargs)+ res = model.generate(input=kwargs["input"])+ print(res)+</code></pre></details></div>+<div class="api-detail" id="e439"><span class="dtag badge-f">function</span><h2><code>field2slice(field)</code></h2><div class="dmod">funasr.bin.tokenize_text · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/bin/tokenize_text.py#L19" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Convert field string to slice</p>+<p>Note that field string accepts 1-based integer.</p>+<h4>Examples:</h4>+<p>>>> field2slice("1-")</p>+<p>slice(0, None, None)</p>+<p>>>> field2slice("1-3")</p>+<p>slice(0, 3, None)</p>+<p>>>> field2slice("-3")</p>+<p>slice(None, 3, None)</p>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code>def field2slice(field: Optional[str]) -> slice:+ """Convert field string to slice++ Note that field string accepts 1-based integer.++ Examples:+ >>> field2slice("1-")+ slice(0, None, None)+ >>> field2slice("1-3")+ slice(0, 3, None)+ >>> field2slice("-3")+ slice(None, 3, None)+ """+ field = field.strip()+ try:+ if "-" in field:+ # e.g. "2-" or "2-5" or "-7"+ s1, s2 = field.split("-", maxsplit=1)+ if s1.strip() == "":+ s1 = None+ else:+ s1 = int(s1)+ if s1 == 0:+ raise ValueError("1-based string")+ if s2.strip() == "":+ s2 = None+ else:+ s2 = int(s2)+ else:+ # e.g. "2"+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/bin/tokenize_text.py#L19" target="_blank" class="src-more">View full source on GitHub →</a></details></div>+<div class="api-detail" id="e440"><span class="dtag badge-f">function</span><h2><code>tokenize(input, output, field, delimiter, token_type, space_symbol, non_linguistic_symbols, bpemodel, log_level, write_vocabulary, vocabulary_size, remove_non_linguistic_symbols, cutoff, add_symbol, cleaner, g2p)</code></h2><div class="dmod">funasr.bin.tokenize_text · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/bin/tokenize_text.py#L65" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Tokenize.</p>+<h4>Args:</h4>+<ul>+<li><code>input</code> — Input audio/text data.</li>+<li><code>output</code> — TODO.</li>+<li><code>field</code> — TODO.</li>+<li><code>delimiter</code> — TODO.</li>+<li><code>token_type</code> — TODO.</li>+<li><code>space_symbol</code> — TODO.</li>+<li><code>non_linguistic_symbols</code> — TODO.</li>+<li><code>bpemodel</code> — TODO.</li>+<li><code>log_level</code> — TODO.</li>+<li><code>write_vocabulary</code> — TODO.</li>+<li><code>vocabulary_size</code> — Size/dimension parameter.</li>+<li><code>remove_non_linguistic_symbols</code> — TODO.</li>+<li><code>cutoff</code> — TODO.</li>+<li><code>add_symbol</code> — TODO.</li>+<li><code>cleaner</code> — TODO.</li>+<li><code>g2p</code> — TODO.</li>+</ul>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code>def tokenize(+ input: str,+ output: str,+ field: Optional[str],+ delimiter: Optional[str],+ token_type: str,+ space_symbol: str,+ non_linguistic_symbols: Optional[str],+ bpemodel: Optional[str],+ log_level: str,+ write_vocabulary: bool,+ vocabulary_size: int,+ remove_non_linguistic_symbols: bool,+ cutoff: int,+ add_symbol: List[str],+ cleaner: Optional[str],+ g2p: Optional[str],+):++ """Tokenize.++ Args:+ input: Input audio/text data.+ output: TODO.+ field: TODO.+ delimiter: TODO.+ token_type: TODO.+ space_symbol: TODO.+ non_linguistic_symbols: TODO.+ bpemodel: TODO.+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/bin/tokenize_text.py#L65" target="_blank" class="src-more">View full source on GitHub →</a></details></div>+<div class="api-detail" id="e441"><span class="dtag badge-f">function</span><h2><code>get_parser()</code></h2><div class="dmod">funasr.bin.tokenize_text · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/bin/tokenize_text.py#L207" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Get parser.</p>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code>def get_parser() -> argparse.ArgumentParser:+ """Get parser."""+ parser = argparse.ArgumentParser(+ description="Tokenize texts",+ formatter_class=argparse.ArgumentDefaultsHelpFormatter,+ )+ parser.add_argument(+ "--log_level",+ type=lambda x: x.upper(),+ default="INFO",+ choices=("CRITICAL", "ERROR", "WARNING", "INFO", "DEBUG", "NOTSET"),+ help="The verbose level of logging",+ )++ parser.add_argument("--input", "-i", required=True, help="Input text. - indicates sys.stdin")+ parser.add_argument("--output", "-o", required=True, help="Output text. - indicates sys.stdout")+ parser.add_argument(+ "--field",+ "-f",+ help="The target columns of the input text as 1-based integer. e.g 2-",+ )+ parser.add_argument(+ "--token_type",+ "-t",+ default="char",+ choices=["char", "bpe", "word", "phn"],+ help="Token type",+ )+ parser.add_argument("--delimiter", "-d", default=None, help="The delimiter")+ parser.add_argument("--space_symbol", default="<space>", help="The space symbol")+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/bin/tokenize_text.py#L207" target="_blank" class="src-more">View full source on GitHub →</a></details></div>+<div class="api-detail" id="e442"><span class="dtag badge-f">function</span><h2><code>main(cmd)</code></h2><div class="dmod">funasr.bin.tokenize_text · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/bin/tokenize_text.py#L289" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Main.</p>+<h4>Args:</h4>+<ul>+<li><code>cmd</code> — TODO.</li>+</ul>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code>def main(cmd=None):+ """Main.++ Args:+ cmd: TODO.+ """+ print(get_commandline_args(), file=sys.stderr)+ parser = get_parser()+ args = parser.parse_args(cmd)+ kwargs = vars(args)+ tokenize(**kwargs)+</code></pre></details></div>+<div class="api-detail" id="e443"><span class="dtag badge-f">function</span><h2><code>main_hydra(kwargs)</code></h2><div class="dmod">funasr.bin.train · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/bin/train.py#L42" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Main hydra.</p>+<h4>Args:</h4>+<ul>+<li><code>kwargs</code> — Additional keyword arguments.</li>+</ul>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code>def main_hydra(kwargs: DictConfig):+ """Main hydra.++ Args:+ kwargs: Additional keyword arguments.+ """+ if kwargs.get("debug", False):+ import pdb++ pdb.set_trace()++ assert "model" in kwargs+ if "model_conf" not in kwargs:+ logging.info("download models from model hub: {}".format(kwargs.get("hub", "ms")))+ kwargs = download_model(is_training=kwargs.get("is_training", True), **kwargs)++ main(**kwargs)+</code></pre></details></div>+<div class="api-detail" id="e444"><span class="dtag badge-f">function</span><h2><code>main(**kwargs)</code></h2><div class="dmod">funasr.bin.train · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/bin/train.py#L61" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Main.</p>+<h4>Args:</h4>+<ul>+<li><code>**kwargs</code> — Additional keyword arguments.</li>+</ul>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code>def main(**kwargs):++ # set random seed+ """Main.++ Args:+ **kwargs: Additional keyword arguments.+ """+ set_all_random_seed(kwargs.get("seed", 0))+ torch.backends.cudnn.enabled = kwargs.get("cudnn_enabled", torch.backends.cudnn.enabled)+ torch.backends.cudnn.benchmark = kwargs.get("cudnn_benchmark", torch.backends.cudnn.benchmark)+ torch.backends.cudnn.deterministic = kwargs.get("cudnn_deterministic", True)+ # open tf32+ torch.backends.cuda.matmul.allow_tf32 = kwargs.get("enable_tf32", True)++ local_rank = int(os.environ.get("LOCAL_RANK", 0))+ if local_rank == 0:+ tables.print()+ # Check if we are using DDP or FSDP+ use_ddp = "WORLD_SIZE" in os.environ and int(os.environ["WORLD_SIZE"]) > 1+ use_fsdp = kwargs.get("use_fsdp", False)+ # use_ddp = False if use_fsdp else use_fsdp+ if use_ddp or use_fsdp:+ dist.init_process_group(backend=kwargs.get("backend", "nccl"), init_method="env://")+ torch.cuda.set_device(local_rank)++ logging.info("Build model, frontend, tokenizer")+ device = kwargs.get("device", "cuda")+ kwargs["device"] = "cpu"+ model = AutoModel(**kwargs)+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/bin/train.py#L61" target="_blank" class="src-more">View full source on GitHub →</a></details></div>+<div class="api-detail" id="e445"><span class="dtag badge-f">function</span><h2><code>main_hydra(kwargs)</code></h2><div class="dmod">funasr.bin.train_ds · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/bin/train_ds.py#L45" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Main hydra.</p>+<h4>Args:</h4>+<ul>+<li><code>kwargs</code> — Additional keyword arguments.</li>+</ul>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code>def main_hydra(kwargs: DictConfig):+ """Main hydra.++ Args:+ kwargs: Additional keyword arguments.+ """+ if kwargs.get("debug", False):+ import pdb++ pdb.set_trace()++ assert "model" in kwargs+ if "model_conf" not in kwargs:+ logging.info("download models from model hub: {}".format(kwargs.get("hub", "ms")))+ kwargs = download_model(is_training=kwargs.get("is_training", True), **kwargs)++ main(**kwargs)+</code></pre></details></div>+<div class="api-detail" id="e446"><span class="dtag badge-f">function</span><h2><code>main(**kwargs)</code></h2><div class="dmod">funasr.bin.train_ds · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/bin/train_ds.py#L64" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Main.</p>+<h4>Args:</h4>+<ul>+<li><code>**kwargs</code> — Additional keyword arguments.</li>+</ul>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code>def main(**kwargs):++ # set random seed+ """Main.++ Args:+ **kwargs: Additional keyword arguments.+ """+ set_all_random_seed(kwargs.get("seed", 0))+ torch.backends.cudnn.enabled = kwargs.get("cudnn_enabled", torch.backends.cudnn.enabled)+ torch.backends.cudnn.benchmark = kwargs.get("cudnn_benchmark", torch.backends.cudnn.benchmark)+ torch.backends.cudnn.deterministic = kwargs.get("cudnn_deterministic", True)+ # open tf32+ torch.backends.cuda.matmul.allow_tf32 = kwargs.get("enable_tf32", True)++ rank = int(os.environ.get("RANK", 0))+ local_rank = int(os.environ.get("LOCAL_RANK", 0))+ world_size = int(os.environ.get("WORLD_SIZE", 1))++ if local_rank == 0:+ tables.print()++ use_ddp = world_size > 1+ use_fsdp = kwargs.get("use_fsdp", False)+ use_deepspeed = kwargs.get("use_deepspeed", False)+ if use_deepspeed:+ logging.info(f"use_deepspeed: {use_deepspeed}")+ deepspeed.init_distributed(dist_backend=kwargs.get("backend", "nccl"))+ elif use_ddp or use_fsdp:+ logging.info(f"use_ddp: {use_ddp}, use_fsdp: {use_fsdp}")+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/bin/train_ds.py#L64" target="_blank" class="src-more">View full source on GitHub →</a></details></div>+<div class="api-detail" id="e447"><span class="dtag badge-f">function</span><h2><code>download_dataset()</code></h2><div class="dmod">funasr.download.download_dataset_from_hub · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/download/download_dataset_from_hub.py#L1" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Download dataset.</p>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code>def download_dataset():+ """Download dataset."""+ pass+</code></pre></details></div>+<div class="api-detail" id="e448"><span class="dtag badge-f">function</span><h2><code>download_dataset_from_ms(**kwargs)</code></h2><div class="dmod">funasr.download.download_dataset_from_hub · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/download/download_dataset_from_hub.py#L6" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Download dataset from ms.</p>+<h4>Args:</h4>+<ul>+<li><code>**kwargs</code> — Additional keyword arguments.</li>+</ul>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code>def download_dataset_from_ms(**kwargs):+ """Download dataset from ms.++ Args:+ **kwargs: Additional keyword arguments.+ """+ from modelscope.msdatasets import MsDataset++ dataset_name = kwargs.get("dataset_name", "speech_asr/speech_asr_aishell1_trainsets")+ subset_name = kwargs.get("subset_name", "default")+ split = kwargs.get("split", "train")+ data_dump_dir = kwargs.get("data_dump_dir", None)+ ds = MsDataset.load(+ dataset_name=dataset_name, subset_name=subset_name, split=split, cache_dir=data_dump_dir+ )+</code></pre></details></div>+<div class="api-detail" id="e449"><span class="dtag badge-f">function</span><h2><code>download_model(**kwargs)</code></h2><div class="dmod">funasr.download.download_model_from_hub · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/download/download_model_from_hub.py#L9" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Download model from hub and parse its configuration.</p>+<p>Resolves model name aliases, downloads from ModelScope or HuggingFace,</p>+<p>reads config.yaml and configuration.json, and returns complete kwargs</p>+<p>for model instantiation.</p>+<h4>Args:</h4>+<ul>+<li><code>**kwargs</code> — Must include 'model' (str). Optional: 'hub', 'model_revision',</li>+<li class="sub">'is_training', etc.</li>+</ul>+<h4>Returns:</h4>+<ul>+<li><code>dict</code> — Complete kwargs with resolved paths, model class name, and config.</li>+</ul>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code>def download_model(**kwargs):++ """Download model from hub and parse its configuration.++ Resolves model name aliases, downloads from ModelScope or HuggingFace,+ reads config.yaml and configuration.json, and returns complete kwargs+ for model instantiation.++ Args:+ **kwargs: Must include 'model' (str). Optional: 'hub', 'model_revision',+ 'is_training', etc.++ Returns:+ dict: Complete kwargs with resolved paths, model class name, and config.+ """+ hub = kwargs.get("hub", "ms")+ if hub == "ms" or hub == "modelscope":+ kwargs = download_from_ms(**kwargs)+ elif hub == "hf" or hub == "huggingface":+ kwargs = download_from_hf(**kwargs)+ elif hub == "openai":+ model_or_path = kwargs.get("model")+ if os.path.exists(model_or_path):+ # local path+ kwargs["model_path"] = model_or_path+ kwargs["model"] = "WhisperWarp"+ else:+ # model name+ if model_or_path in name_maps_openai:+ model_or_path = name_maps_openai[model_or_path]+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/download/download_model_from_hub.py#L9" target="_blank" class="src-more">View full source on GitHub →</a></details></div>+<div class="api-detail" id="e450"><span class="dtag badge-f">function</span><h2><code>download_from_ms(**kwargs)</code></h2><div class="dmod">funasr.download.download_model_from_hub · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/download/download_model_from_hub.py#L44" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Download from ms.</p>+<h4>Args:</h4>+<ul>+<li><code>**kwargs</code> — Additional keyword arguments.</li>+</ul>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code>def download_from_ms(**kwargs):+ """Download from ms.++ Args:+ **kwargs: Additional keyword arguments.+ """+ model_or_path = kwargs.get("model")+ if model_or_path in name_maps_ms:+ model_or_path = name_maps_ms[model_or_path]+ model_revision = kwargs.get("model_revision", "master")+ if not os.path.exists(model_or_path) and "model_path" not in kwargs:+ try:+ model_or_path = get_or_download_model_dir(+ model_or_path,+ model_revision,+ is_training=kwargs.get("is_training"),+ check_latest=kwargs.get("check_latest", True),+ )+ except Exception as e:+ print(f"Download: {model_or_path} failed!: {e}")++ kwargs["model_path"] = model_or_path if "model_path" not in kwargs else kwargs["model_path"]+ model_or_path = kwargs["model_path"]+ if os.path.exists(os.path.join(model_or_path, "configuration.json")):+ with open(os.path.join(model_or_path, "configuration.json"), "r", encoding="utf-8") as f:+ conf_json = json.load(f)++ cfg = {}+ if "file_path_metas" in conf_json:+ add_file_root_path(model_or_path, conf_json["file_path_metas"], cfg)+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/download/download_model_from_hub.py#L44" target="_blank" class="src-more">View full source on GitHub →</a></details></div>+<div class="api-detail" id="e451"><span class="dtag badge-f">function</span><h2><code>download_from_hf(**kwargs)</code></h2><div class="dmod">funasr.download.download_model_from_hub · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/download/download_model_from_hub.py#L122" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Download from hf.</p>+<h4>Args:</h4>+<ul>+<li><code>**kwargs</code> — Additional keyword arguments.</li>+</ul>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code>def download_from_hf(**kwargs):+ """Download from hf.++ Args:+ **kwargs: Additional keyword arguments.+ """+ model_or_path = kwargs.get("model")+ if model_or_path in name_maps_hf:+ model_or_path = name_maps_hf[model_or_path]+ model_revision = kwargs.get("model_revision", "master")+ if not os.path.exists(model_or_path) and "model_path" not in kwargs:+ try:+ model_or_path = get_or_download_model_dir_hf(+ model_or_path,+ model_revision,+ is_training=kwargs.get("is_training"),+ check_latest=kwargs.get("check_latest", True),+ )+ except Exception as e:+ print(f"Download: {model_or_path} failed!: {e}")++ kwargs["model_path"] = model_or_path if "model_path" not in kwargs else kwargs["model_path"]++ if os.path.exists(os.path.join(model_or_path, "configuration.json")):+ with open(os.path.join(model_or_path, "configuration.json"), "r", encoding="utf-8") as f:+ conf_json = json.load(f)++ cfg = {}+ if "file_path_metas" in conf_json:+ add_file_root_path(model_or_path, conf_json["file_path_metas"], cfg)+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/download/download_model_from_hub.py#L122" target="_blank" class="src-more">View full source on GitHub →</a></details></div>+<div class="api-detail" id="e452"><span class="dtag badge-f">function</span><h2><code>add_file_root_path(model_or_path, file_path_metas, cfg)</code></h2><div class="dmod">funasr.download.download_model_from_hub · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/download/download_model_from_hub.py#L191" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Add file root path.</p>+<h4>Args:</h4>+<ul>+<li><code>model_or_path</code> — TODO.</li>+<li><code>file_path_metas</code> — TODO.</li>+<li><code>cfg</code> — Configuration overrides.</li>+</ul>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code>def add_file_root_path(model_or_path: str, file_path_metas: dict, cfg={}):++ """Add file root path.++ Args:+ model_or_path: TODO.+ file_path_metas: TODO.+ cfg: Configuration overrides.+ """+ if isinstance(file_path_metas, dict):+ if isinstance(cfg, list):+ cfg.append({})++ for k, v in file_path_metas.items():+ if isinstance(v, str):+ p = os.path.join(model_or_path, v)+ if os.path.exists(p):+ if isinstance(cfg, dict):+ cfg[k] = p+ elif isinstance(cfg, list):+ # if len(cfg) == 0:+ # cfg.append({})+ cfg[-1][k] = p++ elif isinstance(v, dict):+ if isinstance(cfg, dict):+ if k not in cfg:+ cfg[k] = {}+ add_file_root_path(model_or_path, v, cfg[k])+ # elif isinstance(cfg, list):+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/download/download_model_from_hub.py#L191" target="_blank" class="src-more">View full source on GitHub →</a></details></div>+<div class="api-detail" id="e453"><span class="dtag badge-f">function</span><h2><code>get_or_download_model_dir(model, model_revision, is_training, check_latest)</code></h2><div class="dmod">funasr.download.download_model_from_hub · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/download/download_model_from_hub.py#L241" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Get local model directory or download model if necessary.</p>+<h4>Args:</h4>+<ul>+<li><code>model (str)</code> — model id or path to local model directory.</li>+<li><code>model_revision (str, optional)</code> — model version number.</li>+</ul>+<p>:param is_training:</p>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code>def get_or_download_model_dir(+ model,+ model_revision=None,+ is_training=False,+ check_latest=True,+):+ """Get local model directory or download model if necessary.++ Args:+ model (str): model id or path to local model directory.+ model_revision (str, optional): model version number.+ :param is_training:+ """+ from modelscope.hub.check_model import check_local_model_is_latest+ from modelscope.hub.snapshot_download import snapshot_download++ from modelscope.utils.constant import Invoke, ThirdParty++ key = Invoke.LOCAL_TRAINER if is_training else Invoke.PIPELINE++ if os.path.exists(model) and check_latest:+ model_cache_dir = model if os.path.isdir(model) else os.path.dirname(model)+ try:+ check_local_model_is_latest(+ model_cache_dir, user_agent={Invoke.KEY: key, ThirdParty.KEY: "funasr"}+ )+ except:+ print("could not check the latest version")+ else:+ model_cache_dir = snapshot_download(+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/download/download_model_from_hub.py#L241" target="_blank" class="src-more">View full source on GitHub →</a></details></div>+<div class="api-detail" id="e454"><span class="dtag badge-f">function</span><h2><code>get_or_download_model_dir_hf(model, model_revision, is_training, check_latest)</code></h2><div class="dmod">funasr.download.download_model_from_hub · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/download/download_model_from_hub.py#L276" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Get local model directory or download model if necessary.</p>+<h4>Args:</h4>+<ul>+<li><code>model (str)</code> — model id or path to local model directory.</li>+<li><code>model_revision (str, optional)</code> — model version number.</li>+</ul>+<p>:param is_training:</p>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code>def get_or_download_model_dir_hf(+ model,+ model_revision=None,+ is_training=False,+ check_latest=True,+):+ """Get local model directory or download model if necessary.++ Args:+ model (str): model id or path to local model directory.+ model_revision (str, optional): model version number.+ :param is_training:+ """+ from huggingface_hub import snapshot_download++ model_cache_dir = snapshot_download(model)+ return model_cache_dir+</code></pre></details></div>+<div class="api-detail" id="e455"><span class="dtag badge-f">function</span><h2><code>download_from_url(url)</code></h2><div class="dmod">funasr.download.file · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/download/file.py#L14" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Download from url.</p>+<h4>Args:</h4>+<ul>+<li><code>url</code> — TODO.</li>+</ul>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code>def download_from_url(url):+ """Download from url.++ Args:+ url: TODO.+ """+ result = urlparse(url)+ file_path = None+ if result.scheme is not None and len(result.scheme) > 0:+ storage = HTTPStorage()+ # bytes+ data = storage.read(url)+ work_dir = tempfile.TemporaryDirectory().name+ if not os.path.exists(work_dir):+ os.makedirs(work_dir)+ file_path = os.path.join(work_dir, os.path.basename(url))+ with open(file_path, "wb") as fb:+ fb.write(data)+ assert file_path is not None, f"failed to download: {url}"+ return file_path+</code></pre></details></div>+<div class="api-detail" id="e456"><span class="dtag badge-c">class</span><h2>Storage</h2><div class="dmod">funasr.download.file · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/download/file.py#L36" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Abstract class of storage.</p>+<p>All backends need to implement two apis: ``read()`` and ``read_text()``.</p>+<p>``read()`` reads the file as a byte stream and ``read_text()`` reads</p>+<p>the file as texts.</p>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code>class Storage(metaclass=ABCMeta):+ """Abstract class of storage.++ All backends need to implement two apis: ``read()`` and ``read_text()``.+ ``read()`` reads the file as a byte stream and ``read_text()`` reads+ the file as texts.+ """++ @abstractmethod+ def read(self, filepath: str):+ """Read.++ Args:+ filepath: TODO.+ """+ pass++ @abstractmethod+ def read_text(self, filepath: str):+ """Read text.++ Args:+ filepath: TODO.+ """+ pass++ @abstractmethod+ def write(self, obj: bytes, filepath: Union[str, Path]) -> None:+ """Write.++</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/download/file.py#L36" target="_blank" class="src-more">View full source on GitHub →</a></details><h3 class="mt">Methods</h3><div class="mblk"><code>.read(filepath)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/download/file.py#L45" target="_blank" class="gh-link">L45</a><div class="mdoc"><p>Read.</p>+<h4>Args:</h4>+<ul>+<li><code>filepath</code> — TODO.</li>+</ul>+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def read(self, filepath: str):+ """Read.++ Args:+ filepath: TODO.+ """+ pass+</code></pre></details></div><div class="mblk"><code>.read_text(filepath)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/download/file.py#L54" target="_blank" class="gh-link">L54</a><div class="mdoc"><p>Read text.</p>+<h4>Args:</h4>+<ul>+<li><code>filepath</code> — TODO.</li>+</ul>+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def read_text(self, filepath: str):+ """Read text.++ Args:+ filepath: TODO.+ """+ pass+</code></pre></details></div><div class="mblk"><code>.write(obj, filepath)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/download/file.py#L63" target="_blank" class="gh-link">L63</a><div class="mdoc"><p>Write.</p>+<h4>Args:</h4>+<ul>+<li><code>obj</code> — TODO.</li>+<li><code>filepath</code> — TODO.</li>+</ul>+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def write(self, obj: bytes, filepath: Union[str, Path]) -> None:+ """Write.++ Args:+ obj: TODO.+ filepath: TODO.+ """+ pass+</code></pre></details></div><div class="mblk"><code>.write_text(obj, filepath, encoding)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/download/file.py#L73" target="_blank" class="gh-link">L73</a><div class="mdoc"><p>Write text.</p>+<h4>Args:</h4>+<ul>+<li><code>obj</code> — TODO.</li>+<li><code>filepath</code> — TODO.</li>+<li><code>encoding</code> — TODO.</li>+</ul>+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def write_text(self, obj: str, filepath: Union[str, Path], encoding: str = "utf-8") -> None:+ """Write text.++ Args:+ obj: TODO.+ filepath: TODO.+ encoding: TODO.+ """+ pass+</code></pre></details></div></div>+<div class="api-detail" id="e457"><span class="dtag badge-m">method</span><h2><code>Storage.read(filepath)</code></h2><div class="dmod">funasr.download.file.Storage · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/download/file.py#L45" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Read.</p>+<h4>Args:</h4>+<ul>+<li><code>filepath</code> — TODO.</li>+</ul>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def read(self, filepath: str):+ """Read.++ Args:+ filepath: TODO.+ """+ pass+</code></pre></details></div>+<div class="api-detail" id="e458"><span class="dtag badge-m">method</span><h2><code>Storage.read_text(filepath)</code></h2><div class="dmod">funasr.download.file.Storage · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/download/file.py#L54" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Read text.</p>+<h4>Args:</h4>+<ul>+<li><code>filepath</code> — TODO.</li>+</ul>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def read_text(self, filepath: str):+ """Read text.++ Args:+ filepath: TODO.+ """+ pass+</code></pre></details></div>+<div class="api-detail" id="e459"><span class="dtag badge-m">method</span><h2><code>Storage.write(obj, filepath)</code></h2><div class="dmod">funasr.download.file.Storage · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/download/file.py#L63" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Write.</p>+<h4>Args:</h4>+<ul>+<li><code>obj</code> — TODO.</li>+<li><code>filepath</code> — TODO.</li>+</ul>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def write(self, obj: bytes, filepath: Union[str, Path]) -> None:+ """Write.++ Args:+ obj: TODO.+ filepath: TODO.+ """+ pass+</code></pre></details></div>+<div class="api-detail" id="e460"><span class="dtag badge-m">method</span><h2><code>Storage.write_text(obj, filepath, encoding)</code></h2><div class="dmod">funasr.download.file.Storage · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/download/file.py#L73" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Write text.</p>+<h4>Args:</h4>+<ul>+<li><code>obj</code> — TODO.</li>+<li><code>filepath</code> — TODO.</li>+<li><code>encoding</code> — TODO.</li>+</ul>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def write_text(self, obj: str, filepath: Union[str, Path], encoding: str = "utf-8") -> None:+ """Write text.++ Args:+ obj: TODO.+ filepath: TODO.+ encoding: TODO.+ """+ pass+</code></pre></details></div>+<div class="api-detail" id="e461"><span class="dtag badge-c">class</span><h2>LocalStorage</h2><div class="dmod">funasr.download.file · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/download/file.py#L84" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Local hard disk storage</p>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code>class LocalStorage(Storage):+ """Local hard disk storage"""++ def read(self, filepath: Union[str, Path]) -> bytes:+ """Read data from a given ``filepath`` with 'rb' mode.++ Args:+ filepath (str or Path): Path to read data.++ Returns:+ bytes: Expected bytes object.+ """+ with open(filepath, "rb") as f:+ content = f.read()+ return content++ def read_text(self, filepath: Union[str, Path], encoding: str = "utf-8") -> str:+ """Read data from a given ``filepath`` with 'r' mode.++ Args:+ filepath (str or Path): Path to read data.+ encoding (str): The encoding format used to open the ``filepath``.+ Default: 'utf-8'.++ Returns:+ str: Expected text reading from ``filepath``.+ """+ with open(filepath, "r", encoding=encoding) as f:+ value_buf = f.read()+ return value_buf+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/download/file.py#L84" target="_blank" class="src-more">View full source on GitHub →</a></details><h3 class="mt">Methods</h3><div class="mblk"><code>.read(filepath)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/download/file.py#L87" target="_blank" class="gh-link">L87</a><div class="mdoc"><p>Read data from a given ``filepath`` with 'rb' mode.</p>+<h4>Args:</h4>+<ul>+<li><code>filepath (str or Path)</code> — Path to read data.</li>+</ul>+<h4>Returns:</h4>+<ul>+<li><code>bytes</code> — Expected bytes object.</li>+</ul>+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def read(self, filepath: Union[str, Path]) -> bytes:+ """Read data from a given ``filepath`` with 'rb' mode.++ Args:+ filepath (str or Path): Path to read data.++ Returns:+ bytes: Expected bytes object.+ """+ with open(filepath, "rb") as f:+ content = f.read()+ return content+</code></pre></details></div><div class="mblk"><code>.read_text(filepath, encoding)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/download/file.py#L100" target="_blank" class="gh-link">L100</a><div class="mdoc"><p>Read data from a given ``filepath`` with 'r' mode.</p>+<h4>Args:</h4>+<ul>+<li><code>filepath (str or Path)</code> — Path to read data.</li>+<li><code>encoding (str)</code> — The encoding format used to open the ``filepath``.</li>+<li><code>Default</code> — 'utf-8'.</li>+</ul>+<h4>Returns:</h4>+<ul>+<li><code>str</code> — Expected text reading from ``filepath``.</li>+</ul>+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def read_text(self, filepath: Union[str, Path], encoding: str = "utf-8") -> str:+ """Read data from a given ``filepath`` with 'r' mode.++ Args:+ filepath (str or Path): Path to read data.+ encoding (str): The encoding format used to open the ``filepath``.+ Default: 'utf-8'.++ Returns:+ str: Expected text reading from ``filepath``.+ """+ with open(filepath, "r", encoding=encoding) as f:+ value_buf = f.read()+ return value_buf+</code></pre></details></div><div class="mblk"><code>.write(obj, filepath)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/download/file.py#L115" target="_blank" class="gh-link">L115</a><div class="mdoc"><p>Write data to a given ``filepath`` with 'wb' mode.</p>+<h4>Note:</h4>+<p>``write`` will create a directory if the directory of ``filepath``</p>+<p>does not exist.</p>+<h4>Args:</h4>+<ul>+<li><code>obj (bytes)</code> — Data to be written.</li>+<li><code>filepath (str or Path)</code> — Path to write data.</li>+</ul>+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def write(self, obj: bytes, filepath: Union[str, Path]) -> None:+ """Write data to a given ``filepath`` with 'wb' mode.++ Note:+ ``write`` will create a directory if the directory of ``filepath``+ does not exist.++ Args:+ obj (bytes): Data to be written.+ filepath (str or Path): Path to write data.+ """+ dirname = os.path.dirname(filepath)+ if dirname and not os.path.exists(dirname):+ os.makedirs(dirname, exist_ok=True)++ with open(filepath, "wb") as f:+ f.write(obj)+</code></pre></details></div><div class="mblk"><code>.write_text(obj, filepath, encoding)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/download/file.py#L133" target="_blank" class="gh-link">L133</a><div class="mdoc"><p>Write data to a given ``filepath`` with 'w' mode.</p>+<h4>Note:</h4>+<p>``write_text`` will create a directory if the directory of</p>+<p>``filepath`` does not exist.</p>+<h4>Args:</h4>+<ul>+<li><code>obj (str)</code> — Data to be written.</li>+<li><code>filepath (str or Path)</code> — Path to write data.</li>+<li><code>encoding (str)</code> — The encoding format used to open the ``filepath``.</li>+<li><code>Default</code> — 'utf-8'.</li>+</ul>+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def write_text(self, obj: str, filepath: Union[str, Path], encoding: str = "utf-8") -> None:+ """Write data to a given ``filepath`` with 'w' mode.++ Note:+ ``write_text`` will create a directory if the directory of+ ``filepath`` does not exist.++ Args:+ obj (str): Data to be written.+ filepath (str or Path): Path to write data.+ encoding (str): The encoding format used to open the ``filepath``.+ Default: 'utf-8'.+ """+ dirname = os.path.dirname(filepath)+ if dirname and not os.path.exists(dirname):+ os.makedirs(dirname, exist_ok=True)++ with open(filepath, "w", encoding=encoding) as f:+ f.write(obj)+</code></pre></details></div><div class="mblk"><code>.as_local_path(filepath)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/download/file.py#L154" target="_blank" class="gh-link">L154</a><div class="mdoc"><p>Only for unified API and do nothing.</p>+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def as_local_path(self, filepath: Union[str, Path]) -> Generator[Union[str, Path], None, None]:+ """Only for unified API and do nothing."""+ yield filepath+</code></pre></details></div></div>+<div class="api-detail" id="e462"><span class="dtag badge-m">method</span><h2><code>LocalStorage.read(filepath)</code></h2><div class="dmod">funasr.download.file.LocalStorage · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/download/file.py#L87" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Read data from a given ``filepath`` with 'rb' mode.</p>+<h4>Args:</h4>+<ul>+<li><code>filepath (str or Path)</code> — Path to read data.</li>+</ul>+<h4>Returns:</h4>+<ul>+<li><code>bytes</code> — Expected bytes object.</li>+</ul>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def read(self, filepath: Union[str, Path]) -> bytes:+ """Read data from a given ``filepath`` with 'rb' mode.++ Args:+ filepath (str or Path): Path to read data.++ Returns:+ bytes: Expected bytes object.+ """+ with open(filepath, "rb") as f:+ content = f.read()+ return content+</code></pre></details></div>+<div class="api-detail" id="e463"><span class="dtag badge-m">method</span><h2><code>LocalStorage.read_text(filepath, encoding)</code></h2><div class="dmod">funasr.download.file.LocalStorage · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/download/file.py#L100" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Read data from a given ``filepath`` with 'r' mode.</p>+<h4>Args:</h4>+<ul>+<li><code>filepath (str or Path)</code> — Path to read data.</li>+<li><code>encoding (str)</code> — The encoding format used to open the ``filepath``.</li>+<li><code>Default</code> — 'utf-8'.</li>+</ul>+<h4>Returns:</h4>+<ul>+<li><code>str</code> — Expected text reading from ``filepath``.</li>+</ul>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def read_text(self, filepath: Union[str, Path], encoding: str = "utf-8") -> str:+ """Read data from a given ``filepath`` with 'r' mode.++ Args:+ filepath (str or Path): Path to read data.+ encoding (str): The encoding format used to open the ``filepath``.+ Default: 'utf-8'.++ Returns:+ str: Expected text reading from ``filepath``.+ """+ with open(filepath, "r", encoding=encoding) as f:+ value_buf = f.read()+ return value_buf+</code></pre></details></div>+<div class="api-detail" id="e464"><span class="dtag badge-m">method</span><h2><code>LocalStorage.write(obj, filepath)</code></h2><div class="dmod">funasr.download.file.LocalStorage · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/download/file.py#L115" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Write data to a given ``filepath`` with 'wb' mode.</p>+<h4>Note:</h4>+<p>``write`` will create a directory if the directory of ``filepath``</p>+<p>does not exist.</p>+<h4>Args:</h4>+<ul>+<li><code>obj (bytes)</code> — Data to be written.</li>+<li><code>filepath (str or Path)</code> — Path to write data.</li>+</ul>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def write(self, obj: bytes, filepath: Union[str, Path]) -> None:+ """Write data to a given ``filepath`` with 'wb' mode.++ Note:+ ``write`` will create a directory if the directory of ``filepath``+ does not exist.++ Args:+ obj (bytes): Data to be written.+ filepath (str or Path): Path to write data.+ """+ dirname = os.path.dirname(filepath)+ if dirname and not os.path.exists(dirname):+ os.makedirs(dirname, exist_ok=True)++ with open(filepath, "wb") as f:+ f.write(obj)+</code></pre></details></div>+<div class="api-detail" id="e465"><span class="dtag badge-m">method</span><h2><code>LocalStorage.write_text(obj, filepath, encoding)</code></h2><div class="dmod">funasr.download.file.LocalStorage · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/download/file.py#L133" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Write data to a given ``filepath`` with 'w' mode.</p>+<h4>Note:</h4>+<p>``write_text`` will create a directory if the directory of</p>+<p>``filepath`` does not exist.</p>+<h4>Args:</h4>+<ul>+<li><code>obj (str)</code> — Data to be written.</li>+<li><code>filepath (str or Path)</code> — Path to write data.</li>+<li><code>encoding (str)</code> — The encoding format used to open the ``filepath``.</li>+<li><code>Default</code> — 'utf-8'.</li>+</ul>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def write_text(self, obj: str, filepath: Union[str, Path], encoding: str = "utf-8") -> None:+ """Write data to a given ``filepath`` with 'w' mode.++ Note:+ ``write_text`` will create a directory if the directory of+ ``filepath`` does not exist.++ Args:+ obj (str): Data to be written.+ filepath (str or Path): Path to write data.+ encoding (str): The encoding format used to open the ``filepath``.+ Default: 'utf-8'.+ """+ dirname = os.path.dirname(filepath)+ if dirname and not os.path.exists(dirname):+ os.makedirs(dirname, exist_ok=True)++ with open(filepath, "w", encoding=encoding) as f:+ f.write(obj)+</code></pre></details></div>+<div class="api-detail" id="e466"><span class="dtag badge-m">method</span><h2><code>LocalStorage.as_local_path(filepath)</code></h2><div class="dmod">funasr.download.file.LocalStorage · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/download/file.py#L154" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Only for unified API and do nothing.</p>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def as_local_path(self, filepath: Union[str, Path]) -> Generator[Union[str, Path], None, None]:+ """Only for unified API and do nothing."""+ yield filepath+</code></pre></details></div>+<div class="api-detail" id="e467"><span class="dtag badge-c">class</span><h2>HTTPStorage</h2><div class="dmod">funasr.download.file · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/download/file.py#L159" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>HTTP and HTTPS storage.</p>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code>class HTTPStorage(Storage):+ """HTTP and HTTPS storage."""++ def read(self, url):+ # TODO @wenmeng.zwm add progress bar if file is too large+ """Read.++ Args:+ url: TODO.+ """+ r = requests.get(url)+ r.raise_for_status()+ return r.content++ def read_text(self, url):+ """Read text.++ Args:+ url: TODO.+ """+ r = requests.get(url)+ r.raise_for_status()+ return r.text++ @contextlib.contextmanager+ def as_local_path(self, filepath: str) -> Generator[Union[str, Path], None, None]:+ """Download a file from ``filepath``.++ ``as_local_path`` is decorated by :meth:`contextlib.contextmanager`. It+ can be called with ``with`` statement, and when exists from the+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/download/file.py#L159" target="_blank" class="src-more">View full source on GitHub →</a></details><h3 class="mt">Methods</h3><div class="mblk"><code>.read(url)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/download/file.py#L162" target="_blank" class="gh-link">L162</a><div class="mdoc"><p>Read.</p>+<h4>Args:</h4>+<ul>+<li><code>url</code> — TODO.</li>+</ul>+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def read(self, url):+ # TODO @wenmeng.zwm add progress bar if file is too large+ """Read.++ Args:+ url: TODO.+ """+ r = requests.get(url)+ r.raise_for_status()+ return r.content+</code></pre></details></div><div class="mblk"><code>.read_text(url)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/download/file.py#L173" target="_blank" class="gh-link">L173</a><div class="mdoc"><p>Read text.</p>+<h4>Args:</h4>+<ul>+<li><code>url</code> — TODO.</li>+</ul>+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def read_text(self, url):+ """Read text.++ Args:+ url: TODO.+ """+ r = requests.get(url)+ r.raise_for_status()+ return r.text+</code></pre></details></div><div class="mblk"><code>.as_local_path(filepath)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/download/file.py#L184" target="_blank" class="gh-link">L184</a><div class="mdoc"><p>Download a file from ``filepath``.</p>+<p>``as_local_path`` is decorated by :meth:`contextlib.contextmanager`. It</p>+<p>can be called with ``with`` statement, and when exists from the</p>+<p>``with`` statement, the temporary path will be released.</p>+<h4>Args:</h4>+<ul>+<li><code>filepath (str)</code> — Download a file from ``filepath``.</li>+</ul>+<h4>Examples:</h4>+<p>>>> storage = HTTPStorage()</p>+<p>>>> # After existing from the ``with`` clause,</p>+<p>>>> # the path will be removed</p>+<p>>>> with storage.get_local_path('http://path/to/file') as path:</p>+<p>... # do something here</p>+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def as_local_path(self, filepath: str) -> Generator[Union[str, Path], None, None]:+ """Download a file from ``filepath``.++ ``as_local_path`` is decorated by :meth:`contextlib.contextmanager`. It+ can be called with ``with`` statement, and when exists from the+ ``with`` statement, the temporary path will be released.++ Args:+ filepath (str): Download a file from ``filepath``.++ Examples:+ >>> storage = HTTPStorage()+ >>> # After existing from the ``with`` clause,+ >>> # the path will be removed+ >>> with storage.get_local_path('http://path/to/file') as path:+ ... # do something here+ """+ try:+ f = tempfile.NamedTemporaryFile(delete=False)+ f.write(self.read(filepath))+ f.close()+ yield f.name+ finally:+ os.remove(f.name)+</code></pre></details></div><div class="mblk"><code>.write(obj, url)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/download/file.py#L209" target="_blank" class="gh-link">L209</a><div class="mdoc"><p>Write.</p>+<h4>Args:</h4>+<ul>+<li><code>obj</code> — TODO.</li>+<li><code>url</code> — TODO.</li>+</ul>+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def write(self, obj: bytes, url: Union[str, Path]) -> None:+ """Write.++ Args:+ obj: TODO.+ url: TODO.+ """+ raise NotImplementedError("write is not supported by HTTP Storage")+</code></pre></details></div><div class="mblk"><code>.write_text(obj, url, encoding)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/download/file.py#L218" target="_blank" class="gh-link">L218</a><div class="mdoc"><p>Write text.</p>+<h4>Args:</h4>+<ul>+<li><code>obj</code> — TODO.</li>+<li><code>url</code> — TODO.</li>+<li><code>encoding</code> — TODO.</li>+</ul>+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def write_text(self, obj: str, url: Union[str, Path], encoding: str = "utf-8") -> None:+ """Write text.++ Args:+ obj: TODO.+ url: TODO.+ encoding: TODO.+ """+ raise NotImplementedError("write_text is not supported by HTTP Storage")+</code></pre></details></div></div>+<div class="api-detail" id="e468"><span class="dtag badge-m">method</span><h2><code>HTTPStorage.read(url)</code></h2><div class="dmod">funasr.download.file.HTTPStorage · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/download/file.py#L162" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Read.</p>+<h4>Args:</h4>+<ul>+<li><code>url</code> — TODO.</li>+</ul>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def read(self, url):+ # TODO @wenmeng.zwm add progress bar if file is too large+ """Read.++ Args:+ url: TODO.+ """+ r = requests.get(url)+ r.raise_for_status()+ return r.content+</code></pre></details></div>+<div class="api-detail" id="e469"><span class="dtag badge-m">method</span><h2><code>HTTPStorage.read_text(url)</code></h2><div class="dmod">funasr.download.file.HTTPStorage · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/download/file.py#L173" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Read text.</p>+<h4>Args:</h4>+<ul>+<li><code>url</code> — TODO.</li>+</ul>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def read_text(self, url):+ """Read text.++ Args:+ url: TODO.+ """+ r = requests.get(url)+ r.raise_for_status()+ return r.text+</code></pre></details></div>+<div class="api-detail" id="e470"><span class="dtag badge-m">method</span><h2><code>HTTPStorage.as_local_path(filepath)</code></h2><div class="dmod">funasr.download.file.HTTPStorage · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/download/file.py#L184" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Download a file from ``filepath``.</p>+<p>``as_local_path`` is decorated by :meth:`contextlib.contextmanager`. It</p>+<p>can be called with ``with`` statement, and when exists from the</p>+<p>``with`` statement, the temporary path will be released.</p>+<h4>Args:</h4>+<ul>+<li><code>filepath (str)</code> — Download a file from ``filepath``.</li>+</ul>+<h4>Examples:</h4>+<p>>>> storage = HTTPStorage()</p>+<p>>>> # After existing from the ``with`` clause,</p>+<p>>>> # the path will be removed</p>+<p>>>> with storage.get_local_path('http://path/to/file') as path:</p>+<p>... # do something here</p>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def as_local_path(self, filepath: str) -> Generator[Union[str, Path], None, None]:+ """Download a file from ``filepath``.++ ``as_local_path`` is decorated by :meth:`contextlib.contextmanager`. It+ can be called with ``with`` statement, and when exists from the+ ``with`` statement, the temporary path will be released.++ Args:+ filepath (str): Download a file from ``filepath``.++ Examples:+ >>> storage = HTTPStorage()+ >>> # After existing from the ``with`` clause,+ >>> # the path will be removed+ >>> with storage.get_local_path('http://path/to/file') as path:+ ... # do something here+ """+ try:+ f = tempfile.NamedTemporaryFile(delete=False)+ f.write(self.read(filepath))+ f.close()+ yield f.name+ finally:+ os.remove(f.name)+</code></pre></details></div>+<div class="api-detail" id="e471"><span class="dtag badge-m">method</span><h2><code>HTTPStorage.write(obj, url)</code></h2><div class="dmod">funasr.download.file.HTTPStorage · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/download/file.py#L209" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Write.</p>+<h4>Args:</h4>+<ul>+<li><code>obj</code> — TODO.</li>+<li><code>url</code> — TODO.</li>+</ul>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def write(self, obj: bytes, url: Union[str, Path]) -> None:+ """Write.++ Args:+ obj: TODO.+ url: TODO.+ """+ raise NotImplementedError("write is not supported by HTTP Storage")+</code></pre></details></div>+<div class="api-detail" id="e472"><span class="dtag badge-m">method</span><h2><code>HTTPStorage.write_text(obj, url, encoding)</code></h2><div class="dmod">funasr.download.file.HTTPStorage · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/download/file.py#L218" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Write text.</p>+<h4>Args:</h4>+<ul>+<li><code>obj</code> — TODO.</li>+<li><code>url</code> — TODO.</li>+<li><code>encoding</code> — TODO.</li>+</ul>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def write_text(self, obj: str, url: Union[str, Path], encoding: str = "utf-8") -> None:+ """Write text.++ Args:+ obj: TODO.+ url: TODO.+ encoding: TODO.+ """+ raise NotImplementedError("write_text is not supported by HTTP Storage")+</code></pre></details></div>+<div class="api-detail" id="e473"><span class="dtag badge-c">class</span><h2>OSSStorage</h2><div class="dmod">funasr.download.file · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/download/file.py#L229" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>OSS storage.</p>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code>class OSSStorage(Storage):+ """OSS storage."""++ def __init__(self, oss_config_file=None):+ # read from config file or env var+ """Initialize OSSStorage.++ Args:+ oss_config_file: TODO.+ """+ raise NotImplementedError("OSSStorage.__init__ to be implemented in the future")++ def read(self, filepath):+ """Read.++ Args:+ filepath: TODO.+ """+ raise NotImplementedError("OSSStorage.read to be implemented in the future")++ def read_text(self, filepath, encoding="utf-8"):+ """Read text.++ Args:+ filepath: TODO.+ encoding: TODO.+ """+ raise NotImplementedError("OSSStorage.read_text to be implemented in the future")++ @contextlib.contextmanager+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/download/file.py#L229" target="_blank" class="src-more">View full source on GitHub →</a></details><h3 class="mt">Methods</h3><div class="mblk"><code>.read(filepath)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/download/file.py#L241" target="_blank" class="gh-link">L241</a><div class="mdoc"><p>Read.</p>+<h4>Args:</h4>+<ul>+<li><code>filepath</code> — TODO.</li>+</ul>+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def read(self, filepath):+ """Read.++ Args:+ filepath: TODO.+ """+ raise NotImplementedError("OSSStorage.read to be implemented in the future")+</code></pre></details></div><div class="mblk"><code>.read_text(filepath, encoding)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/download/file.py#L249" target="_blank" class="gh-link">L249</a><div class="mdoc"><p>Read text.</p>+<h4>Args:</h4>+<ul>+<li><code>filepath</code> — TODO.</li>+<li><code>encoding</code> — TODO.</li>+</ul>+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def read_text(self, filepath, encoding="utf-8"):+ """Read text.++ Args:+ filepath: TODO.+ encoding: TODO.+ """+ raise NotImplementedError("OSSStorage.read_text to be implemented in the future")+</code></pre></details></div><div class="mblk"><code>.as_local_path(filepath)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/download/file.py#L259" target="_blank" class="gh-link">L259</a><div class="mdoc"><p>Download a file from ``filepath``.</p>+<p>``as_local_path`` is decorated by :meth:`contextlib.contextmanager`. It</p>+<p>can be called with ``with`` statement, and when exists from the</p>+<p>``with`` statement, the temporary path will be released.</p>+<h4>Args:</h4>+<ul>+<li><code>filepath (str)</code> — Download a file from ``filepath``.</li>+</ul>+<h4>Examples:</h4>+<p>>>> storage = OSSStorage()</p>+<p>>>> # After existing from the ``with`` clause,</p>+<p>>>> # the path will be removed</p>+<p>>>> with storage.get_local_path('http://path/to/file') as path:</p>+<p>... # do something here</p>+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def as_local_path(self, filepath: str) -> Generator[Union[str, Path], None, None]:+ """Download a file from ``filepath``.++ ``as_local_path`` is decorated by :meth:`contextlib.contextmanager`. It+ can be called with ``with`` statement, and when exists from the+ ``with`` statement, the temporary path will be released.++ Args:+ filepath (str): Download a file from ``filepath``.++ Examples:+ >>> storage = OSSStorage()+ >>> # After existing from the ``with`` clause,+ >>> # the path will be removed+ >>> with storage.get_local_path('http://path/to/file') as path:+ ... # do something here+ """+ try:+ f = tempfile.NamedTemporaryFile(delete=False)+ f.write(self.read(filepath))+ f.close()+ yield f.name+ finally:+ os.remove(f.name)+</code></pre></details></div><div class="mblk"><code>.write(obj, filepath)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/download/file.py#L284" target="_blank" class="gh-link">L284</a><div class="mdoc"><p>Write.</p>+<h4>Args:</h4>+<ul>+<li><code>obj</code> — TODO.</li>+<li><code>filepath</code> — TODO.</li>+</ul>+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def write(self, obj: bytes, filepath: Union[str, Path]) -> None:+ """Write.++ Args:+ obj: TODO.+ filepath: TODO.+ """+ raise NotImplementedError("OSSStorage.write to be implemented in the future")+</code></pre></details></div><div class="mblk"><code>.write_text(obj, filepath, encoding)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/download/file.py#L293" target="_blank" class="gh-link">L293</a><div class="mdoc"><p>Write text.</p>+<h4>Args:</h4>+<ul>+<li><code>obj</code> — TODO.</li>+<li><code>filepath</code> — TODO.</li>+<li><code>encoding</code> — TODO.</li>+</ul>+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def write_text(self, obj: str, filepath: Union[str, Path], encoding: str = "utf-8") -> None:+ """Write text.++ Args:+ obj: TODO.+ filepath: TODO.+ encoding: TODO.+ """+ raise NotImplementedError("OSSStorage.write_text to be implemented in the future")+</code></pre></details></div></div>+<div class="api-detail" id="e474"><span class="dtag badge-m">method</span><h2><code>OSSStorage.read(filepath)</code></h2><div class="dmod">funasr.download.file.OSSStorage · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/download/file.py#L241" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Read.</p>+<h4>Args:</h4>+<ul>+<li><code>filepath</code> — TODO.</li>+</ul>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def read(self, filepath):+ """Read.++ Args:+ filepath: TODO.+ """+ raise NotImplementedError("OSSStorage.read to be implemented in the future")+</code></pre></details></div>+<div class="api-detail" id="e475"><span class="dtag badge-m">method</span><h2><code>OSSStorage.read_text(filepath, encoding)</code></h2><div class="dmod">funasr.download.file.OSSStorage · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/download/file.py#L249" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Read text.</p>+<h4>Args:</h4>+<ul>+<li><code>filepath</code> — TODO.</li>+<li><code>encoding</code> — TODO.</li>+</ul>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def read_text(self, filepath, encoding="utf-8"):+ """Read text.++ Args:+ filepath: TODO.+ encoding: TODO.+ """+ raise NotImplementedError("OSSStorage.read_text to be implemented in the future")+</code></pre></details></div>+<div class="api-detail" id="e476"><span class="dtag badge-m">method</span><h2><code>OSSStorage.as_local_path(filepath)</code></h2><div class="dmod">funasr.download.file.OSSStorage · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/download/file.py#L259" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Download a file from ``filepath``.</p>+<p>``as_local_path`` is decorated by :meth:`contextlib.contextmanager`. It</p>+<p>can be called with ``with`` statement, and when exists from the</p>+<p>``with`` statement, the temporary path will be released.</p>+<h4>Args:</h4>+<ul>+<li><code>filepath (str)</code> — Download a file from ``filepath``.</li>+</ul>+<h4>Examples:</h4>+<p>>>> storage = OSSStorage()</p>+<p>>>> # After existing from the ``with`` clause,</p>+<p>>>> # the path will be removed</p>+<p>>>> with storage.get_local_path('http://path/to/file') as path:</p>+<p>... # do something here</p>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def as_local_path(self, filepath: str) -> Generator[Union[str, Path], None, None]:+ """Download a file from ``filepath``.++ ``as_local_path`` is decorated by :meth:`contextlib.contextmanager`. It+ can be called with ``with`` statement, and when exists from the+ ``with`` statement, the temporary path will be released.++ Args:+ filepath (str): Download a file from ``filepath``.++ Examples:+ >>> storage = OSSStorage()+ >>> # After existing from the ``with`` clause,+ >>> # the path will be removed+ >>> with storage.get_local_path('http://path/to/file') as path:+ ... # do something here+ """+ try:+ f = tempfile.NamedTemporaryFile(delete=False)+ f.write(self.read(filepath))+ f.close()+ yield f.name+ finally:+ os.remove(f.name)+</code></pre></details></div>+<div class="api-detail" id="e477"><span class="dtag badge-m">method</span><h2><code>OSSStorage.write(obj, filepath)</code></h2><div class="dmod">funasr.download.file.OSSStorage · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/download/file.py#L284" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Write.</p>+<h4>Args:</h4>+<ul>+<li><code>obj</code> — TODO.</li>+<li><code>filepath</code> — TODO.</li>+</ul>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def write(self, obj: bytes, filepath: Union[str, Path]) -> None:+ """Write.++ Args:+ obj: TODO.+ filepath: TODO.+ """+ raise NotImplementedError("OSSStorage.write to be implemented in the future")+</code></pre></details></div>+<div class="api-detail" id="e478"><span class="dtag badge-m">method</span><h2><code>OSSStorage.write_text(obj, filepath, encoding)</code></h2><div class="dmod">funasr.download.file.OSSStorage · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/download/file.py#L293" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Write text.</p>+<h4>Args:</h4>+<ul>+<li><code>obj</code> — TODO.</li>+<li><code>filepath</code> — TODO.</li>+<li><code>encoding</code> — TODO.</li>+</ul>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def write_text(self, obj: str, filepath: Union[str, Path], encoding: str = "utf-8") -> None:+ """Write text.++ Args:+ obj: TODO.+ filepath: TODO.+ encoding: TODO.+ """+ raise NotImplementedError("OSSStorage.write_text to be implemented in the future")+</code></pre></details></div>+<div class="api-detail" id="e479"><span class="dtag badge-c">class</span><h2>File</h2><div class="dmod">funasr.download.file · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/download/file.py#L307" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p class="muted">No documentation yet.</p></div><details class="src-block"><summary>📄 Source code</summary><pre><code>class File(object):+ _prefix_to_storage: dict = {+ "oss": OSSStorage,+ "http": HTTPStorage,+ "https": HTTPStorage,+ "local": LocalStorage,+ }++ @staticmethod+ def _get_storage(uri):+ """Internal: get storage.++ Args:+ uri: TODO.+ """+ assert isinstance(uri, str), f"uri should be str type, but got {type(uri)}"++ if "://" not in uri:+ # local path+ storage_type = "local"+ else:+ prefix, _ = uri.split("://")+ storage_type = prefix++ assert storage_type in File._prefix_to_storage, (+ f"Unsupported uri {uri}, valid prefixs: " f"{list(File._prefix_to_storage.keys())}"+ )++ if storage_type not in G_STORAGES:+ G_STORAGES[storage_type] = File._prefix_to_storage[storage_type]()+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/download/file.py#L307" target="_blank" class="src-more">View full source on GitHub →</a></details><h3 class="mt">Methods</h3><div class="mblk"><code>.read(uri)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/download/file.py#L341" target="_blank" class="gh-link">L341</a><div class="mdoc"><p>Read data from a given ``filepath`` with 'rb' mode.</p>+<h4>Args:</h4>+<ul>+<li><code>filepath (str or Path)</code> — Path to read data.</li>+</ul>+<h4>Returns:</h4>+<ul>+<li><code>bytes</code> — Expected bytes object.</li>+</ul>+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def read(uri: str) -> bytes:+ """Read data from a given ``filepath`` with 'rb' mode.++ Args:+ filepath (str or Path): Path to read data.++ Returns:+ bytes: Expected bytes object.+ """+ storage = File._get_storage(uri)+ return storage.read(uri)+</code></pre></details></div><div class="mblk"><code>.read_text(uri, encoding)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/download/file.py#L354" target="_blank" class="gh-link">L354</a><div class="mdoc"><p>Read data from a given ``filepath`` with 'r' mode.</p>+<h4>Args:</h4>+<ul>+<li><code>filepath (str or Path)</code> — Path to read data.</li>+<li><code>encoding (str)</code> — The encoding format used to open the ``filepath``.</li>+<li><code>Default</code> — 'utf-8'.</li>+</ul>+<h4>Returns:</h4>+<ul>+<li><code>str</code> — Expected text reading from ``filepath``.</li>+</ul>+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def read_text(uri: Union[str, Path], encoding: str = "utf-8") -> str:+ """Read data from a given ``filepath`` with 'r' mode.++ Args:+ filepath (str or Path): Path to read data.+ encoding (str): The encoding format used to open the ``filepath``.+ Default: 'utf-8'.++ Returns:+ str: Expected text reading from ``filepath``.+ """+ storage = File._get_storage(uri)+ return storage.read_text(uri)+</code></pre></details></div><div class="mblk"><code>.write(obj, uri)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/download/file.py#L369" target="_blank" class="gh-link">L369</a><div class="mdoc"><p>Write data to a given ``filepath`` with 'wb' mode.</p>+<h4>Note:</h4>+<p>``write`` will create a directory if the directory of ``filepath``</p>+<p>does not exist.</p>+<h4>Args:</h4>+<ul>+<li><code>obj (bytes)</code> — Data to be written.</li>+<li><code>filepath (str or Path)</code> — Path to write data.</li>+</ul>+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def write(obj: bytes, uri: Union[str, Path]) -> None:+ """Write data to a given ``filepath`` with 'wb' mode.++ Note:+ ``write`` will create a directory if the directory of ``filepath``+ does not exist.++ Args:+ obj (bytes): Data to be written.+ filepath (str or Path): Path to write data.+ """+ storage = File._get_storage(uri)+ return storage.write(obj, uri)+</code></pre></details></div><div class="mblk"><code>.write_text(obj, uri, encoding)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/download/file.py#L384" target="_blank" class="gh-link">L384</a><div class="mdoc"><p>Write data to a given ``filepath`` with 'w' mode.</p>+<h4>Note:</h4>+<p>``write_text`` will create a directory if the directory of</p>+<p>``filepath`` does not exist.</p>+<h4>Args:</h4>+<ul>+<li><code>obj (str)</code> — Data to be written.</li>+<li><code>filepath (str or Path)</code> — Path to write data.</li>+<li><code>encoding (str)</code> — The encoding format used to open the ``filepath``.</li>+<li><code>Default</code> — 'utf-8'.</li>+</ul>+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def write_text(obj: str, uri: str, encoding: str = "utf-8") -> None:+ """Write data to a given ``filepath`` with 'w' mode.++ Note:+ ``write_text`` will create a directory if the directory of+ ``filepath`` does not exist.++ Args:+ obj (str): Data to be written.+ filepath (str or Path): Path to write data.+ encoding (str): The encoding format used to open the ``filepath``.+ Default: 'utf-8'.+ """+ storage = File._get_storage(uri)+ return storage.write_text(obj, uri)+</code></pre></details></div><div class="mblk"><code>.as_local_path(uri)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/download/file.py#L401" target="_blank" class="gh-link">L401</a><div class="mdoc"><p>Only for unified API and do nothing.</p>+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def as_local_path(uri: str) -> Generator[Union[str, Path], None, None]:+ """Only for unified API and do nothing."""+ storage = File._get_storage(uri)+ with storage.as_local_path(uri) as local_path:+ yield local_path+</code></pre></details></div></div>+<div class="api-detail" id="e480"><span class="dtag badge-m">method</span><h2><code>File.read(uri)</code></h2><div class="dmod">funasr.download.file.File · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/download/file.py#L341" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Read data from a given ``filepath`` with 'rb' mode.</p>+<h4>Args:</h4>+<ul>+<li><code>filepath (str or Path)</code> — Path to read data.</li>+</ul>+<h4>Returns:</h4>+<ul>+<li><code>bytes</code> — Expected bytes object.</li>+</ul>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def read(uri: str) -> bytes:+ """Read data from a given ``filepath`` with 'rb' mode.++ Args:+ filepath (str or Path): Path to read data.++ Returns:+ bytes: Expected bytes object.+ """+ storage = File._get_storage(uri)+ return storage.read(uri)+</code></pre></details></div>+<div class="api-detail" id="e481"><span class="dtag badge-m">method</span><h2><code>File.read_text(uri, encoding)</code></h2><div class="dmod">funasr.download.file.File · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/download/file.py#L354" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Read data from a given ``filepath`` with 'r' mode.</p>+<h4>Args:</h4>+<ul>+<li><code>filepath (str or Path)</code> — Path to read data.</li>+<li><code>encoding (str)</code> — The encoding format used to open the ``filepath``.</li>+<li><code>Default</code> — 'utf-8'.</li>+</ul>+<h4>Returns:</h4>+<ul>+<li><code>str</code> — Expected text reading from ``filepath``.</li>+</ul>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def read_text(uri: Union[str, Path], encoding: str = "utf-8") -> str:+ """Read data from a given ``filepath`` with 'r' mode.++ Args:+ filepath (str or Path): Path to read data.+ encoding (str): The encoding format used to open the ``filepath``.+ Default: 'utf-8'.++ Returns:+ str: Expected text reading from ``filepath``.+ """+ storage = File._get_storage(uri)+ return storage.read_text(uri)+</code></pre></details></div>+<div class="api-detail" id="e482"><span class="dtag badge-m">method</span><h2><code>File.write(obj, uri)</code></h2><div class="dmod">funasr.download.file.File · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/download/file.py#L369" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Write data to a given ``filepath`` with 'wb' mode.</p>+<h4>Note:</h4>+<p>``write`` will create a directory if the directory of ``filepath``</p>+<p>does not exist.</p>+<h4>Args:</h4>+<ul>+<li><code>obj (bytes)</code> — Data to be written.</li>+<li><code>filepath (str or Path)</code> — Path to write data.</li>+</ul>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def write(obj: bytes, uri: Union[str, Path]) -> None:+ """Write data to a given ``filepath`` with 'wb' mode.++ Note:+ ``write`` will create a directory if the directory of ``filepath``+ does not exist.++ Args:+ obj (bytes): Data to be written.+ filepath (str or Path): Path to write data.+ """+ storage = File._get_storage(uri)+ return storage.write(obj, uri)+</code></pre></details></div>+<div class="api-detail" id="e483"><span class="dtag badge-m">method</span><h2><code>File.write_text(obj, uri, encoding)</code></h2><div class="dmod">funasr.download.file.File · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/download/file.py#L384" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Write data to a given ``filepath`` with 'w' mode.</p>+<h4>Note:</h4>+<p>``write_text`` will create a directory if the directory of</p>+<p>``filepath`` does not exist.</p>+<h4>Args:</h4>+<ul>+<li><code>obj (str)</code> — Data to be written.</li>+<li><code>filepath (str or Path)</code> — Path to write data.</li>+<li><code>encoding (str)</code> — The encoding format used to open the ``filepath``.</li>+<li><code>Default</code> — 'utf-8'.</li>+</ul>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def write_text(obj: str, uri: str, encoding: str = "utf-8") -> None:+ """Write data to a given ``filepath`` with 'w' mode.++ Note:+ ``write_text`` will create a directory if the directory of+ ``filepath`` does not exist.++ Args:+ obj (str): Data to be written.+ filepath (str or Path): Path to write data.+ encoding (str): The encoding format used to open the ``filepath``.+ Default: 'utf-8'.+ """+ storage = File._get_storage(uri)+ return storage.write_text(obj, uri)+</code></pre></details></div>+<div class="api-detail" id="e484"><span class="dtag badge-m">method</span><h2><code>File.as_local_path(uri)</code></h2><div class="dmod">funasr.download.file.File · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/download/file.py#L401" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Only for unified API and do nothing.</p>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def as_local_path(uri: str) -> Generator[Union[str, Path], None, None]:+ """Only for unified API and do nothing."""+ storage = File._get_storage(uri)+ with storage.as_local_path(uri) as local_path:+ yield local_path+</code></pre></details></div>+<div class="api-detail" id="e485"><span class="dtag badge-f">function</span><h2><code>main()</code></h2><div class="dmod">funasr.download.runtime_sdk_download_tool · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/download/runtime_sdk_download_tool.py#L8" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Main.</p>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code>def main():+ """Main."""+ parser = argparse.ArgumentParser()+ parser.add_argument("--model-name", type=str, required=True)+ parser.add_argument("--export-dir", type=str, required=True)+ parser.add_argument("--export", type=str2bool, default=True, help="whether to export model")+ parser.add_argument("--type", type=str, default="onnx", help='["onnx", "torchscript", "bladedisc"]')+ parser.add_argument("--device", type=str, default="cpu", help='["cpu", "cuda"]')+ parser.add_argument("--quantize", type=str2bool, default=False, help="export quantized model")+ parser.add_argument("--fallback-num", type=int, default=0, help="amp fallback number")+ parser.add_argument("--audio_in", type=str, default=None, help='["wav", "wav.scp"]')+ parser.add_argument("--model_revision", type=str, default=None, help="model_revision")+ parser.add_argument("--calib_num", type=int, default=200, help="calib max num")+ args = parser.parse_args()++ model_dir = args.model_name+ output_dir = args.model_name+ if not Path(args.model_name).exists():+ from modelscope.hub.snapshot_download import snapshot_download++ try:+ model_dir = snapshot_download(+ args.model_name, cache_dir=args.export_dir, revision=args.model_revision+ )+ output_dir = os.path.join(args.export_dir, args.model_name)+ except:+ raise "model_dir must be model_name in modelscope or local path downloaded from modelscope, but is {}".format(+ model_dir+ )+ if args.export:+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/download/runtime_sdk_download_tool.py#L8" target="_blank" class="src-more">View full source on GitHub →</a></details></div>+<div class="api-detail" id="e486"><span class="dtag badge-f">function</span><h2><code>add_gradient_noise(model, iteration, duration, eta, scale_factor)</code></h2><div class="dmod">funasr.train_utils.add_gradient_noise · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/train_utils/add_gradient_noise.py#L4" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Adds noise from a standard normal distribution to the gradients.</p>+<p>The standard deviation (`sigma`) is controlled</p>+<p>by the three hyper-parameters below.</p>+<p>`sigma` goes to zero (no noise) with more iterations.</p>+<h4>Args:</h4>+<ul>+<li><code>model</code> — Model.</li>+<li><code>iteration</code> — Number of iterations.</li>+<li><code>duration</code> — {100, 1000}: Number of durations to control</li>+<li class="sub">the interval of the `sigma` change.</li>+<li><code>eta</code> — {0.01, 0.3, 1.0}: The magnitude of `sigma`.</li>+<li><code>scale_factor</code> — {0.55}: The scale of `sigma`.</li>+</ul>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code>def add_gradient_noise(+ model: torch.nn.Module,+ iteration: int,+ duration: float = 100,+ eta: float = 1.0,+ scale_factor: float = 0.55,+):+ """Adds noise from a standard normal distribution to the gradients.++ The standard deviation (`sigma`) is controlled+ by the three hyper-parameters below.+ `sigma` goes to zero (no noise) with more iterations.++ Args:+ model: Model.+ iteration: Number of iterations.+ duration: {100, 1000}: Number of durations to control+ the interval of the `sigma` change.+ eta: {0.01, 0.3, 1.0}: The magnitude of `sigma`.+ scale_factor: {0.55}: The scale of `sigma`.+ """+ interval = (iteration // duration) + 1+ sigma = eta / interval**scale_factor+ for param in model.parameters():+ if param.grad is not None:+ _shape = param.grad.size()+ noise = sigma * torch.randn(_shape).to(param.device)+ param.grad += noise+</code></pre></details></div>+<div class="api-detail" id="e487"><span class="dtag badge-f">function</span><h2><code>average_checkpoints(output_dir, last_n, **kwargs)</code></h2><div class="dmod">funasr.train_utils.average_nbest_models · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/train_utils/average_nbest_models.py#L61" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Average the last 'last_n' checkpoints' model state_dicts.</p>+<p>If a tensor is of type torch.int, perform sum instead of average.</p>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code>def average_checkpoints(output_dir: str, last_n: int = 5, **kwargs):+ """+ Average the last 'last_n' checkpoints' model state_dicts.+ If a tensor is of type torch.int, perform sum instead of average.+ """+ checkpoint_paths = _get_checkpoint_paths(output_dir, last_n, **kwargs)+ print(f"average_checkpoints: {checkpoint_paths}")+ state_dicts = []++ # Load state_dicts from checkpoints+ for path in checkpoint_paths:+ if os.path.isfile(path):+ state_dicts.append(torch.load(path, map_location="cpu")["state_dict"])+ else:+ print(f"Checkpoint file {path} not found.")++ # Check if we have any state_dicts to average+ if len(state_dicts) < 1:+ print("No checkpoints found for averaging.")+ return++ # Average or sum weights+ avg_state_dict = OrderedDict()+ for key in state_dicts[0].keys():+ tensors = [state_dict[key].cpu() for state_dict in state_dicts]+ # Check the type of the tensor+ if str(tensors[0].dtype).startswith("torch.int"):+ # Perform sum for integer tensors+ summed_tensor = sum(tensors)+ avg_state_dict[key] = summed_tensor+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/train_utils/average_nbest_models.py#L61" target="_blank" class="src-more">View full source on GitHub →</a></details></div>+<div class="api-detail" id="e488"><span class="dtag badge-f">function</span><h2><code>to_device(data, device, dtype, non_blocking, copy)</code></h2><div class="dmod">funasr.train_utils.device_funcs · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/train_utils/device_funcs.py#L8" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Change the device of object recursively</p>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code>def to_device(data, device=None, dtype=None, non_blocking=False, copy=False):+ """Change the device of object recursively"""+ if isinstance(data, dict):+ return {k: to_device(v, device, dtype, non_blocking, copy) for k, v in data.items()}+ elif dataclasses.is_dataclass(data) and not isinstance(data, type):+ return type(data)(+ *[to_device(v, device, dtype, non_blocking, copy) for v in dataclasses.astuple(data)]+ )+ # maybe namedtuple. I don't know the correct way to judge namedtuple.+ elif isinstance(data, tuple) and type(data) is not tuple:+ return type(data)(*[to_device(o, device, dtype, non_blocking, copy) for o in data])+ elif isinstance(data, (list, tuple)):+ return type(data)(to_device(v, device, dtype, non_blocking, copy) for v in data)+ elif isinstance(data, np.ndarray):+ return to_device(torch.from_numpy(data), device, dtype, non_blocking, copy)+ elif isinstance(data, torch.Tensor):+ return data.to(device, dtype, non_blocking, copy)+ else:+ return data+</code></pre></details></div>+<div class="api-detail" id="e489"><span class="dtag badge-f">function</span><h2><code>force_gatherable(data, device)</code></h2><div class="dmod">funasr.train_utils.device_funcs · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/train_utils/device_funcs.py#L29" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Change object to gatherable in torch.nn.DataParallel recursively</p>+<p>The difference from to_device() is changing to torch.Tensor if float or int</p>+<p>value is found.</p>+<p>The restriction to the returned value in DataParallel:</p>+<p>The object must be</p>+<ul>+<li>torch.cuda.Tensor</li>+<li>1 or more dimension. 0-dimension-tensor sends warning.</li>+</ul>+<p>or a list, tuple, dict.</p>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code>def force_gatherable(data, device):+ """Change object to gatherable in torch.nn.DataParallel recursively++ The difference from to_device() is changing to torch.Tensor if float or int+ value is found.++ The restriction to the returned value in DataParallel:+ The object must be+ - torch.cuda.Tensor+ - 1 or more dimension. 0-dimension-tensor sends warning.+ or a list, tuple, dict.++ """+ if isinstance(data, dict):+ return {k: force_gatherable(v, device) for k, v in data.items()}+ # DataParallel can't handle NamedTuple well+ elif isinstance(data, tuple) and type(data) is not tuple:+ return type(data)(*[force_gatherable(o, device) for o in data])+ elif isinstance(data, (list, tuple, set)):+ return type(data)(force_gatherable(v, device) for v in data)+ elif isinstance(data, np.ndarray):+ return force_gatherable(torch.from_numpy(data), device)+ elif isinstance(data, torch.Tensor):+ if data.dim() == 0:+ # To 1-dim array+ data = data[None]+ return data.to(device)+ elif isinstance(data, float):+ return torch.tensor([data], dtype=torch.float, device=device)+ elif isinstance(data, int):+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/train_utils/device_funcs.py#L29" target="_blank" class="src-more">View full source on GitHub →</a></details></div>+<div class="api-detail" id="e490"><span class="dtag badge-c">class</span><h2>ForwardAdaptor</h2><div class="dmod">funasr.train_utils.forward_adaptor · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/train_utils/forward_adaptor.py#L4" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Wrapped module to parallelize specified method</p>+<p>torch.nn.DataParallel parallelizes only "forward()"</p>+<p>and, maybe, the method having the other name can't be applied</p>+<p>except for wrapping the module just like this class.</p>+<h4>Examples:</h4>+<p>>>> class A(torch.nn.Module):</p>+<p>... def foo(self, x):</p>+<p>... ...</p>+<p>>>> model = A()</p>+<p>>>> model = ForwardAdaptor(model, "foo")</p>+<p>>>> model = torch.nn.DataParallel(model, device_ids=[0, 1])</p>+<p>>>> x = torch.randn(2, 10)</p>+<p>>>> model(x)</p>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code>class ForwardAdaptor(torch.nn.Module):+ """Wrapped module to parallelize specified method++ torch.nn.DataParallel parallelizes only "forward()"+ and, maybe, the method having the other name can't be applied+ except for wrapping the module just like this class.++ Examples:+ >>> class A(torch.nn.Module):+ ... def foo(self, x):+ ... ...+ >>> model = A()+ >>> model = ForwardAdaptor(model, "foo")+ >>> model = torch.nn.DataParallel(model, device_ids=[0, 1])+ >>> x = torch.randn(2, 10)+ >>> model(x)+ """++ def __init__(self, module: torch.nn.Module, name: str):+ """Initialize ForwardAdaptor.++ Args:+ module: TODO.+ name: TODO.+ """+ super().__init__()+ self.module = module+ self.name = name+ if not hasattr(module, name):+ raise ValueError(f"{module} doesn't have {name}")+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/train_utils/forward_adaptor.py#L4" target="_blank" class="src-more">View full source on GitHub →</a></details><h3 class="mt">Methods</h3><div class="mblk"><code>.forward(*args, **kwargs)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/train_utils/forward_adaptor.py#L35" target="_blank" class="gh-link">L35</a><div class="mdoc"><p>Forward pass for training.</p>+<h4>Args:</h4>+<ul>+<li><code>*args</code> — Variable positional arguments.</li>+<li><code>**kwargs</code> — Additional keyword arguments.</li>+</ul>+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def forward(self, *args, **kwargs):+ """Forward pass for training.++ Args:+ *args: Variable positional arguments.+ **kwargs: Additional keyword arguments.+ """+ func = getattr(self.module, self.name)+ return func(*args, **kwargs)+</code></pre></details></div></div>+<div class="api-detail" id="e491"><span class="dtag badge-m">method</span><h2><code>ForwardAdaptor.forward(*args, **kwargs)</code></h2><div class="dmod">funasr.train_utils.forward_adaptor.ForwardAdaptor · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/train_utils/forward_adaptor.py#L35" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Forward pass for training.</p>+<h4>Args:</h4>+<ul>+<li><code>*args</code> — Variable positional arguments.</li>+<li><code>**kwargs</code> — Additional keyword arguments.</li>+</ul>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def forward(self, *args, **kwargs):+ """Forward pass for training.++ Args:+ *args: Variable positional arguments.+ **kwargs: Additional keyword arguments.+ """+ func = getattr(self.module, self.name)+ return func(*args, **kwargs)+</code></pre></details></div>+<div class="api-detail" id="e492"><span class="dtag badge-f">function</span><h2><code>initialize(model, init)</code></h2><div class="dmod">funasr.train_utils.initialize · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/train_utils/initialize.py#L9" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Initialize weights of a neural network module.</p>+<p>Parameters are initialized using the given method or distribution.</p>+<p>Custom initialization routines can be implemented into submodules</p>+<p>as function `espnet_initialization_fn` within the custom module.</p>+<h4>Args:</h4>+<ul>+<li><code>model</code> — Target.</li>+<li><code>init</code> — Method of initialization.</li>+</ul>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code>def initialize(model: torch.nn.Module, init: str):+ """Initialize weights of a neural network module.++ Parameters are initialized using the given method or distribution.++ Custom initialization routines can be implemented into submodules+ as function `espnet_initialization_fn` within the custom module.++ Args:+ model: Target.+ init: Method of initialization.+ """++ # weight init+ for p in model.parameters():+ if p.dim() > 1:+ if init == "xavier_uniform":+ torch.nn.init.xavier_uniform_(p.data)+ elif init == "xavier_normal":+ torch.nn.init.xavier_normal_(p.data)+ elif init == "kaiming_uniform":+ torch.nn.init.kaiming_uniform_(p.data, nonlinearity="relu")+ elif init == "kaiming_normal":+ torch.nn.init.kaiming_normal_(p.data, nonlinearity="relu")+ else:+ raise ValueError("Unknown initialization: " + init)+ # bias init+ for p in model.parameters():+ if p.dim() == 1:+ p.data.zero_()+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/train_utils/initialize.py#L9" target="_blank" class="src-more">View full source on GitHub →</a></details></div>+<div class="api-detail" id="e493"><span class="dtag badge-f">function</span><h2><code>load_pretrained_model(path, model, ignore_init_mismatch, map_location, oss_bucket, scope_map, excludes, **kwargs)</code></h2><div class="dmod">funasr.train_utils.load_pretrained_model · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/train_utils/load_pretrained_model.py#L14" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Load a model state and set it to the model.</p>+<h4>Args:</h4>+<ul>+<li><code>init_param</code> — <file_path>:<src_key>:<dst_key>:<exclude_Keys></li>+</ul>+<h4>Examples:</h4>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code>def load_pretrained_model(+ path: str,+ model: torch.nn.Module,+ ignore_init_mismatch: bool = True,+ map_location: str = "cpu",+ oss_bucket=None,+ scope_map=[],+ excludes=None,+ **kwargs,+):+ """Load a model state and set it to the model.++ Args:+ init_param: <file_path>:<src_key>:<dst_key>:<exclude_Keys>++ Examples:++ """++ obj = model+ dst_state = obj.state_dict()++ logging.info(f"ckpt: {path}")++ if oss_bucket is None:+ ori_state = torch.load(path, map_location=map_location)+ else:+ buffer = BytesIO(oss_bucket.get_object(path).read())+ ori_state = torch.load(buffer, map_location=map_location)++</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/train_utils/load_pretrained_model.py#L14" target="_blank" class="src-more">View full source on GitHub →</a></details></div>+<div class="api-detail" id="e494"><span class="dtag badge-f">function</span><h2><code>get_human_readable_count(number)</code></h2><div class="dmod">funasr.train_utils.model_summary · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/train_utils/model_summary.py#L5" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Return human_readable_count</p>+<p>Originated from:</p>+<p>https://github.com/PyTorchLightning/pytorch-lightning/blob/master/pytorch_lightning/core/memory.py</p>+<p>Abbreviates an integer number with K, M, B, T for thousands, millions,</p>+<p>billions and trillions, respectively.</p>+<h4>Examples:</h4>+<p>>>> get_human_readable_count(123)</p>+<p>'123 '</p>+<p>>>> get_human_readable_count(1234) # (one thousand)</p>+<p>'1 K'</p>+<p>>>> get_human_readable_count(2e6) # (two million)</p>+<p>'2 M'</p>+<p>>>> get_human_readable_count(3e9) # (three billion)</p>+<p>'3 B'</p>+<p>>>> get_human_readable_count(4e12) # (four trillion)</p>+<p>'4 T'</p>+<p>>>> get_human_readable_count(5e15) # (more than trillion)</p>+<p>'5,000 T'</p>+<h4>Args:</h4>+<ul>+<li><code>number</code> — a positive integer number</li>+</ul>+<p>Return:</p>+<p>A string formatted according to the pattern described above.</p>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code>def get_human_readable_count(number: int) -> str:+ """Return human_readable_count++ Originated from:+ https://github.com/PyTorchLightning/pytorch-lightning/blob/master/pytorch_lightning/core/memory.py++ Abbreviates an integer number with K, M, B, T for thousands, millions,+ billions and trillions, respectively.+ Examples:+ >>> get_human_readable_count(123)+ '123 '+ >>> get_human_readable_count(1234) # (one thousand)+ '1 K'+ >>> get_human_readable_count(2e6) # (two million)+ '2 M'+ >>> get_human_readable_count(3e9) # (three billion)+ '3 B'+ >>> get_human_readable_count(4e12) # (four trillion)+ '4 T'+ >>> get_human_readable_count(5e15) # (more than trillion)+ '5,000 T'+ Args:+ number: a positive integer number+ Return:+ A string formatted according to the pattern described above.+ """+ assert number >= 0+ labels = [" ", "K", "M", "B", "T"]+ num_digits = int(np.floor(np.log10(number)) + 1 if number > 0 else 1)+ num_groups = int(np.ceil(num_digits / 3))+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/train_utils/model_summary.py#L5" target="_blank" class="src-more">View full source on GitHub →</a></details></div>+<div class="api-detail" id="e495"><span class="dtag badge-f">function</span><h2><code>to_bytes(dtype)</code></h2><div class="dmod">funasr.train_utils.model_summary · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/train_utils/model_summary.py#L42" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>To bytes.</p>+<h4>Args:</h4>+<ul>+<li><code>dtype</code> — TODO.</li>+</ul>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code>def to_bytes(dtype) -> int:+ # torch.float16 -> 16+ """To bytes.++ Args:+ dtype: TODO.+ """+ return int(str(dtype)[-2:]) // 8+</code></pre></details></div>+<div class="api-detail" id="e496"><span class="dtag badge-f">function</span><h2><code>model_summary(model)</code></h2><div class="dmod">funasr.train_utils.model_summary · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/train_utils/model_summary.py#L52" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Model summary.</p>+<h4>Args:</h4>+<ul>+<li><code>model</code> — Model instance or model name.</li>+</ul>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code>def model_summary(model: torch.nn.Module) -> str:+ """Model summary.++ Args:+ model: Model instance or model name.+ """+ message = "Model structure:\n"+ message += str(model)++ tot_params, num_params = 0, 0+ for name, param in model.named_parameters():+ print(+ "name: {}, dtype: {}, device: {}, trainable: {}, shape: {}, numel: {}".format(+ name, param.dtype, param.device, param.requires_grad, param.shape, param.numel()+ )+ )+ tot_params += param.numel()+ if param.requires_grad:+ num_params += param.numel()++ percent_trainable = "{:.1f}".format(num_params * 100.0 / tot_params)+ tot_params = get_human_readable_count(tot_params)+ num_params = get_human_readable_count(num_params)+ message += "\n\nModel summary:\n"+ message += f" Class Name: {model.__class__.__name__}\n"+ message += f" Total Number of model parameters: {tot_params}\n"+ message += f" Number of trainable parameters: {num_params} ({percent_trainable}%)\n"++ dtype = next(iter(model.parameters())).dtype+ message += f" Type: {dtype}"+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/train_utils/model_summary.py#L52" target="_blank" class="src-more">View full source on GitHub →</a></details></div>+<div class="api-detail" id="e497"><span class="dtag badge-f">function</span><h2><code>recursive_sum(obj, weight, distributed)</code></h2><div class="dmod">funasr.train_utils.recursive_op · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/train_utils/recursive_op.py#L9" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Recursive sum.</p>+<h4>Args:</h4>+<ul>+<li><code>obj</code> — TODO.</li>+<li><code>weight</code> — TODO.</li>+<li><code>distributed</code> — TODO.</li>+</ul>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code>def recursive_sum(obj, weight: torch.Tensor, distributed: bool = False):+ """Recursive sum.++ Args:+ obj: TODO.+ weight: TODO.+ distributed: TODO.+ """+ assert weight.dim() == 1, weight.size()+ if isinstance(obj, (tuple, list)):+ return type(obj)(recursive_sum(v, weight, distributed) for v in obj)+ elif isinstance(obj, dict):+ return {k: recursive_sum(v, weight, distributed) for k, v in obj.items()}+ elif isinstance(obj, torch.Tensor):+ assert obj.size() == weight.size(), (obj.size(), weight.size())+ obj = (obj * weight.type(obj.dtype)).sum()+ if distributed:+ torch.distributed.all_reduce(obj, op=ReduceOp.SUM)+ return obj+ elif obj is None:+ return None+ else:+ raise ValueError(type(obj))+</code></pre></details></div>+<div class="api-detail" id="e498"><span class="dtag badge-f">function</span><h2><code>recursive_divide(a, b)</code></h2><div class="dmod">funasr.train_utils.recursive_op · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/train_utils/recursive_op.py#L34" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Recursive divide.</p>+<h4>Args:</h4>+<ul>+<li><code>a</code> — TODO.</li>+<li><code>b</code> — TODO.</li>+</ul>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code>def recursive_divide(a, b: torch.Tensor):+ """Recursive divide.++ Args:+ a: TODO.+ b: TODO.+ """+ if isinstance(a, (tuple, list)):+ return type(a)(recursive_divide(v, b) for v in a)+ elif isinstance(a, dict):+ return {k: recursive_divide(v, b) for k, v in a.items()}+ elif isinstance(a, torch.Tensor):+ assert a.size() == b.size(), (a.size(), b.size())+ return a / b.type(a.dtype)+ elif a is None:+ return None+ else:+ raise ValueError(type(a))+</code></pre></details></div>+<div class="api-detail" id="e499"><span class="dtag badge-f">function</span><h2><code>recursive_average(obj, weight, distributed)</code></h2><div class="dmod">funasr.train_utils.recursive_op · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/train_utils/recursive_op.py#L54" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Recursive average.</p>+<h4>Args:</h4>+<ul>+<li><code>obj</code> — TODO.</li>+<li><code>weight</code> — TODO.</li>+<li><code>distributed</code> — TODO.</li>+</ul>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code>def recursive_average(obj, weight: torch.Tensor, distributed: bool = False):+ """Recursive average.++ Args:+ obj: TODO.+ weight: TODO.+ distributed: TODO.+ """+ obj = recursive_sum(obj, weight, distributed)+ weight = weight.sum()+ if distributed:+ torch.distributed.all_reduce(weight, op=ReduceOp.SUM)+ # Normalize weight to be sum-to-1+ obj = recursive_divide(obj, weight)+ return obj, weight+</code></pre></details></div>+<div class="api-detail" id="e500"><span class="dtag badge-f">function</span><h2><code>set_all_random_seed(seed)</code></h2><div class="dmod">funasr.train_utils.set_all_random_seed · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/train_utils/set_all_random_seed.py#L7" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Set all random seed.</p>+<h4>Args:</h4>+<ul>+<li><code>seed</code> — TODO.</li>+</ul>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code>def set_all_random_seed(seed: int):+ """Set all random seed.++ Args:+ seed: TODO.+ """+ random.seed(seed)+ np.random.seed(seed)+ torch.random.manual_seed(seed)+</code></pre></details></div>+<div class="api-detail" id="e501"><span class="dtag badge-f">function</span><h2><code>maybe_autocast(enabled)</code></h2><div class="dmod">funasr.train_utils.trainer · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/train_utils/trainer.py#L25" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Maybe autocast.</p>+<h4>Args:</h4>+<ul>+<li><code>enabled</code> — TODO.</li>+</ul>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code>def maybe_autocast(enabled):+ """Maybe autocast.++ Args:+ enabled: TODO.+ """+ if enabled:+ with autocast():+ yield+ else:+ yield+</code></pre></details></div>+<div class="api-detail" id="e502"><span class="dtag badge-c">class</span><h2>Trainer</h2><div class="dmod">funasr.train_utils.trainer · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/train_utils/trainer.py#L38" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>A simple trainer class for training a PyTorch model, saving checkpoints at the end of each epoch,</p>+<p>and optionally resuming from a saved checkpoint.</p>+<p>Attributes:</p>+<ul>+<li><code>max_epoch (int)</code> — Maximum number of epochs for training.</li>+<li><code>model (torch.nn.Module)</code> — The model to be trained.</li>+<li><code>optim (torch.optim.Optimizer)</code> — The optimizer to use for training.</li>+<li><code>scheduler (torch.optim.lr_scheduler._LRScheduler)</code> — The learning rate scheduler.</li>+<li><code>dataloader_train (torch.utils.data.DataLoader)</code> — DataLoader for the training dataset.</li>+<li><code>dataloader_val (torch.utils.data.DataLoader)</code> — DataLoader for the validation dataset.</li>+<li><code>output_dir (str)</code> — Directory where model checkpoints will be saved.</li>+<li><code>resume (str, optional)</code> — Path to a checkpoint to resume training from.</li>+</ul>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code>class Trainer:+ """+ A simple trainer class for training a PyTorch model, saving checkpoints at the end of each epoch,+ and optionally resuming from a saved checkpoint.++ Attributes:+ max_epoch (int): Maximum number of epochs for training.+ model (torch.nn.Module): The model to be trained.+ optim (torch.optim.Optimizer): The optimizer to use for training.+ scheduler (torch.optim.lr_scheduler._LRScheduler): The learning rate scheduler.+ dataloader_train (torch.utils.data.DataLoader): DataLoader for the training dataset.+ dataloader_val (torch.utils.data.DataLoader): DataLoader for the validation dataset.+ output_dir (str): Directory where model checkpoints will be saved.+ resume (str, optional): Path to a checkpoint to resume training from.+ """++ def __init__(+ self,+ local_rank,+ use_ddp: bool = False,+ use_fsdp: bool = False,+ use_fp16: bool = False,+ output_dir: str = "./",+ **kwargs,+ ):+ """+ Initializes the Trainer class with the model, optimizer, scheduler, dataloaders, and other settings.++ Args:+ model (torch.nn.Module): The model to be trained.+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/train_utils/trainer.py#L38" target="_blank" class="src-more">View full source on GitHub →</a></details><h3 class="mt">Methods</h3><div class="mblk"><code>.save_checkpoint(epoch, step, model, optim, scheduler, scaler, step_in_epoch, **kwargs)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/train_utils/trainer.py#L143" target="_blank" class="gh-link">L143</a><div class="mdoc"><p>Saves a checkpoint containing the model's state, the optimizer's state,</p>+<p>and the scheduler's state at the end of the given epoch. This method is</p>+<p>intended to be called at the end of each epoch to save the training progress.</p>+<h4>Args:</h4>+<ul>+<li><code>epoch (int)</code> — The epoch number at which the checkpoint is being saved.</li>+</ul>+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def save_checkpoint(+ self,+ epoch,+ step=None,+ model=None,+ optim=None,+ scheduler=None,+ scaler=None,+ step_in_epoch=None,+ **kwargs,+ ):+ """+ Saves a checkpoint containing the model's state, the optimizer's state,+ and the scheduler's state at the end of the given epoch. This method is+ intended to be called at the end of each epoch to save the training progress.++ Args:+ epoch (int): The epoch number at which the checkpoint is being saved.+ """++ step_in_epoch = None if step is None else step_in_epoch+ if self.rank == 0:+ logging.info(f"Save checkpoint: {epoch}, rank: {self.local_rank}\n")+ # self.step_or_epoch += 1+ state = {+ "epoch": epoch,+ "step": step,+ "total_step": self.batch_total,+ "state_dict": model.state_dict(),+ "optimizer": optim.state_dict(),+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/train_utils/trainer.py#L143" target="_blank" class="src-more">View full source →</a></details></div><div class="mblk"><code>.resume_checkpoint(model, optim, scheduler, scaler)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/train_utils/trainer.py#L260" target="_blank" class="gh-link">L260</a><div class="mdoc"><p>Resumes training from a checkpoint at the given file path.</p>+<p>Loads the model's state, the optimizer's state, and the scheduler's state.</p>+<h4>Args:</h4>+<ul>+<li><code>resume_path (str)</code> — The file path to the checkpoint to resume from.</li>+</ul>+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def resume_checkpoint(+ self,+ model=None,+ optim=None,+ scheduler=None,+ scaler=None,+ ):+ """+ Resumes training from a checkpoint at the given file path.+ Loads the model's state, the optimizer's state, and the scheduler's state.++ Args:+ resume_path (str): The file path to the checkpoint to resume from.+ """+ if self.resume:+ ckpt = os.path.join(self.output_dir, "model.pt")+ if os.path.isfile(ckpt):+ checkpoint = torch.load(ckpt, map_location="cpu")+ self.start_epoch = checkpoint["epoch"]+ # self.model.load_state_dict(checkpoint['state_dict'])+ src_state = checkpoint["state_dict"]+ dst_state = model.state_dict()+ for k in dst_state.keys():+ if not k.startswith("module.") and "module." + k in src_state.keys():+ k_ddp = "module." + k+ elif k.startswith("module.") and "module." + k not in src_state.keys():+ k_ddp = k.replace("module.", "", 1)+ else:+ k_ddp = k++</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/train_utils/trainer.py#L260" target="_blank" class="src-more">View full source →</a></details></div><div class="mblk"><code>.train_epoch(model, optim, scheduler, scaler, dataloader_train, dataloader_val, epoch, writer, **kwargs)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/train_utils/trainer.py#L340" target="_blank" class="gh-link">L340</a><div class="mdoc"><p>Defines the training process for a single epoch with gradient accumulation.</p>+<h4>Args:</h4>+<ul>+<li><code>epoch (int)</code> — The current epoch number.</li>+</ul>+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def train_epoch(+ self,+ model=None,+ optim=None,+ scheduler=None,+ scaler=None,+ dataloader_train=None,+ dataloader_val=None,+ epoch=None,+ writer=None,+ **kwargs,+ ):+ """+ Defines the training process for a single epoch with gradient accumulation.+ Args:+ epoch (int): The current epoch number.+ """+ if self.use_ddp or self.use_fsdp:+ dist.barrier()+ logging.info(f"Train epoch: {epoch}, rank: {self.rank}\n")+ model.train()++ # Set the number of steps for gradient accumulation+ accum_grad = self.accum_grad+ # Initialize the gradient accumulation+ optim.zero_grad()+ speed_stats = {}++ iterator_stop = torch.tensor(0).to(self.device)++</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/train_utils/trainer.py#L340" target="_blank" class="src-more">View full source →</a></details></div><div class="mblk"><code>.validate_epoch(model, dataloader_val, epoch, writer, **kwargs)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/train_utils/trainer.py#L537" target="_blank" class="gh-link">L537</a><div class="mdoc"><p>Defines the validation process for a single epoch.</p>+<p>Should be implemented with the actual model validation steps.</p>+<h4>Args:</h4>+<ul>+<li><code>epoch (int)</code> — The current epoch number.</li>+</ul>+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def validate_epoch(+ self,+ model=None,+ dataloader_val=None,+ epoch=None,+ writer=None,+ **kwargs,+ ):+ """+ Defines the validation process for a single epoch.+ Should be implemented with the actual model validation steps.++ Args:+ epoch (int): The current epoch number.+ """+ if self.use_ddp or self.use_fsdp:+ dist.barrier()+ logging.info(f"Validate epoch: {epoch}, rank: {self.rank}\n")+ model.eval()++ with torch.no_grad():++ speed_stats = {}+ time5 = time.perf_counter()+ iterator_stop = torch.tensor(0).to(self.device)+ dataloader_val.batch_sampler.set_epoch(epoch)+ for batch_idx, batch in enumerate(dataloader_val):+ if self.use_ddp or self.use_fsdp:+ dist.all_reduce(iterator_stop, dist.ReduceOp.SUM)+ if iterator_stop > 0:+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/train_utils/trainer.py#L537" target="_blank" class="src-more">View full source →</a></details></div><div class="mblk"><code>.log(epoch, batch_idx, step_in_epoch, batch_num_epoch, lr, loss, speed_stats, stats, writer, tag, data_split_i, data_split_num, log_step, **kwargs)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/train_utils/trainer.py#L651" target="_blank" class="gh-link">L651</a><div class="mdoc"><p>Log.</p>+<h4>Args:</h4>+<ul>+<li><code>epoch</code> — TODO.</li>+<li><code>batch_idx</code> — TODO.</li>+<li><code>step_in_epoch</code> — TODO.</li>+<li><code>batch_num_epoch</code> — TODO.</li>+<li><code>lr</code> — TODO.</li>+<li><code>loss</code> — TODO.</li>+<li><code>speed_stats</code> — TODO.</li>+<li><code>stats</code> — TODO.</li>+<li><code>writer</code> — TODO.</li>+<li><code>tag</code> — TODO.</li>+<li><code>data_split_i</code> — TODO.</li>+<li><code>data_split_num</code> — TODO.</li>+<li><code>log_step</code> — TODO.</li>+<li><code>**kwargs</code> — Additional keyword arguments.</li>+</ul>+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def log(+ self,+ epoch=0,+ batch_idx=0,+ step_in_epoch=0,+ batch_num_epoch=-1,+ lr=0.0,+ loss=0.0,+ speed_stats=None,+ stats=None,+ writer=None,+ tag="train",+ data_split_i=0,+ data_split_num=1,+ log_step=None,+ **kwargs,+ ):++ """Log.++ Args:+ epoch: TODO.+ batch_idx: TODO.+ step_in_epoch: TODO.+ batch_num_epoch: TODO.+ lr: TODO.+ loss: TODO.+ speed_stats: TODO.+ stats: TODO.+ writer: TODO.+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/train_utils/trainer.py#L651" target="_blank" class="src-more">View full source →</a></details></div><div class="mblk"><code>.close(writer)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/train_utils/trainer.py#L744" target="_blank" class="gh-link">L744</a><div class="mdoc"><p>Close.</p>+<h4>Args:</h4>+<ul>+<li><code>writer</code> — TODO.</li>+</ul>+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def close(self, writer=None):++ """Close.++ Args:+ writer: TODO.+ """+ if self.use_ddp or self.use_fsdp:+ dist.barrier()++ if writer is not None:+ writer.close()++ if self.use_ddp or self.use_fsdp:+ torch.distributed.destroy_process_group()+</code></pre></details></div></div>+<div class="api-detail" id="e503"><span class="dtag badge-m">method</span><h2><code>Trainer.save_checkpoint(epoch, step, model, optim, scheduler, scaler, step_in_epoch, **kwargs)</code></h2><div class="dmod">funasr.train_utils.trainer.Trainer · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/train_utils/trainer.py#L143" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Saves a checkpoint containing the model's state, the optimizer's state,</p>+<p>and the scheduler's state at the end of the given epoch. This method is</p>+<p>intended to be called at the end of each epoch to save the training progress.</p>+<h4>Args:</h4>+<ul>+<li><code>epoch (int)</code> — The epoch number at which the checkpoint is being saved.</li>+</ul>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def save_checkpoint(+ self,+ epoch,+ step=None,+ model=None,+ optim=None,+ scheduler=None,+ scaler=None,+ step_in_epoch=None,+ **kwargs,+ ):+ """+ Saves a checkpoint containing the model's state, the optimizer's state,+ and the scheduler's state at the end of the given epoch. This method is+ intended to be called at the end of each epoch to save the training progress.++ Args:+ epoch (int): The epoch number at which the checkpoint is being saved.+ """++ step_in_epoch = None if step is None else step_in_epoch+ if self.rank == 0:+ logging.info(f"Save checkpoint: {epoch}, rank: {self.local_rank}\n")+ # self.step_or_epoch += 1+ state = {+ "epoch": epoch,+ "step": step,+ "total_step": self.batch_total,+ "state_dict": model.state_dict(),+ "optimizer": optim.state_dict(),+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/train_utils/trainer.py#L143" target="_blank" class="src-more">View full source on GitHub →</a></details></div>+<div class="api-detail" id="e504"><span class="dtag badge-m">method</span><h2><code>Trainer.resume_checkpoint(model, optim, scheduler, scaler)</code></h2><div class="dmod">funasr.train_utils.trainer.Trainer · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/train_utils/trainer.py#L260" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Resumes training from a checkpoint at the given file path.</p>+<p>Loads the model's state, the optimizer's state, and the scheduler's state.</p>+<h4>Args:</h4>+<ul>+<li><code>resume_path (str)</code> — The file path to the checkpoint to resume from.</li>+</ul>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def resume_checkpoint(+ self,+ model=None,+ optim=None,+ scheduler=None,+ scaler=None,+ ):+ """+ Resumes training from a checkpoint at the given file path.+ Loads the model's state, the optimizer's state, and the scheduler's state.++ Args:+ resume_path (str): The file path to the checkpoint to resume from.+ """+ if self.resume:+ ckpt = os.path.join(self.output_dir, "model.pt")+ if os.path.isfile(ckpt):+ checkpoint = torch.load(ckpt, map_location="cpu")+ self.start_epoch = checkpoint["epoch"]+ # self.model.load_state_dict(checkpoint['state_dict'])+ src_state = checkpoint["state_dict"]+ dst_state = model.state_dict()+ for k in dst_state.keys():+ if not k.startswith("module.") and "module." + k in src_state.keys():+ k_ddp = "module." + k+ elif k.startswith("module.") and "module." + k not in src_state.keys():+ k_ddp = k.replace("module.", "", 1)+ else:+ k_ddp = k++</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/train_utils/trainer.py#L260" target="_blank" class="src-more">View full source on GitHub →</a></details></div>+<div class="api-detail" id="e505"><span class="dtag badge-m">method</span><h2><code>Trainer.train_epoch(model, optim, scheduler, scaler, dataloader_train, dataloader_val, epoch, writer, **kwargs)</code></h2><div class="dmod">funasr.train_utils.trainer.Trainer · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/train_utils/trainer.py#L340" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Defines the training process for a single epoch with gradient accumulation.</p>+<h4>Args:</h4>+<ul>+<li><code>epoch (int)</code> — The current epoch number.</li>+</ul>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def train_epoch(+ self,+ model=None,+ optim=None,+ scheduler=None,+ scaler=None,+ dataloader_train=None,+ dataloader_val=None,+ epoch=None,+ writer=None,+ **kwargs,+ ):+ """+ Defines the training process for a single epoch with gradient accumulation.+ Args:+ epoch (int): The current epoch number.+ """+ if self.use_ddp or self.use_fsdp:+ dist.barrier()+ logging.info(f"Train epoch: {epoch}, rank: {self.rank}\n")+ model.train()++ # Set the number of steps for gradient accumulation+ accum_grad = self.accum_grad+ # Initialize the gradient accumulation+ optim.zero_grad()+ speed_stats = {}++ iterator_stop = torch.tensor(0).to(self.device)++</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/train_utils/trainer.py#L340" target="_blank" class="src-more">View full source on GitHub →</a></details></div>+<div class="api-detail" id="e506"><span class="dtag badge-m">method</span><h2><code>Trainer.validate_epoch(model, dataloader_val, epoch, writer, **kwargs)</code></h2><div class="dmod">funasr.train_utils.trainer.Trainer · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/train_utils/trainer.py#L537" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Defines the validation process for a single epoch.</p>+<p>Should be implemented with the actual model validation steps.</p>+<h4>Args:</h4>+<ul>+<li><code>epoch (int)</code> — The current epoch number.</li>+</ul>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def validate_epoch(+ self,+ model=None,+ dataloader_val=None,+ epoch=None,+ writer=None,+ **kwargs,+ ):+ """+ Defines the validation process for a single epoch.+ Should be implemented with the actual model validation steps.++ Args:+ epoch (int): The current epoch number.+ """+ if self.use_ddp or self.use_fsdp:+ dist.barrier()+ logging.info(f"Validate epoch: {epoch}, rank: {self.rank}\n")+ model.eval()++ with torch.no_grad():++ speed_stats = {}+ time5 = time.perf_counter()+ iterator_stop = torch.tensor(0).to(self.device)+ dataloader_val.batch_sampler.set_epoch(epoch)+ for batch_idx, batch in enumerate(dataloader_val):+ if self.use_ddp or self.use_fsdp:+ dist.all_reduce(iterator_stop, dist.ReduceOp.SUM)+ if iterator_stop > 0:+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/train_utils/trainer.py#L537" target="_blank" class="src-more">View full source on GitHub →</a></details></div>+<div class="api-detail" id="e507"><span class="dtag badge-m">method</span><h2><code>Trainer.log(epoch, batch_idx, step_in_epoch, batch_num_epoch, lr, loss, speed_stats, stats, writer, tag, data_split_i, data_split_num, log_step, **kwargs)</code></h2><div class="dmod">funasr.train_utils.trainer.Trainer · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/train_utils/trainer.py#L651" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Log.</p>+<h4>Args:</h4>+<ul>+<li><code>epoch</code> — TODO.</li>+<li><code>batch_idx</code> — TODO.</li>+<li><code>step_in_epoch</code> — TODO.</li>+<li><code>batch_num_epoch</code> — TODO.</li>+<li><code>lr</code> — TODO.</li>+<li><code>loss</code> — TODO.</li>+<li><code>speed_stats</code> — TODO.</li>+<li><code>stats</code> — TODO.</li>+<li><code>writer</code> — TODO.</li>+<li><code>tag</code> — TODO.</li>+<li><code>data_split_i</code> — TODO.</li>+<li><code>data_split_num</code> — TODO.</li>+<li><code>log_step</code> — TODO.</li>+<li><code>**kwargs</code> — Additional keyword arguments.</li>+</ul>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def log(+ self,+ epoch=0,+ batch_idx=0,+ step_in_epoch=0,+ batch_num_epoch=-1,+ lr=0.0,+ loss=0.0,+ speed_stats=None,+ stats=None,+ writer=None,+ tag="train",+ data_split_i=0,+ data_split_num=1,+ log_step=None,+ **kwargs,+ ):++ """Log.++ Args:+ epoch: TODO.+ batch_idx: TODO.+ step_in_epoch: TODO.+ batch_num_epoch: TODO.+ lr: TODO.+ loss: TODO.+ speed_stats: TODO.+ stats: TODO.+ writer: TODO.+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/train_utils/trainer.py#L651" target="_blank" class="src-more">View full source on GitHub →</a></details></div>+<div class="api-detail" id="e508"><span class="dtag badge-m">method</span><h2><code>Trainer.close(writer)</code></h2><div class="dmod">funasr.train_utils.trainer.Trainer · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/train_utils/trainer.py#L744" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Close.</p>+<h4>Args:</h4>+<ul>+<li><code>writer</code> — TODO.</li>+</ul>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def close(self, writer=None):++ """Close.++ Args:+ writer: TODO.+ """+ if self.use_ddp or self.use_fsdp:+ dist.barrier()++ if writer is not None:+ writer.close()++ if self.use_ddp or self.use_fsdp:+ torch.distributed.destroy_process_group()+</code></pre></details></div>+<div class="api-detail" id="e509"><span class="dtag badge-f">function</span><h2><code>maybe_autocast(dtype, use_deepspeed)</code></h2><div class="dmod">funasr.train_utils.trainer_ds · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/train_utils/trainer_ds.py#L27" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Maybe autocast.</p>+<h4>Args:</h4>+<ul>+<li><code>dtype</code> — TODO.</li>+<li><code>use_deepspeed</code> — TODO.</li>+</ul>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code>def maybe_autocast(dtype=None, use_deepspeed=False):+ """Maybe autocast.++ Args:+ dtype: TODO.+ use_deepspeed: TODO.+ """+ if use_deepspeed:+ with torch.cuda.amp.autocast(enabled=True, dtype=dtype, cache_enabled=False):+ yield+ else:+ if dtype == torch.float16 or dtype == torch.bfloat16:+ with autocast(enabled=True, dtype=dtype):+ yield+ else:+ yield+</code></pre></details></div>+<div class="api-detail" id="e510"><span class="dtag badge-c">class</span><h2>Trainer</h2><div class="dmod">funasr.train_utils.trainer_ds · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/train_utils/trainer_ds.py#L45" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>A simple trainer class for training a PyTorch model, saving checkpoints at the end of each epoch,</p>+<p>and optionally resuming from a saved checkpoint.</p>+<p>Attributes:</p>+<ul>+<li><code>max_epoch (int)</code> — Maximum number of epochs for training.</li>+<li><code>model (torch.nn.Module)</code> — The model to be trained.</li>+<li><code>optim (torch.optim.Optimizer)</code> — The optimizer to use for training.</li>+<li><code>scheduler (torch.optim.lr_scheduler._LRScheduler)</code> — The learning rate scheduler.</li>+<li><code>dataloader_train (torch.utils.data.DataLoader)</code> — DataLoader for the training dataset.</li>+<li><code>dataloader_val (torch.utils.data.DataLoader)</code> — DataLoader for the validation dataset.</li>+<li><code>output_dir (str)</code> — Directory where model checkpoints will be saved.</li>+<li><code>resume (str, optional)</code> — Path to a checkpoint to resume training from.</li>+</ul>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code>class Trainer:+ """+ A simple trainer class for training a PyTorch model, saving checkpoints at the end of each epoch,+ and optionally resuming from a saved checkpoint.++ Attributes:+ max_epoch (int): Maximum number of epochs for training.+ model (torch.nn.Module): The model to be trained.+ optim (torch.optim.Optimizer): The optimizer to use for training.+ scheduler (torch.optim.lr_scheduler._LRScheduler): The learning rate scheduler.+ dataloader_train (torch.utils.data.DataLoader): DataLoader for the training dataset.+ dataloader_val (torch.utils.data.DataLoader): DataLoader for the validation dataset.+ output_dir (str): Directory where model checkpoints will be saved.+ resume (str, optional): Path to a checkpoint to resume training from.+ """++ def __init__(+ self,+ rank=0,+ local_rank=0,+ world_size=1,+ use_ddp: bool = False,+ use_fsdp: bool = False,+ use_fp16: bool = False,+ use_bf16: bool = False,+ use_deepspeed: bool = False,+ output_dir: str = "./",+ **kwargs,+ ):+ """+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/train_utils/trainer_ds.py#L45" target="_blank" class="src-more">View full source on GitHub →</a></details><h3 class="mt">Methods</h3><div class="mblk"><code>.save_checkpoint(epoch, step, model, optim, scheduler, scaler, step_in_epoch, **kwargs)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/train_utils/trainer_ds.py#L171" target="_blank" class="gh-link">L171</a><div class="mdoc"><p>Saves a checkpoint containing the model's state, the optimizer's state,</p>+<p>and the scheduler's state at the end of the given epoch. This method is</p>+<p>intended to be called at the end of each epoch to save the training progress.</p>+<h4>Args:</h4>+<ul>+<li><code>epoch (int)</code> — The epoch number at which the checkpoint is being saved.</li>+</ul>+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def save_checkpoint(+ self,+ epoch,+ step=None,+ model=None,+ optim=None,+ scheduler=None,+ scaler=None,+ step_in_epoch=None,+ **kwargs,+ ):+ """+ Saves a checkpoint containing the model's state, the optimizer's state,+ and the scheduler's state at the end of the given epoch. This method is+ intended to be called at the end of each epoch to save the training progress.++ Args:+ epoch (int): The epoch number at which the checkpoint is being saved.+ """+ if self.use_ddp or self.use_fsdp:+ dist.barrier()+ step_in_epoch = None if step is None else step_in_epoch+ if self.use_deepspeed:++ logging.info(f"Save checkpoint: {epoch}, rank: {self.local_rank}\n")+ # self.step_or_epoch += 1+ state = {+ "epoch": epoch,+ # "state_dict": model.state_dict(),+ # "optimizer": optim.state_dict(),+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/train_utils/trainer_ds.py#L171" target="_blank" class="src-more">View full source →</a></details></div><div class="mblk"><code>.resume_checkpoint(model, optim, scheduler, scaler)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/train_utils/trainer_ds.py#L414" target="_blank" class="gh-link">L414</a><div class="mdoc"><p>Resumes training from a checkpoint at the given file path.</p>+<p>Loads the model's state, the optimizer's state, and the scheduler's state.</p>+<h4>Args:</h4>+<ul>+<li><code>resume_path (str)</code> — The file path to the checkpoint to resume from.</li>+</ul>+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def resume_checkpoint(+ self,+ model=None,+ optim=None,+ scheduler=None,+ scaler=None,+ ):+ """+ Resumes training from a checkpoint at the given file path.+ Loads the model's state, the optimizer's state, and the scheduler's state.++ Args:+ resume_path (str): The file path to the checkpoint to resume from.+ """+ if self.resume:++ if self.use_deepspeed:+ ckpt = os.path.join(self.output_dir, "model.pt")+ if os.path.exists(ckpt):+ _, checkpoint = model.load_checkpoint(self.output_dir, "model.pt")+ self.start_epoch = checkpoint["epoch"]+ self.saved_ckpts = checkpoint["saved_ckpts"]+ self.val_acc_step_or_epoch = (+ checkpoint["val_acc_step_or_epoch"]+ if "val_acc_step_or_epoch" in checkpoint+ else {}+ )+ self.val_loss_step_or_epoch = (+ checkpoint["val_loss_step_or_epoch"]+ if "val_loss_step_or_epoch" in checkpoint+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/train_utils/trainer_ds.py#L414" target="_blank" class="src-more">View full source →</a></details></div><div class="mblk"><code>.train_epoch(model, optim, scheduler, scaler, dataloader_train, dataloader_val, epoch, **kwargs)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/train_utils/trainer_ds.py#L554" target="_blank" class="gh-link">L554</a><div class="mdoc"><p>Defines the training process for a single epoch with gradient accumulation.</p>+<h4>Args:</h4>+<ul>+<li><code>epoch (int)</code> — The current epoch number.</li>+</ul>+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def train_epoch(+ self,+ model=None,+ optim=None,+ scheduler=None,+ scaler=None,+ dataloader_train=None,+ dataloader_val=None,+ epoch=None,+ **kwargs,+ ):+ """+ Defines the training process for a single epoch with gradient accumulation.+ Args:+ epoch (int): The current epoch number.+ """+ if self.use_ddp or self.use_fsdp or self.use_deepspeed:+ dist.barrier()+ logging.info(f"Train epoch: {epoch}, rank: {self.rank}\n")+ model.train()++ # Set the number of steps for gradient accumulation+ accum_grad = self.accum_grad+ # Initialize the gradient accumulation+ optim.zero_grad()+ speed_stats = {}++ iterator_stop = torch.tensor(0).to(self.device)++ dataloader_train.batch_sampler.set_epoch(epoch)+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/train_utils/trainer_ds.py#L554" target="_blank" class="src-more">View full source →</a></details></div><div class="mblk"><code>.forward_step(model, batch, loss_dict)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/train_utils/trainer_ds.py#L676" target="_blank" class="gh-link">L676</a><div class="mdoc"><p>Forward step.</p>+<h4>Args:</h4>+<ul>+<li><code>model</code> — Model instance or model name.</li>+<li><code>batch</code> — TODO.</li>+<li><code>loss_dict</code> — TODO.</li>+</ul>+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def forward_step(self, model, batch, loss_dict={}):+ """Forward step.++ Args:+ model: Model instance or model name.+ batch: TODO.+ loss_dict: TODO.+ """+ with maybe_autocast(dtype=self.dtype, use_deepspeed=self.use_deepspeed):+ retval = model(**batch)++ loss, stats, weight = retval+ stats = {k: v for k, v in stats.items() if v is not None}++ loss_dict["loss"] = loss+ loss_dict["stats"] = stats+ loss_dict["weight"] = weight+</code></pre></details></div><div class="mblk"><code>.backward_step(model, scaler, loss_dict)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/train_utils/trainer_ds.py#L694" target="_blank" class="gh-link">L694</a><div class="mdoc"><p>Backward step.</p>+<h4>Args:</h4>+<ul>+<li><code>model</code> — Model instance or model name.</li>+<li><code>scaler</code> — TODO.</li>+<li><code>loss_dict</code> — TODO.</li>+</ul>+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def backward_step(self, model, scaler, loss_dict={}):+ """Backward step.++ Args:+ model: Model instance or model name.+ scaler: TODO.+ loss_dict: TODO.+ """+ loss = loss_dict["loss"]++ if self.use_deepspeed:+ scaled_loss = model.backward(loss)+ else:+ loss = loss / self.accum_grad+ if scaler:+ scaler.scale(loss).backward()+ else:+ loss.backward()+</code></pre></details></div><div class="mblk"><code>.update_step(model, optim, scheduler, scaler, loss_dict)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/train_utils/trainer_ds.py#L713" target="_blank" class="gh-link">L713</a><div class="mdoc"><p>Update step.</p>+<h4>Args:</h4>+<ul>+<li><code>model</code> — Model instance or model name.</li>+<li><code>optim</code> — TODO.</li>+<li><code>scheduler</code> — TODO.</li>+<li><code>scaler</code> — TODO.</li>+<li><code>loss_dict</code> — TODO.</li>+</ul>+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def update_step(self, model, optim, scheduler, scaler, loss_dict=None):+ """Update step.++ Args:+ model: Model instance or model name.+ optim: TODO.+ scheduler: TODO.+ scaler: TODO.+ loss_dict: TODO.+ """+ batch_idx = loss_dict["batch_idx"]+ if self.use_deepspeed:+ model.step()+ else:+ if (batch_idx + 1) % self.accum_grad == 0:+ # Perform gradient clipping if it is set+ if self.grad_clip > 0:+ grad_norm = torch.nn.utils.clip_grad_norm_(+ model.parameters(),+ max_norm=self.grad_clip,+ norm_type=self.grad_clip_type,+ )+ if not torch.isfinite(grad_norm):+ logging.warning(+ f"The grad norm is {grad_norm}. Skipping updating the model."+ )+ optim.zero_grad() # Reset gradients+ return++ # Execute an optimization step (update model parameters)+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/train_utils/trainer_ds.py#L713" target="_blank" class="src-more">View full source →</a></details></div><div class="mblk"><code>.validate_epoch(model, dataloader_val, epoch, writer, **kwargs)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/train_utils/trainer_ds.py#L754" target="_blank" class="gh-link">L754</a><div class="mdoc"><p>Defines the validation process for a single epoch.</p>+<p>Should be implemented with the actual model validation steps.</p>+<h4>Args:</h4>+<ul>+<li><code>epoch (int)</code> — The current epoch number.</li>+</ul>+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def validate_epoch(+ self,+ model=None,+ dataloader_val=None,+ epoch=None,+ writer=None,+ **kwargs,+ ):+ """+ Defines the validation process for a single epoch.+ Should be implemented with the actual model validation steps.++ Args:+ epoch (int): The current epoch number.+ """+ self.val_loss_avg = 0.0+ self.val_acc_avg = 0.0++ if self.use_ddp or self.use_fsdp or self.use_deepspeed:+ dist.barrier()+ logging.info(f"Validate epoch: {epoch}, rank: {self.rank}\n")+ model.eval()++ with torch.no_grad():++ speed_stats = {}+ time_beg = time.perf_counter()+ time5 = time_beg++ dataloader_val.batch_sampler.set_epoch(epoch)+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/train_utils/trainer_ds.py#L754" target="_blank" class="src-more">View full source →</a></details></div><div class="mblk"><code>.log(loss_dict, tag, **kwargs)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/train_utils/trainer_ds.py#L848" target="_blank" class="gh-link">L848</a><div class="mdoc"><p>Log.</p>+<h4>Args:</h4>+<ul>+<li><code>loss_dict</code> — TODO.</li>+<li><code>tag</code> — TODO.</li>+<li><code>**kwargs</code> — Additional keyword arguments.</li>+</ul>+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def log(+ self,+ loss_dict: dict = None,+ tag="train",+ **kwargs,+ ):+ """Log.++ Args:+ loss_dict: TODO.+ tag: TODO.+ **kwargs: Additional keyword arguments.+ """+ loss = loss_dict["loss"].detach().cpu().item()+ epoch = loss_dict["epoch"]+ batch_idx = loss_dict["batch_idx"]+ step_in_epoch = loss_dict["step_in_epoch"]+ batch_total = loss_dict["batch_total"]+ batch_num_epoch = loss_dict["batch_num_epoch"]+ lr = loss_dict["lr"]++ speed_stats = loss_dict["speed_stats"]+ stats = loss_dict["stats"]+ data_split_i = loss_dict["data_split_i"]+ data_split_num = loss_dict["data_split_num"]+ log_step = loss_dict.get("log_step", None)++ if (batch_idx + 1) % self.log_interval == 0:+ batch_idx = log_step if log_step is not None else batch_idx+ gpu_info = (+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/train_utils/trainer_ds.py#L848" target="_blank" class="src-more">View full source →</a></details></div><div class="mblk"><code>.close(writer)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/train_utils/trainer_ds.py#L929" target="_blank" class="gh-link">L929</a><div class="mdoc"><p>Close.</p>+<h4>Args:</h4>+<ul>+<li><code>writer</code> — TODO.</li>+</ul>+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def close(self, writer=None):++ """Close.++ Args:+ writer: TODO.+ """+ if self.use_ddp or self.use_fsdp:+ dist.barrier()++ if writer is not None:+ writer.close()++ if self.use_ddp or self.use_fsdp:+ torch.distributed.destroy_process_group()+</code></pre></details></div><div class="mblk"><code>.warp_model(model, **kwargs)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/train_utils/trainer_ds.py#L945" target="_blank" class="gh-link">L945</a><div class="mdoc"><p>Warp model.</p>+<h4>Args:</h4>+<ul>+<li><code>model</code> — Model instance or model name.</li>+<li><code>**kwargs</code> — Additional keyword arguments.</li>+</ul>+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def warp_model(self, model, **kwargs):++ """Warp model.++ Args:+ model: Model instance or model name.+ **kwargs: Additional keyword arguments.+ """+ if self.use_deepspeed:+ from deepspeed.runtime.zero.stage_1_and_2 import (+ estimate_zero2_model_states_mem_needs_all_live,+ )+ from deepspeed.runtime.zero.stage3 import estimate_zero3_model_states_mem_needs_all_live+ from deepspeed.utils.zero_to_fp32 import convert_zero_checkpoint_to_fp32_state_dict++ local_world_size = int(os.environ.get("LOCAL_WORLD_SIZE", 1))+ world_size = int(os.environ.get("WORLD_SIZE", 1))++ # NOTE(xcsong): look in detail how the memory estimator API works:+ # https://deepspeed.readthedocs.io/en/latest/memory.html#discussion+ if int(os.environ.get("RANK", 0)) == 0:+ logging.info("Estimating model states memory needs (zero2)...")+ estimate_zero2_model_states_mem_needs_all_live(+ model,+ num_gpus_per_node=local_world_size,+ num_nodes=world_size // local_world_size,+ )+ logging.info("Estimating model states memory needs (zero3)...")+ estimate_zero3_model_states_mem_needs_all_live(+ model,+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/train_utils/trainer_ds.py#L945" target="_blank" class="src-more">View full source →</a></details></div><div class="mblk"><code>.warp_optim_scheduler(model, **kwargs)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/train_utils/trainer_ds.py#L997" target="_blank" class="gh-link">L997</a><div class="mdoc"><p>Warp optim scheduler.</p>+<h4>Args:</h4>+<ul>+<li><code>model</code> — Model instance or model name.</li>+<li><code>**kwargs</code> — Additional keyword arguments.</li>+</ul>+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def warp_optim_scheduler(self, model, **kwargs):+ """Warp optim scheduler.++ Args:+ model: Model instance or model name.+ **kwargs: Additional keyword arguments.+ """+ from funasr.optimizers import optim_classes+ from funasr.schedulers import scheduler_classes+ from omegaconf import OmegaConf, DictConfig+ import json++ # optim+ logging.info("Build optim")+ optim = kwargs.get("optim", "adam")+ assert optim in optim_classes+ optim_class = optim_classes.get(optim)+ optim = optim_class(model.parameters(), **kwargs.get("optim_conf"))++ # scheduler+ logging.info("Build scheduler")+ scheduler = kwargs.get("scheduler", "warmuplr")+ assert scheduler in scheduler_classes+ scheduler_class = scheduler_classes.get(scheduler)+ scheduler = scheduler_class(optim, **kwargs.get("scheduler_conf"))++ if self.use_deepspeed:+ import deepspeed++ args = OmegaConf.create({"deepspeed_config": self.deepspeed_config})+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/train_utils/trainer_ds.py#L997" target="_blank" class="src-more">View full source →</a></details></div></div>+<div class="api-detail" id="e511"><span class="dtag badge-m">method</span><h2><code>Trainer.save_checkpoint(epoch, step, model, optim, scheduler, scaler, step_in_epoch, **kwargs)</code></h2><div class="dmod">funasr.train_utils.trainer_ds.Trainer · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/train_utils/trainer_ds.py#L171" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Saves a checkpoint containing the model's state, the optimizer's state,</p>+<p>and the scheduler's state at the end of the given epoch. This method is</p>+<p>intended to be called at the end of each epoch to save the training progress.</p>+<h4>Args:</h4>+<ul>+<li><code>epoch (int)</code> — The epoch number at which the checkpoint is being saved.</li>+</ul>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def save_checkpoint(+ self,+ epoch,+ step=None,+ model=None,+ optim=None,+ scheduler=None,+ scaler=None,+ step_in_epoch=None,+ **kwargs,+ ):+ """+ Saves a checkpoint containing the model's state, the optimizer's state,+ and the scheduler's state at the end of the given epoch. This method is+ intended to be called at the end of each epoch to save the training progress.++ Args:+ epoch (int): The epoch number at which the checkpoint is being saved.+ """+ if self.use_ddp or self.use_fsdp:+ dist.barrier()+ step_in_epoch = None if step is None else step_in_epoch+ if self.use_deepspeed:++ logging.info(f"Save checkpoint: {epoch}, rank: {self.local_rank}\n")+ # self.step_or_epoch += 1+ state = {+ "epoch": epoch,+ # "state_dict": model.state_dict(),+ # "optimizer": optim.state_dict(),+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/train_utils/trainer_ds.py#L171" target="_blank" class="src-more">View full source on GitHub →</a></details></div>+<div class="api-detail" id="e512"><span class="dtag badge-m">method</span><h2><code>Trainer.resume_checkpoint(model, optim, scheduler, scaler)</code></h2><div class="dmod">funasr.train_utils.trainer_ds.Trainer · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/train_utils/trainer_ds.py#L414" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Resumes training from a checkpoint at the given file path.</p>+<p>Loads the model's state, the optimizer's state, and the scheduler's state.</p>+<h4>Args:</h4>+<ul>+<li><code>resume_path (str)</code> — The file path to the checkpoint to resume from.</li>+</ul>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def resume_checkpoint(+ self,+ model=None,+ optim=None,+ scheduler=None,+ scaler=None,+ ):+ """+ Resumes training from a checkpoint at the given file path.+ Loads the model's state, the optimizer's state, and the scheduler's state.++ Args:+ resume_path (str): The file path to the checkpoint to resume from.+ """+ if self.resume:++ if self.use_deepspeed:+ ckpt = os.path.join(self.output_dir, "model.pt")+ if os.path.exists(ckpt):+ _, checkpoint = model.load_checkpoint(self.output_dir, "model.pt")+ self.start_epoch = checkpoint["epoch"]+ self.saved_ckpts = checkpoint["saved_ckpts"]+ self.val_acc_step_or_epoch = (+ checkpoint["val_acc_step_or_epoch"]+ if "val_acc_step_or_epoch" in checkpoint+ else {}+ )+ self.val_loss_step_or_epoch = (+ checkpoint["val_loss_step_or_epoch"]+ if "val_loss_step_or_epoch" in checkpoint+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/train_utils/trainer_ds.py#L414" target="_blank" class="src-more">View full source on GitHub →</a></details></div>+<div class="api-detail" id="e513"><span class="dtag badge-m">method</span><h2><code>Trainer.train_epoch(model, optim, scheduler, scaler, dataloader_train, dataloader_val, epoch, **kwargs)</code></h2><div class="dmod">funasr.train_utils.trainer_ds.Trainer · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/train_utils/trainer_ds.py#L554" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Defines the training process for a single epoch with gradient accumulation.</p>+<h4>Args:</h4>+<ul>+<li><code>epoch (int)</code> — The current epoch number.</li>+</ul>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def train_epoch(+ self,+ model=None,+ optim=None,+ scheduler=None,+ scaler=None,+ dataloader_train=None,+ dataloader_val=None,+ epoch=None,+ **kwargs,+ ):+ """+ Defines the training process for a single epoch with gradient accumulation.+ Args:+ epoch (int): The current epoch number.+ """+ if self.use_ddp or self.use_fsdp or self.use_deepspeed:+ dist.barrier()+ logging.info(f"Train epoch: {epoch}, rank: {self.rank}\n")+ model.train()++ # Set the number of steps for gradient accumulation+ accum_grad = self.accum_grad+ # Initialize the gradient accumulation+ optim.zero_grad()+ speed_stats = {}++ iterator_stop = torch.tensor(0).to(self.device)++ dataloader_train.batch_sampler.set_epoch(epoch)+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/train_utils/trainer_ds.py#L554" target="_blank" class="src-more">View full source on GitHub →</a></details></div>+<div class="api-detail" id="e514"><span class="dtag badge-m">method</span><h2><code>Trainer.forward_step(model, batch, loss_dict)</code></h2><div class="dmod">funasr.train_utils.trainer_ds.Trainer · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/train_utils/trainer_ds.py#L676" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Forward step.</p>+<h4>Args:</h4>+<ul>+<li><code>model</code> — Model instance or model name.</li>+<li><code>batch</code> — TODO.</li>+<li><code>loss_dict</code> — TODO.</li>+</ul>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def forward_step(self, model, batch, loss_dict={}):+ """Forward step.++ Args:+ model: Model instance or model name.+ batch: TODO.+ loss_dict: TODO.+ """+ with maybe_autocast(dtype=self.dtype, use_deepspeed=self.use_deepspeed):+ retval = model(**batch)++ loss, stats, weight = retval+ stats = {k: v for k, v in stats.items() if v is not None}++ loss_dict["loss"] = loss+ loss_dict["stats"] = stats+ loss_dict["weight"] = weight+</code></pre></details></div>+<div class="api-detail" id="e515"><span class="dtag badge-m">method</span><h2><code>Trainer.backward_step(model, scaler, loss_dict)</code></h2><div class="dmod">funasr.train_utils.trainer_ds.Trainer · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/train_utils/trainer_ds.py#L694" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Backward step.</p>+<h4>Args:</h4>+<ul>+<li><code>model</code> — Model instance or model name.</li>+<li><code>scaler</code> — TODO.</li>+<li><code>loss_dict</code> — TODO.</li>+</ul>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def backward_step(self, model, scaler, loss_dict={}):+ """Backward step.++ Args:+ model: Model instance or model name.+ scaler: TODO.+ loss_dict: TODO.+ """+ loss = loss_dict["loss"]++ if self.use_deepspeed:+ scaled_loss = model.backward(loss)+ else:+ loss = loss / self.accum_grad+ if scaler:+ scaler.scale(loss).backward()+ else:+ loss.backward()+</code></pre></details></div>+<div class="api-detail" id="e516"><span class="dtag badge-m">method</span><h2><code>Trainer.update_step(model, optim, scheduler, scaler, loss_dict)</code></h2><div class="dmod">funasr.train_utils.trainer_ds.Trainer · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/train_utils/trainer_ds.py#L713" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Update step.</p>+<h4>Args:</h4>+<ul>+<li><code>model</code> — Model instance or model name.</li>+<li><code>optim</code> — TODO.</li>+<li><code>scheduler</code> — TODO.</li>+<li><code>scaler</code> — TODO.</li>+<li><code>loss_dict</code> — TODO.</li>+</ul>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def update_step(self, model, optim, scheduler, scaler, loss_dict=None):+ """Update step.++ Args:+ model: Model instance or model name.+ optim: TODO.+ scheduler: TODO.+ scaler: TODO.+ loss_dict: TODO.+ """+ batch_idx = loss_dict["batch_idx"]+ if self.use_deepspeed:+ model.step()+ else:+ if (batch_idx + 1) % self.accum_grad == 0:+ # Perform gradient clipping if it is set+ if self.grad_clip > 0:+ grad_norm = torch.nn.utils.clip_grad_norm_(+ model.parameters(),+ max_norm=self.grad_clip,+ norm_type=self.grad_clip_type,+ )+ if not torch.isfinite(grad_norm):+ logging.warning(+ f"The grad norm is {grad_norm}. Skipping updating the model."+ )+ optim.zero_grad() # Reset gradients+ return++ # Execute an optimization step (update model parameters)+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/train_utils/trainer_ds.py#L713" target="_blank" class="src-more">View full source on GitHub →</a></details></div>+<div class="api-detail" id="e517"><span class="dtag badge-m">method</span><h2><code>Trainer.validate_epoch(model, dataloader_val, epoch, writer, **kwargs)</code></h2><div class="dmod">funasr.train_utils.trainer_ds.Trainer · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/train_utils/trainer_ds.py#L754" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Defines the validation process for a single epoch.</p>+<p>Should be implemented with the actual model validation steps.</p>+<h4>Args:</h4>+<ul>+<li><code>epoch (int)</code> — The current epoch number.</li>+</ul>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def validate_epoch(+ self,+ model=None,+ dataloader_val=None,+ epoch=None,+ writer=None,+ **kwargs,+ ):+ """+ Defines the validation process for a single epoch.+ Should be implemented with the actual model validation steps.++ Args:+ epoch (int): The current epoch number.+ """+ self.val_loss_avg = 0.0+ self.val_acc_avg = 0.0++ if self.use_ddp or self.use_fsdp or self.use_deepspeed:+ dist.barrier()+ logging.info(f"Validate epoch: {epoch}, rank: {self.rank}\n")+ model.eval()++ with torch.no_grad():++ speed_stats = {}+ time_beg = time.perf_counter()+ time5 = time_beg++ dataloader_val.batch_sampler.set_epoch(epoch)+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/train_utils/trainer_ds.py#L754" target="_blank" class="src-more">View full source on GitHub →</a></details></div>+<div class="api-detail" id="e518"><span class="dtag badge-m">method</span><h2><code>Trainer.log(loss_dict, tag, **kwargs)</code></h2><div class="dmod">funasr.train_utils.trainer_ds.Trainer · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/train_utils/trainer_ds.py#L848" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Log.</p>+<h4>Args:</h4>+<ul>+<li><code>loss_dict</code> — TODO.</li>+<li><code>tag</code> — TODO.</li>+<li><code>**kwargs</code> — Additional keyword arguments.</li>+</ul>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def log(+ self,+ loss_dict: dict = None,+ tag="train",+ **kwargs,+ ):+ """Log.++ Args:+ loss_dict: TODO.+ tag: TODO.+ **kwargs: Additional keyword arguments.+ """+ loss = loss_dict["loss"].detach().cpu().item()+ epoch = loss_dict["epoch"]+ batch_idx = loss_dict["batch_idx"]+ step_in_epoch = loss_dict["step_in_epoch"]+ batch_total = loss_dict["batch_total"]+ batch_num_epoch = loss_dict["batch_num_epoch"]+ lr = loss_dict["lr"]++ speed_stats = loss_dict["speed_stats"]+ stats = loss_dict["stats"]+ data_split_i = loss_dict["data_split_i"]+ data_split_num = loss_dict["data_split_num"]+ log_step = loss_dict.get("log_step", None)++ if (batch_idx + 1) % self.log_interval == 0:+ batch_idx = log_step if log_step is not None else batch_idx+ gpu_info = (+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/train_utils/trainer_ds.py#L848" target="_blank" class="src-more">View full source on GitHub →</a></details></div>+<div class="api-detail" id="e519"><span class="dtag badge-m">method</span><h2><code>Trainer.close(writer)</code></h2><div class="dmod">funasr.train_utils.trainer_ds.Trainer · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/train_utils/trainer_ds.py#L929" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Close.</p>+<h4>Args:</h4>+<ul>+<li><code>writer</code> — TODO.</li>+</ul>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def close(self, writer=None):++ """Close.++ Args:+ writer: TODO.+ """+ if self.use_ddp or self.use_fsdp:+ dist.barrier()++ if writer is not None:+ writer.close()++ if self.use_ddp or self.use_fsdp:+ torch.distributed.destroy_process_group()+</code></pre></details></div>+<div class="api-detail" id="e520"><span class="dtag badge-m">method</span><h2><code>Trainer.warp_model(model, **kwargs)</code></h2><div class="dmod">funasr.train_utils.trainer_ds.Trainer · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/train_utils/trainer_ds.py#L945" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Warp model.</p>+<h4>Args:</h4>+<ul>+<li><code>model</code> — Model instance or model name.</li>+<li><code>**kwargs</code> — Additional keyword arguments.</li>+</ul>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def warp_model(self, model, **kwargs):++ """Warp model.++ Args:+ model: Model instance or model name.+ **kwargs: Additional keyword arguments.+ """+ if self.use_deepspeed:+ from deepspeed.runtime.zero.stage_1_and_2 import (+ estimate_zero2_model_states_mem_needs_all_live,+ )+ from deepspeed.runtime.zero.stage3 import estimate_zero3_model_states_mem_needs_all_live+ from deepspeed.utils.zero_to_fp32 import convert_zero_checkpoint_to_fp32_state_dict++ local_world_size = int(os.environ.get("LOCAL_WORLD_SIZE", 1))+ world_size = int(os.environ.get("WORLD_SIZE", 1))++ # NOTE(xcsong): look in detail how the memory estimator API works:+ # https://deepspeed.readthedocs.io/en/latest/memory.html#discussion+ if int(os.environ.get("RANK", 0)) == 0:+ logging.info("Estimating model states memory needs (zero2)...")+ estimate_zero2_model_states_mem_needs_all_live(+ model,+ num_gpus_per_node=local_world_size,+ num_nodes=world_size // local_world_size,+ )+ logging.info("Estimating model states memory needs (zero3)...")+ estimate_zero3_model_states_mem_needs_all_live(+ model,+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/train_utils/trainer_ds.py#L945" target="_blank" class="src-more">View full source on GitHub →</a></details></div>+<div class="api-detail" id="e521"><span class="dtag badge-m">method</span><h2><code>Trainer.warp_optim_scheduler(model, **kwargs)</code></h2><div class="dmod">funasr.train_utils.trainer_ds.Trainer · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/train_utils/trainer_ds.py#L997" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Warp optim scheduler.</p>+<h4>Args:</h4>+<ul>+<li><code>model</code> — Model instance or model name.</li>+<li><code>**kwargs</code> — Additional keyword arguments.</li>+</ul>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def warp_optim_scheduler(self, model, **kwargs):+ """Warp optim scheduler.++ Args:+ model: Model instance or model name.+ **kwargs: Additional keyword arguments.+ """+ from funasr.optimizers import optim_classes+ from funasr.schedulers import scheduler_classes+ from omegaconf import OmegaConf, DictConfig+ import json++ # optim+ logging.info("Build optim")+ optim = kwargs.get("optim", "adam")+ assert optim in optim_classes+ optim_class = optim_classes.get(optim)+ optim = optim_class(model.parameters(), **kwargs.get("optim_conf"))++ # scheduler+ logging.info("Build scheduler")+ scheduler = kwargs.get("scheduler", "warmuplr")+ assert scheduler in scheduler_classes+ scheduler_class = scheduler_classes.get(scheduler)+ scheduler = scheduler_class(optim, **kwargs.get("scheduler_conf"))++ if self.use_deepspeed:+ import deepspeed++ args = OmegaConf.create({"deepspeed_config": self.deepspeed_config})+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/train_utils/trainer_ds.py#L997" target="_blank" class="src-more">View full source on GitHub →</a></details></div>+<div class="api-detail" id="e522"><span class="dtag badge-c">class</span><h2>AudioDataset</h2><div class="dmod">funasr.datasets.datasets · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/datasets/audio_datasets/datasets.py#L10" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>AudioDataset</p>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code>class AudioDataset(torch.utils.data.Dataset):+ """+ AudioDataset+ """++ def __init__(+ self,+ path,+ index_ds: str = None,+ frontend=None,+ tokenizer=None,+ is_training: bool = True,+ int_pad_value: int = -1,+ float_pad_value: float = 0.0,+ **kwargs,+ ):+ """Initialize AudioDataset.++ Args:+ path: TODO.+ index_ds: TODO.+ frontend: Audio frontend for feature extraction.+ tokenizer: Tokenizer instance for text encoding/decoding.+ is_training: Boolean flag for training.+ int_pad_value: TODO.+ float_pad_value: TODO.+ **kwargs: Additional keyword arguments.+ """+ super().__init__()+ index_ds_class = tables.index_ds_classes.get(index_ds)+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/datasets/audio_datasets/datasets.py#L10" target="_blank" class="src-more">View full source on GitHub →</a></details><h3 class="mt">Methods</h3><div class="mblk"><code>.get_source_len(index)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/datasets/audio_datasets/datasets.py#L67" target="_blank" class="gh-link">L67</a><div class="mdoc"><p>Get source len.</p>+<h4>Args:</h4>+<ul>+<li><code>index</code> — TODO.</li>+</ul>+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def get_source_len(self, index):+ """Get source len.++ Args:+ index: TODO.+ """+ item = self.index_ds[index]+ return self.index_ds.get_source_len(item)+</code></pre></details></div><div class="mblk"><code>.get_target_len(index)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/datasets/audio_datasets/datasets.py#L76" target="_blank" class="gh-link">L76</a><div class="mdoc"><p>Get target len.</p>+<h4>Args:</h4>+<ul>+<li><code>index</code> — TODO.</li>+</ul>+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def get_target_len(self, index):+ """Get target len.++ Args:+ index: TODO.+ """+ item = self.index_ds[index]+ return self.index_ds.get_target_len(item)+</code></pre></details></div><div class="mblk"><code>.collator(samples)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/datasets/audio_datasets/datasets.py#L127" target="_blank" class="gh-link">L127</a><div class="mdoc"><p>Collator.</p>+<h4>Args:</h4>+<ul>+<li><code>samples</code> — TODO.</li>+</ul>+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def collator(self, samples: list = None):+ """Collator.++ Args:+ samples: TODO.+ """+ outputs = {}+ for sample in samples:+ for key in sample.keys():+ if key not in outputs:+ outputs[key] = []+ outputs[key].append(sample[key])++ for key, data_list in outputs.items():+ if isinstance(data_list[0], torch.Tensor):+ if data_list[0].dtype == torch.int64 or data_list[0].dtype == torch.int32:++ pad_value = self.int_pad_value+ else:+ pad_value = self.float_pad_value++ outputs[key] = torch.nn.utils.rnn.pad_sequence(+ data_list, batch_first=True, padding_value=pad_value+ )+ return outputs+</code></pre></details></div></div>+<div class="api-detail" id="e523"><span class="dtag badge-m">method</span><h2><code>AudioDataset.get_source_len(index)</code></h2><div class="dmod">funasr.datasets.datasets.AudioDataset · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/datasets/audio_datasets/datasets.py#L67" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Get source len.</p>+<h4>Args:</h4>+<ul>+<li><code>index</code> — TODO.</li>+</ul>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def get_source_len(self, index):+ """Get source len.++ Args:+ index: TODO.+ """+ item = self.index_ds[index]+ return self.index_ds.get_source_len(item)+</code></pre></details></div>+<div class="api-detail" id="e524"><span class="dtag badge-m">method</span><h2><code>AudioDataset.get_target_len(index)</code></h2><div class="dmod">funasr.datasets.datasets.AudioDataset · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/datasets/audio_datasets/datasets.py#L76" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Get target len.</p>+<h4>Args:</h4>+<ul>+<li><code>index</code> — TODO.</li>+</ul>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def get_target_len(self, index):+ """Get target len.++ Args:+ index: TODO.+ """+ item = self.index_ds[index]+ return self.index_ds.get_target_len(item)+</code></pre></details></div>+<div class="api-detail" id="e525"><span class="dtag badge-m">method</span><h2><code>AudioDataset.collator(samples)</code></h2><div class="dmod">funasr.datasets.datasets.AudioDataset · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/datasets/audio_datasets/datasets.py#L127" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Collator.</p>+<h4>Args:</h4>+<ul>+<li><code>samples</code> — TODO.</li>+</ul>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def collator(self, samples: list = None):+ """Collator.++ Args:+ samples: TODO.+ """+ outputs = {}+ for sample in samples:+ for key in sample.keys():+ if key not in outputs:+ outputs[key] = []+ outputs[key].append(sample[key])++ for key, data_list in outputs.items():+ if isinstance(data_list[0], torch.Tensor):+ if data_list[0].dtype == torch.int64 or data_list[0].dtype == torch.int32:++ pad_value = self.int_pad_value+ else:+ pad_value = self.float_pad_value++ outputs[key] = torch.nn.utils.rnn.pad_sequence(+ data_list, batch_first=True, padding_value=pad_value+ )+ return outputs+</code></pre></details></div>+<div class="api-detail" id="e526"><span class="dtag badge-c">class</span><h2>AudioDatasetHotword</h2><div class="dmod">funasr.datasets.datasets · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/datasets/audio_datasets/datasets.py#L155" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p class="muted">No documentation yet.</p></div><details class="src-block"><summary>📄 Source code</summary><pre><code>class AudioDatasetHotword(AudioDataset):+ # for finetuning contextual_paraformer and seaco_paraformer+ def __init__(+ self,+ *args,+ seaco_id: bool = 0,+ **kwargs,+ ):+ """Initialize AudioDatasetHotword.++ Args:+ *args: Variable positional arguments.+ **kwargs: Additional keyword arguments.+ """+ super().__init__(*args, **kwargs)+ self.seaco_id = seaco_id++ def __getitem__(self, index):+ """Internal: getitem .++ Args:+ index: TODO.+ """+ item = self.index_ds[index]+ # import pdb;+ # pdb.set_trace()+ source = item["source"]+ data_src = load_audio_text_image_video(source, fs=self.fs)+ if self.preprocessor_speech:+ data_src = self.preprocessor_speech(data_src, fs=self.fs)+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/datasets/audio_datasets/datasets.py#L155" target="_blank" class="src-more">View full source on GitHub →</a></details><h3 class="mt">Methods</h3><div class="mblk"><code>.collator(samples)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/datasets/audio_datasets/datasets.py#L268" target="_blank" class="gh-link">L268</a><div class="mdoc"><p>Collator.</p>+<h4>Args:</h4>+<ul>+<li><code>samples</code> — TODO.</li>+</ul>+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def collator(self, samples: list = None):+ """Collator.++ Args:+ samples: TODO.+ """+ outputs = {}+ hotword_indxs = []+ seaco_id = samples[0]["seaco_id"]+ for sample in samples:+ for key in sample.keys():+ if key == "seaco_id":+ continue+ elif key == "hotword_indx":+ hotword_indxs.append(sample[key])+ else:+ if key not in outputs:+ outputs[key] = []+ outputs[key].append(sample[key])++ for key, data_list in outputs.items():+ if isinstance(data_list[0], torch.Tensor):+ if data_list[0].dtype == torch.int64 or data_list[0].dtype == torch.int32:+ pad_value = self.int_pad_value+ else:+ pad_value = self.float_pad_value+ outputs[key] = torch.nn.utils.rnn.pad_sequence(+ data_list, batch_first=True, padding_value=pad_value+ )++</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/datasets/audio_datasets/datasets.py#L268" target="_blank" class="src-more">View full source →</a></details></div></div>+<div class="api-detail" id="e527"><span class="dtag badge-m">method</span><h2><code>AudioDatasetHotword.collator(samples)</code></h2><div class="dmod">funasr.datasets.datasets.AudioDatasetHotword · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/datasets/audio_datasets/datasets.py#L268" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Collator.</p>+<h4>Args:</h4>+<ul>+<li><code>samples</code> — TODO.</li>+</ul>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def collator(self, samples: list = None):+ """Collator.++ Args:+ samples: TODO.+ """+ outputs = {}+ hotword_indxs = []+ seaco_id = samples[0]["seaco_id"]+ for sample in samples:+ for key in sample.keys():+ if key == "seaco_id":+ continue+ elif key == "hotword_indx":+ hotword_indxs.append(sample[key])+ else:+ if key not in outputs:+ outputs[key] = []+ outputs[key].append(sample[key])++ for key, data_list in outputs.items():+ if isinstance(data_list[0], torch.Tensor):+ if data_list[0].dtype == torch.int64 or data_list[0].dtype == torch.int32:+ pad_value = self.int_pad_value+ else:+ pad_value = self.float_pad_value+ outputs[key] = torch.nn.utils.rnn.pad_sequence(+ data_list, batch_first=True, padding_value=pad_value+ )++</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/datasets/audio_datasets/datasets.py#L268" target="_blank" class="src-more">View full source on GitHub →</a></details></div>+<div class="api-detail" id="e528"><span class="dtag badge-f">function</span><h2><code>EspnetStyleBatchSampler_fn(dataset, **kwargs)</code></h2><div class="dmod">funasr.datasets.espnet_samplers · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/datasets/audio_datasets/espnet_samplers.py#L14" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Espnetstylebatchsampler fn.</p>+<h4>Args:</h4>+<ul>+<li><code>dataset</code> — TODO.</li>+<li><code>**kwargs</code> — Additional keyword arguments.</li>+</ul>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code>def EspnetStyleBatchSampler_fn(dataset, **kwargs):+ """Espnetstylebatchsampler fn.++ Args:+ dataset: TODO.+ **kwargs: Additional keyword arguments.+ """+ dataloader_args = {}++ batch_sampler = EspnetStyleBatchSampler(dataset, **kwargs)+ dataloader_args["batch_sampler"] = batch_sampler+ dataloader_args["num_workers"] = kwargs.get("num_workers", 4)+ dataloader_args["pin_memory"] = kwargs.get("pin_memory", True)++ return dataloader_args+</code></pre></details></div>+<div class="api-detail" id="e529"><span class="dtag badge-c">class</span><h2>EspnetStyleBatchSampler</h2><div class="dmod">funasr.datasets.espnet_samplers · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/datasets/audio_datasets/espnet_samplers.py#L37" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p class="muted">No documentation yet.</p></div><details class="src-block"><summary>📄 Source code</summary><pre><code>class EspnetStyleBatchSampler(DistributedSampler):+ def __init__(+ self,+ dataset,+ batch_size,+ batch_type="token",+ rank=None,+ num_replicas=None,+ rank_split=False,+ shuffle=True,+ drop_last=False,+ is_training: bool = True,+ sort_size: int = 1024,+ start_step: int = 0,+ **kwargs,+ ):++ """Initialize EspnetStyleBatchSampler.++ Args:+ dataset: TODO.+ batch_size: Number of samples per batch.+ batch_type: TODO.+ rank: TODO.+ num_replicas: TODO.+ rank_split: TODO.+ shuffle: TODO.+ drop_last: TODO.+ is_training: Boolean flag for training.+ sort_size: Size/dimension parameter.+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/datasets/audio_datasets/espnet_samplers.py#L37" target="_blank" class="src-more">View full source on GitHub →</a></details><h3 class="mt">Methods</h3><div class="mblk"><code>.set_epoch(epoch)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/datasets/audio_datasets/espnet_samplers.py#L187" target="_blank" class="gh-link">L187</a><div class="mdoc"><p>Set epoch.</p>+<h4>Args:</h4>+<ul>+<li><code>epoch</code> — TODO.</li>+</ul>+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def set_epoch(self, epoch):+ # Set the epoch for shuffling+ """Set epoch.++ Args:+ epoch: TODO.+ """+ self.epoch = epoch+</code></pre></details></div></div>+<div class="api-detail" id="e530"><span class="dtag badge-m">method</span><h2><code>EspnetStyleBatchSampler.set_epoch(epoch)</code></h2><div class="dmod">funasr.datasets.espnet_samplers.EspnetStyleBatchSampler · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/datasets/audio_datasets/espnet_samplers.py#L187" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Set epoch.</p>+<h4>Args:</h4>+<ul>+<li><code>epoch</code> — TODO.</li>+</ul>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def set_epoch(self, epoch):+ # Set the epoch for shuffling+ """Set epoch.++ Args:+ epoch: TODO.+ """+ self.epoch = epoch+</code></pre></details></div>+<div class="api-detail" id="e531"><span class="dtag badge-c">class</span><h2>IndexDSJsonlRankFull</h2><div class="dmod">funasr.datasets.index_ds · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/datasets/audio_datasets/index_ds.py#L16" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p class="muted">No documentation yet.</p></div><details class="src-block"><summary>📄 Source code</summary><pre><code>class IndexDSJsonlRankFull(torch.utils.data.Dataset):++ def __init__(self, path: str, **kwargs):+ """Initialize IndexDSJsonlRankFull.++ Args:+ path: TODO.+ **kwargs: Additional keyword arguments.+ """+ super().__init__()+ self.max_source_length = kwargs.get("max_source_length", 2048)+ self.min_source_length = kwargs.get("min_source_length", 0)+ self.max_target_length = kwargs.get("max_target_length", 2048)+ self.min_target_length = kwargs.get("min_target_length", 0)+ self.max_token_length = kwargs.get("max_token_length", 2200)++ is_training = kwargs.get("is_training", True)+ if not (path.endswith(".jsonl") or path.endswith(".json")):+ # jsonl list file+ data_split_num = kwargs.get("data_split_num", 1)+ data_split_i = kwargs.get("data_split_i", 0)++ if not is_training:+ data_split_num = 1+ data_split_i = 0+ with open(path, encoding="utf-8") as fin:+ file_list_all = fin.readlines()++ num_per_slice = (len(file_list_all) - 1) // data_split_num + 1 # 16+ file_list = file_list_all[+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/datasets/audio_datasets/index_ds.py#L16" target="_blank" class="src-more">View full source on GitHub →</a></details><h3 class="mt">Methods</h3><div class="mblk"><code>.get_source_len(data_dict)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/datasets/audio_datasets/index_ds.py#L158" target="_blank" class="gh-link">L158</a><div class="mdoc"><p>Get source len.</p>+<h4>Args:</h4>+<ul>+<li><code>data_dict</code> — TODO.</li>+</ul>+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def get_source_len(self, data_dict):+ """Get source len.++ Args:+ data_dict: TODO.+ """+ return data_dict.get("source_len", 1)+</code></pre></details></div><div class="mblk"><code>.get_target_len(data_dict)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/datasets/audio_datasets/index_ds.py#L166" target="_blank" class="gh-link">L166</a><div class="mdoc"><p>Get target len.</p>+<h4>Args:</h4>+<ul>+<li><code>data_dict</code> — TODO.</li>+</ul>+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def get_target_len(self, data_dict):++ """Get target len.++ Args:+ data_dict: TODO.+ """+ return data_dict.get("target_len", 0)+</code></pre></details></div></div>+<div class="api-detail" id="e532"><span class="dtag badge-m">method</span><h2><code>IndexDSJsonlRankFull.get_source_len(data_dict)</code></h2><div class="dmod">funasr.datasets.index_ds.IndexDSJsonlRankFull · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/datasets/audio_datasets/index_ds.py#L158" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Get source len.</p>+<h4>Args:</h4>+<ul>+<li><code>data_dict</code> — TODO.</li>+</ul>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def get_source_len(self, data_dict):+ """Get source len.++ Args:+ data_dict: TODO.+ """+ return data_dict.get("source_len", 1)+</code></pre></details></div>+<div class="api-detail" id="e533"><span class="dtag badge-m">method</span><h2><code>IndexDSJsonlRankFull.get_target_len(data_dict)</code></h2><div class="dmod">funasr.datasets.index_ds.IndexDSJsonlRankFull · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/datasets/audio_datasets/index_ds.py#L166" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Get target len.</p>+<h4>Args:</h4>+<ul>+<li><code>data_dict</code> — TODO.</li>+</ul>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def get_target_len(self, data_dict):++ """Get target len.++ Args:+ data_dict: TODO.+ """+ return data_dict.get("target_len", 0)+</code></pre></details></div>+<div class="api-detail" id="e534"><span class="dtag badge-f">function</span><h2><code>gen_scp_from_jsonl(jsonl_file, data_type_list, wav_scp_file, text_file)</code></h2><div class="dmod">funasr.datasets.jsonl2scp · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/datasets/audio_datasets/jsonl2scp.py#L12" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Gen scp from jsonl.</p>+<h4>Args:</h4>+<ul>+<li><code>jsonl_file</code> — TODO.</li>+<li><code>data_type_list</code> — TODO.</li>+<li><code>wav_scp_file</code> — TODO.</li>+<li><code>text_file</code> — TODO.</li>+</ul>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code>def gen_scp_from_jsonl(jsonl_file, data_type_list, wav_scp_file, text_file):++ """Gen scp from jsonl.++ Args:+ jsonl_file: TODO.+ data_type_list: TODO.+ wav_scp_file: TODO.+ text_file: TODO.+ """+ wav_f = open(wav_scp_file, "w")+ text_f = open(text_file, "w")+ with open(jsonl_file, encoding="utf-8") as fin:+ for line in fin:+ data = json.loads(line.strip())++ prompt = data.get("prompt", "<ASR>")+ source = data[data_type_list[0]]+ target = data[data_type_list[1]]+ source_len = data.get("source_len", 1)+ target_len = data.get("target_len", 0)+ if "aishell" in source:+ target = target.replace(" ", "")+ key = data["key"]+ wav_f.write(f"{key}\t{source}\n")+ wav_f.flush()+ text_f.write(f"{key}\t{target}\n")+ text_f.flush()++ wav_f.close()+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/datasets/audio_datasets/jsonl2scp.py#L12" target="_blank" class="src-more">View full source on GitHub →</a></details></div>+<div class="api-detail" id="e535"><span class="dtag badge-f">function</span><h2><code>main_hydra(cfg)</code></h2><div class="dmod">funasr.datasets.jsonl2scp · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/datasets/audio_datasets/jsonl2scp.py#L46" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Main hydra.</p>+<h4>Args:</h4>+<ul>+<li><code>cfg</code> — Configuration overrides.</li>+</ul>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code>def main_hydra(cfg: DictConfig):++ """Main hydra.++ Args:+ cfg: Configuration overrides.+ """+ kwargs = OmegaConf.to_container(cfg, resolve=True)+ print(kwargs)++ scp_file_list = kwargs.get(+ "scp_file_list",+ ("/Users/zhifu/funasr1.0/test_local/wav.scp", "/Users/zhifu/funasr1.0/test_local/text.txt"),+ )+ if isinstance(scp_file_list, str):+ scp_file_list = eval(scp_file_list)+ data_type_list = kwargs.get("data_type_list", ("source", "target"))+ jsonl_file = kwargs.get(+ "jsonl_file_in", "/Users/zhifu/funasr1.0/test_local/audio_datasets.jsonl"+ )+ gen_scp_from_jsonl(jsonl_file, data_type_list, *scp_file_list)+</code></pre></details></div>+<div class="api-detail" id="e536"><span class="dtag badge-c">class</span><h2>SpeechPreprocessSpeedPerturb</h2><div class="dmod">funasr.datasets.preprocessor · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/datasets/audio_datasets/preprocessor.py#L19" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p class="muted">No documentation yet.</p></div><details class="src-block"><summary>📄 Source code</summary><pre><code>class SpeechPreprocessSpeedPerturb(nn.Module):+ def __init__(self, speed_perturb: list = None, **kwargs):+ """Initialize SpeechPreprocessSpeedPerturb.++ Args:+ speed_perturb: TODO.+ **kwargs: Additional keyword arguments.+ """+ super().__init__()+ self.speed_perturb = speed_perturb++ def forward(self, waveform, fs, **kwargs):+ """Forward pass for training.++ Args:+ waveform: TODO.+ fs: TODO.+ **kwargs: Additional keyword arguments.+ """+ if self.speed_perturb is None:+ return waveform+ speed = random.choice(self.speed_perturb)+ if speed != 1.0:+ if not isinstance(waveform, torch.Tensor):+ waveform = torch.tensor(waveform)+ waveform, _ = torchaudio.sox_effects.apply_effects_tensor(+ waveform.view(1, -1), fs, [["speed", str(speed)], ["rate", str(fs)]]+ )+ waveform = waveform.view(-1)++</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/datasets/audio_datasets/preprocessor.py#L19" target="_blank" class="src-more">View full source on GitHub →</a></details><h3 class="mt">Methods</h3><div class="mblk"><code>.forward(waveform, fs, **kwargs)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/datasets/audio_datasets/preprocessor.py#L30" target="_blank" class="gh-link">L30</a><div class="mdoc"><p>Forward pass for training.</p>+<h4>Args:</h4>+<ul>+<li><code>waveform</code> — TODO.</li>+<li><code>fs</code> — TODO.</li>+<li><code>**kwargs</code> — Additional keyword arguments.</li>+</ul>+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def forward(self, waveform, fs, **kwargs):+ """Forward pass for training.++ Args:+ waveform: TODO.+ fs: TODO.+ **kwargs: Additional keyword arguments.+ """+ if self.speed_perturb is None:+ return waveform+ speed = random.choice(self.speed_perturb)+ if speed != 1.0:+ if not isinstance(waveform, torch.Tensor):+ waveform = torch.tensor(waveform)+ waveform, _ = torchaudio.sox_effects.apply_effects_tensor(+ waveform.view(1, -1), fs, [["speed", str(speed)], ["rate", str(fs)]]+ )+ waveform = waveform.view(-1)++ return waveform+</code></pre></details></div></div>+<div class="api-detail" id="e537"><span class="dtag badge-m">method</span><h2><code>SpeechPreprocessSpeedPerturb.forward(waveform, fs, **kwargs)</code></h2><div class="dmod">funasr.datasets.preprocessor.SpeechPreprocessSpeedPerturb · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/datasets/audio_datasets/preprocessor.py#L30" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Forward pass for training.</p>+<h4>Args:</h4>+<ul>+<li><code>waveform</code> — TODO.</li>+<li><code>fs</code> — TODO.</li>+<li><code>**kwargs</code> — Additional keyword arguments.</li>+</ul>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def forward(self, waveform, fs, **kwargs):+ """Forward pass for training.++ Args:+ waveform: TODO.+ fs: TODO.+ **kwargs: Additional keyword arguments.+ """+ if self.speed_perturb is None:+ return waveform+ speed = random.choice(self.speed_perturb)+ if speed != 1.0:+ if not isinstance(waveform, torch.Tensor):+ waveform = torch.tensor(waveform)+ waveform, _ = torchaudio.sox_effects.apply_effects_tensor(+ waveform.view(1, -1), fs, [["speed", str(speed)], ["rate", str(fs)]]+ )+ waveform = waveform.view(-1)++ return waveform+</code></pre></details></div>+<div class="api-detail" id="e538"><span class="dtag badge-c">class</span><h2>TextPreprocessSegDict</h2><div class="dmod">funasr.datasets.preprocessor · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/datasets/audio_datasets/preprocessor.py#L53" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p class="muted">No documentation yet.</p></div><details class="src-block"><summary>📄 Source code</summary><pre><code>class TextPreprocessSegDict(nn.Module):+ def __init__(+ self,+ seg_dict: str = None,+ text_cleaner: Collection[str] = None,+ split_with_space: bool = False,+ **kwargs+ ):+ """Initialize TextPreprocessSegDict.++ Args:+ seg_dict: TODO.+ text_cleaner: TODO.+ split_with_space: TODO.+ **kwargs: Additional keyword arguments.+ """+ super().__init__()++ self.text_cleaner = TextCleaner(text_cleaner)++ def forward(self, text, **kwargs):+ """Forward pass for training.++ Args:+ text: Text tensor or string input.+ **kwargs: Additional keyword arguments.+ """+ text = self.text_cleaner(text)++ return text+</code></pre></details><h3 class="mt">Methods</h3><div class="mblk"><code>.forward(text, **kwargs)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/datasets/audio_datasets/preprocessor.py#L73" target="_blank" class="gh-link">L73</a><div class="mdoc"><p>Forward pass for training.</p>+<h4>Args:</h4>+<ul>+<li><code>text</code> — Text tensor or string input.</li>+<li><code>**kwargs</code> — Additional keyword arguments.</li>+</ul>+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def forward(self, text, **kwargs):+ """Forward pass for training.++ Args:+ text: Text tensor or string input.+ **kwargs: Additional keyword arguments.+ """+ text = self.text_cleaner(text)++ return text+</code></pre></details></div></div>+<div class="api-detail" id="e539"><span class="dtag badge-m">method</span><h2><code>TextPreprocessSegDict.forward(text, **kwargs)</code></h2><div class="dmod">funasr.datasets.preprocessor.TextPreprocessSegDict · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/datasets/audio_datasets/preprocessor.py#L73" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Forward pass for training.</p>+<h4>Args:</h4>+<ul>+<li><code>text</code> — Text tensor or string input.</li>+<li><code>**kwargs</code> — Additional keyword arguments.</li>+</ul>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def forward(self, text, **kwargs):+ """Forward pass for training.++ Args:+ text: Text tensor or string input.+ **kwargs: Additional keyword arguments.+ """+ text = self.text_cleaner(text)++ return text+</code></pre></details></div>+<div class="api-detail" id="e540"><span class="dtag badge-f">function</span><h2><code>CustomDistributedBatchSampler_fn(dataset, **kwargs)</code></h2><div class="dmod">funasr.datasets.samplers · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/datasets/audio_datasets/samplers.py#L20" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Customdistributedbatchsampler fn.</p>+<h4>Args:</h4>+<ul>+<li><code>dataset</code> — TODO.</li>+<li><code>**kwargs</code> — Additional keyword arguments.</li>+</ul>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code>def CustomDistributedBatchSampler_fn(dataset, **kwargs):+ """Customdistributedbatchsampler fn.++ Args:+ dataset: TODO.+ **kwargs: Additional keyword arguments.+ """+ dataloader_args = {}+ batch_type = kwargs.get("batch_type", "example")+ if batch_type == "example":+ batch_sampler = CustomDistributedBatchSampler(dataset, **kwargs)++ else:+ if kwargs.get("sort_size", -1) > 0:+ batch_sampler = CustomDistributedBufferDynamicBatchSampler(dataset, **kwargs)+ else:+ batch_sampler = CustomDistributedDynamicBatchSampler(dataset, **kwargs)+ # batch_sampler = CustomDistributedDynamicBatchSampler(dataset, **kwargs)++ dataloader_args["batch_sampler"] = batch_sampler+ dataloader_args["num_workers"] = kwargs.get("num_workers", 4)+ dataloader_args["pin_memory"] = kwargs.get("pin_memory", True)++ return dataloader_args+</code></pre></details></div>+<div class="api-detail" id="e541"><span class="dtag badge-c">class</span><h2>CustomDistributedBatchSampler</h2><div class="dmod">funasr.datasets.samplers · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/datasets/audio_datasets/samplers.py#L46" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p class="muted">No documentation yet.</p></div><details class="src-block"><summary>📄 Source code</summary><pre><code>class CustomDistributedBatchSampler(Sampler):+ def __init__(+ self,+ dataset,+ batch_size,+ num_replicas=None,+ rank=None,+ shuffle=True,+ drop_last=False,+ is_training: bool = True,+ **kwargs,+ ):++ """Initialize CustomDistributedBatchSampler.++ Args:+ dataset: TODO.+ batch_size: Number of samples per batch.+ num_replicas: TODO.+ rank: TODO.+ shuffle: TODO.+ drop_last: TODO.+ is_training: Boolean flag for training.+ **kwargs: Additional keyword arguments.+ """+ try:+ rank = dist.get_rank()+ num_replicas = dist.get_world_size()+ except:+ rank = 0+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/datasets/audio_datasets/samplers.py#L46" target="_blank" class="src-more">View full source on GitHub →</a></details><h3 class="mt">Methods</h3><div class="mblk"><code>.set_epoch(epoch)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/datasets/audio_datasets/samplers.py#L148" target="_blank" class="gh-link">L148</a><div class="mdoc"><p>Set epoch.</p>+<h4>Args:</h4>+<ul>+<li><code>epoch</code> — TODO.</li>+</ul>+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def set_epoch(self, epoch):+ """Set epoch.++ Args:+ epoch: TODO.+ """+ self.epoch = epoch+</code></pre></details></div></div>+<div class="api-detail" id="e542"><span class="dtag badge-m">method</span><h2><code>CustomDistributedBatchSampler.set_epoch(epoch)</code></h2><div class="dmod">funasr.datasets.samplers.CustomDistributedBatchSampler · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/datasets/audio_datasets/samplers.py#L148" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Set epoch.</p>+<h4>Args:</h4>+<ul>+<li><code>epoch</code> — TODO.</li>+</ul>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def set_epoch(self, epoch):+ """Set epoch.++ Args:+ epoch: TODO.+ """+ self.epoch = epoch+</code></pre></details></div>+<div class="api-detail" id="e543"><span class="dtag badge-c">class</span><h2>CustomDistributedBufferBatchSampler</h2><div class="dmod">funasr.datasets.samplers · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/datasets/audio_datasets/samplers.py#L157" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p class="muted">No documentation yet.</p></div><details class="src-block"><summary>📄 Source code</summary><pre><code>class CustomDistributedBufferBatchSampler(Sampler):+ def __init__(+ self,+ dataset,+ batch_size,+ num_replicas=None,+ rank=None,+ shuffle=True,+ drop_last=False,+ is_training: bool = True,+ sort_size: int = 1024,+ **kwargs,+ ):++ """Initialize CustomDistributedBufferBatchSampler.++ Args:+ dataset: TODO.+ batch_size: Number of samples per batch.+ num_replicas: TODO.+ rank: TODO.+ shuffle: TODO.+ drop_last: TODO.+ is_training: Boolean flag for training.+ sort_size: Size/dimension parameter.+ **kwargs: Additional keyword arguments.+ """+ try:+ rank = dist.get_rank()+ num_replicas = dist.get_world_size()+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/datasets/audio_datasets/samplers.py#L157" target="_blank" class="src-more">View full source on GitHub →</a></details><h3 class="mt">Methods</h3><div class="mblk"><code>.set_epoch(epoch)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/datasets/audio_datasets/samplers.py#L284" target="_blank" class="gh-link">L284</a><div class="mdoc"><p>Set epoch.</p>+<h4>Args:</h4>+<ul>+<li><code>epoch</code> — TODO.</li>+</ul>+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def set_epoch(self, epoch):+ """Set epoch.++ Args:+ epoch: TODO.+ """+ self.epoch = epoch+</code></pre></details></div></div>+<div class="api-detail" id="e544"><span class="dtag badge-m">method</span><h2><code>CustomDistributedBufferBatchSampler.set_epoch(epoch)</code></h2><div class="dmod">funasr.datasets.samplers.CustomDistributedBufferBatchSampler · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/datasets/audio_datasets/samplers.py#L284" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Set epoch.</p>+<h4>Args:</h4>+<ul>+<li><code>epoch</code> — TODO.</li>+</ul>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def set_epoch(self, epoch):+ """Set epoch.++ Args:+ epoch: TODO.+ """+ self.epoch = epoch+</code></pre></details></div>+<div class="api-detail" id="e545"><span class="dtag badge-c">class</span><h2>CustomDistributedDynamicBatchSampler</h2><div class="dmod">funasr.datasets.samplers · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/datasets/audio_datasets/samplers.py#L293" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p class="muted">No documentation yet.</p></div><details class="src-block"><summary>📄 Source code</summary><pre><code>class CustomDistributedDynamicBatchSampler(DistributedSampler):+ def __init__(+ self,+ dataset,+ batch_size,+ num_replicas=None,+ rank=None,+ shuffle=True,+ drop_last=False,+ is_training: bool = True,+ **kwargs,+ ):++ """Initialize CustomDistributedDynamicBatchSampler.++ Args:+ dataset: TODO.+ batch_size: Number of samples per batch.+ num_replicas: TODO.+ rank: TODO.+ shuffle: TODO.+ drop_last: TODO.+ is_training: Boolean flag for training.+ **kwargs: Additional keyword arguments.+ """+ try:+ rank = dist.get_rank()+ num_replicas = dist.get_world_size()+ except:+ rank = 0+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/datasets/audio_datasets/samplers.py#L293" target="_blank" class="src-more">View full source on GitHub →</a></details><h3 class="mt">Methods</h3><div class="mblk"><code>.set_epoch(epoch)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/datasets/audio_datasets/samplers.py#L384" target="_blank" class="gh-link">L384</a><div class="mdoc"><p>Set epoch.</p>+<h4>Args:</h4>+<ul>+<li><code>epoch</code> — TODO.</li>+</ul>+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def set_epoch(self, epoch):+ """Set epoch.++ Args:+ epoch: TODO.+ """+ self.epoch = epoch+</code></pre></details></div></div>+<div class="api-detail" id="e546"><span class="dtag badge-m">method</span><h2><code>CustomDistributedDynamicBatchSampler.set_epoch(epoch)</code></h2><div class="dmod">funasr.datasets.samplers.CustomDistributedDynamicBatchSampler · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/datasets/audio_datasets/samplers.py#L384" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Set epoch.</p>+<h4>Args:</h4>+<ul>+<li><code>epoch</code> — TODO.</li>+</ul>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def set_epoch(self, epoch):+ """Set epoch.++ Args:+ epoch: TODO.+ """+ self.epoch = epoch+</code></pre></details></div>+<div class="api-detail" id="e547"><span class="dtag badge-c">class</span><h2>CustomDistributedBufferDynamicBatchSampler</h2><div class="dmod">funasr.datasets.samplers · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/datasets/audio_datasets/samplers.py#L393" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p class="muted">No documentation yet.</p></div><details class="src-block"><summary>📄 Source code</summary><pre><code>class CustomDistributedBufferDynamicBatchSampler(DistributedSampler):+ def __init__(+ self,+ dataset,+ batch_size,+ batch_type="token",+ num_replicas=None,+ rank=None,+ rank_split=False,+ shuffle=True,+ drop_last=False,+ is_training: bool = True,+ sort_size: int = 1024,+ start_step: int = 0,+ **kwargs,+ ):++ """Initialize CustomDistributedBufferDynamicBatchSampler.++ Args:+ dataset: TODO.+ batch_size: Number of samples per batch.+ batch_type: TODO.+ num_replicas: TODO.+ rank: TODO.+ rank_split: TODO.+ shuffle: TODO.+ drop_last: TODO.+ is_training: Boolean flag for training.+ sort_size: Size/dimension parameter.+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/datasets/audio_datasets/samplers.py#L393" target="_blank" class="src-more">View full source on GitHub →</a></details><h3 class="mt">Methods</h3><div class="mblk"><code>.set_epoch(epoch)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/datasets/audio_datasets/samplers.py#L526" target="_blank" class="gh-link">L526</a><div class="mdoc"><p>Set epoch.</p>+<h4>Args:</h4>+<ul>+<li><code>epoch</code> — TODO.</li>+</ul>+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def set_epoch(self, epoch):+ """Set epoch.++ Args:+ epoch: TODO.+ """+ self.epoch = epoch+</code></pre></details></div></div>+<div class="api-detail" id="e548"><span class="dtag badge-m">method</span><h2><code>CustomDistributedBufferDynamicBatchSampler.set_epoch(epoch)</code></h2><div class="dmod">funasr.datasets.samplers.CustomDistributedBufferDynamicBatchSampler · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/datasets/audio_datasets/samplers.py#L526" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Set epoch.</p>+<h4>Args:</h4>+<ul>+<li><code>epoch</code> — TODO.</li>+</ul>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def set_epoch(self, epoch):+ """Set epoch.++ Args:+ epoch: TODO.+ """+ self.epoch = epoch+</code></pre></details></div>+<div class="api-detail" id="e549"><span class="dtag badge-c">class</span><h2>DistributedSamplerWarp</h2><div class="dmod">funasr.datasets.samplers · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/datasets/audio_datasets/samplers.py#L535" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p class="muted">No documentation yet.</p></div><details class="src-block"><summary>📄 Source code</summary><pre><code>class DistributedSamplerWarp(BatchSampler):+ def __init__(+ self, dataset, batch_size, num_replicas=None, rank=None, shuffle=True, drop_last=False+ ):+ """Initialize DistributedSamplerWarp.++ Args:+ dataset: TODO.+ batch_size: Number of samples per batch.+ num_replicas: TODO.+ rank: TODO.+ shuffle: TODO.+ drop_last: TODO.+ """+ if num_replicas is None:+ if not torch.distributed.is_available():+ raise RuntimeError("Requires distributed package to be available")+ num_replicas = torch.distributed.get_world_size()+ if rank is None:+ if not torch.distributed.is_available():+ raise RuntimeError("Requires distributed package to be available")+ rank = torch.distributed.get_rank()++ self.dataset = dataset+ self.batch_size = batch_size+ self.num_replicas = num_replicas+ self.rank = rank+ self.shuffle = shuffle+ self.drop_last = drop_last++</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/datasets/audio_datasets/samplers.py#L535" target="_blank" class="src-more">View full source on GitHub →</a></details><h3 class="mt">Methods</h3><div class="mblk"><code>.set_epoch(epoch)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/datasets/audio_datasets/samplers.py#L582" target="_blank" class="gh-link">L582</a><div class="mdoc"><p>Set epoch.</p>+<h4>Args:</h4>+<ul>+<li><code>epoch</code> — TODO.</li>+</ul>+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def set_epoch(self, epoch):+ """Set epoch.++ Args:+ epoch: TODO.+ """+ self.epoch = epoch+</code></pre></details></div></div>+<div class="api-detail" id="e550"><span class="dtag badge-m">method</span><h2><code>DistributedSamplerWarp.set_epoch(epoch)</code></h2><div class="dmod">funasr.datasets.samplers.DistributedSamplerWarp · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/datasets/audio_datasets/samplers.py#L582" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Set epoch.</p>+<h4>Args:</h4>+<ul>+<li><code>epoch</code> — TODO.</li>+</ul>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def set_epoch(self, epoch):+ """Set epoch.++ Args:+ epoch: TODO.+ """+ self.epoch = epoch+</code></pre></details></div>+<div class="api-detail" id="e551"><span class="dtag badge-f">function</span><h2><code>gen_jsonl_from_wav_text_list(path, data_type_list, jsonl_file_out, **kwargs)</code></h2><div class="dmod">funasr.datasets.scp2jsonl · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/datasets/audio_datasets/scp2jsonl.py#L13" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Gen jsonl from wav text list.</p>+<h4>Args:</h4>+<ul>+<li><code>path</code> — TODO.</li>+<li><code>data_type_list</code> — TODO.</li>+<li><code>jsonl_file_out</code> — TODO.</li>+<li><code>**kwargs</code> — Additional keyword arguments.</li>+</ul>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code>def gen_jsonl_from_wav_text_list(+ path, data_type_list=("source", "target"), jsonl_file_out: str = None, **kwargs+):+ """Gen jsonl from wav text list.++ Args:+ path: TODO.+ data_type_list: TODO.+ jsonl_file_out: TODO.+ **kwargs: Additional keyword arguments.+ """+ try:+ rank = dist.get_rank()+ world_size = dist.get_world_size()+ except:+ rank = 0+ world_size = 1++ cpu_cores = os.cpu_count() or 1+ print(f"convert wav.scp text to jsonl, ncpu: {cpu_cores}")+ if rank == 0:+ json_dict = {}+ for data_type, data_file in zip(data_type_list, path):+ json_dict[data_type] = {}+ with open(data_file, "r") as f:++ data_file_lists = f.readlines()+ lines_for_each_th = (len(data_file_lists) - 1) // cpu_cores + 1+ task_num = cpu_cores if len(data_file_lists) > cpu_cores else 1+ # import pdb;pdb.set_trace()+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/datasets/audio_datasets/scp2jsonl.py#L13" target="_blank" class="src-more">View full source on GitHub →</a></details></div>+<div class="api-detail" id="e552"><span class="dtag badge-f">function</span><h2><code>parse_context_length(data_list, data_type, id)</code></h2><div class="dmod">funasr.datasets.scp2jsonl · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/datasets/audio_datasets/scp2jsonl.py#L83" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Parse context length.</p>+<h4>Args:</h4>+<ul>+<li><code>data_list</code> — TODO.</li>+<li><code>data_type</code> — TODO.</li>+<li><code>id</code> — TODO.</li>+</ul>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code>def parse_context_length(data_list: list, data_type: str, id=0):+ """Parse context length.++ Args:+ data_list: TODO.+ data_type: TODO.+ id: TODO.+ """+ pbar = tqdm(total=len(data_list), dynamic_ncols=True)+ res = {}+ for i, line in enumerate(data_list):+ pbar.update(1)+ pbar.set_description(f"cpu: {id}")+ lines = line.strip().split(maxsplit=1)+ key = lines[0]+ line = lines[1] if len(lines) > 1 else ""+ line = line.strip()+ if data_type == "source":+ if os.path.exists(line):+ waveform, _ = librosa.load(line, sr=16000)+ sample_num = len(waveform)+ context_len = int(sample_num * 1000 / 16000 / 10)+ else:+ print("source file not found: {}".format(line))+ continue+ else:+ context_len = len(line.split()) if " " in line else len(line)+ res[key] = {data_type: line, f"{data_type}_len": context_len}+ return res+</code></pre></details></div>+<div class="api-detail" id="e553"><span class="dtag badge-f">function</span><h2><code>main_hydra(cfg)</code></h2><div class="dmod">funasr.datasets.scp2jsonl · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/datasets/audio_datasets/scp2jsonl.py#L115" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Main hydra.</p>+<h4>Args:</h4>+<ul>+<li><code>cfg</code> — Configuration overrides.</li>+</ul>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code>def main_hydra(cfg: DictConfig):++ """Main hydra.++ Args:+ cfg: Configuration overrides.+ """+ kwargs = OmegaConf.to_container(cfg, resolve=True)+ print(kwargs)++ scp_file_list = kwargs.get(+ "scp_file_list",+ ("/Users/zhifu/funasr1.0/test_local/wav.scp", "/Users/zhifu/funasr1.0/test_local/text.txt"),+ )+ if isinstance(scp_file_list, str):+ scp_file_list = eval(scp_file_list)+ data_type_list = kwargs.get("data_type_list", ("source", "target"))+ jsonl_file_out = kwargs.get(+ "jsonl_file_out", "/Users/zhifu/funasr1.0/test_local/audio_datasets.jsonl"+ )+ gen_jsonl_from_wav_text_list(+ scp_file_list, data_type_list=data_type_list, jsonl_file_out=jsonl_file_out+ )+</code></pre></details></div>+<div class="api-detail" id="e554"><span class="dtag badge-f">function</span><h2><code>gen_jsonl_from_wav_text_list(path, data_type_list, jsonl_file_out, **kwargs)</code></h2><div class="dmod">funasr.datasets.scp2len · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/datasets/audio_datasets/scp2len.py#L13" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Gen jsonl from wav text list.</p>+<h4>Args:</h4>+<ul>+<li><code>path</code> — TODO.</li>+<li><code>data_type_list</code> — TODO.</li>+<li><code>jsonl_file_out</code> — TODO.</li>+<li><code>**kwargs</code> — Additional keyword arguments.</li>+</ul>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code>def gen_jsonl_from_wav_text_list(+ path, data_type_list=("source",), jsonl_file_out: str = None, **kwargs+):+ """Gen jsonl from wav text list.++ Args:+ path: TODO.+ data_type_list: TODO.+ jsonl_file_out: TODO.+ **kwargs: Additional keyword arguments.+ """+ try:+ rank = dist.get_rank()+ world_size = dist.get_world_size()+ except:+ rank = 0+ world_size = 1++ cpu_cores = os.cpu_count() or 1+ print(f"convert wav.scp text to jsonl, ncpu: {cpu_cores}")+ if rank == 0:+ json_dict = {}+ # for data_type, data_file in zip(data_type_list, path):+ data_type = data_type_list[0]+ data_file = path+ json_dict[data_type] = {}+ with open(data_file, "r") as f:++ data_file_lists = f.readlines()+ print("")+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/datasets/audio_datasets/scp2len.py#L13" target="_blank" class="src-more">View full source on GitHub →</a></details></div>+<div class="api-detail" id="e555"><span class="dtag badge-f">function</span><h2><code>parse_context_length(data_list, data_type, id)</code></h2><div class="dmod">funasr.datasets.scp2len · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/datasets/audio_datasets/scp2len.py#L85" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Parse context length.</p>+<h4>Args:</h4>+<ul>+<li><code>data_list</code> — TODO.</li>+<li><code>data_type</code> — TODO.</li>+<li><code>id</code> — TODO.</li>+</ul>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code>def parse_context_length(data_list: list, data_type: str, id=0):+ """Parse context length.++ Args:+ data_list: TODO.+ data_type: TODO.+ id: TODO.+ """+ pbar = tqdm(total=len(data_list), dynamic_ncols=True)+ res = {}+ for i, line in enumerate(data_list):+ pbar.update(1)+ pbar.set_description(f"cpu: {id}")+ lines = line.strip().split(maxsplit=1)+ key = lines[0]+ line = lines[1] if len(lines) > 1 else ""+ line = line.strip()+ if os.path.exists(line):+ waveform, _ = librosa.load(line, sr=16000)+ sample_num = len(waveform)+ context_len = int(sample_num / 16000 * 1000 / 10)+ else:+ context_len = len(line.split()) if " " in line else len(line)+ res[key] = {data_type: line, f"{data_type}_len": context_len}+ return res+</code></pre></details></div>+<div class="api-detail" id="e556"><span class="dtag badge-f">function</span><h2><code>main_hydra(cfg)</code></h2><div class="dmod">funasr.datasets.scp2len · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/datasets/audio_datasets/scp2len.py#L113" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Main hydra.</p>+<h4>Args:</h4>+<ul>+<li><code>cfg</code> — Configuration overrides.</li>+</ul>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code>def main_hydra(cfg: DictConfig):++ """Main hydra.++ Args:+ cfg: Configuration overrides.+ """+ kwargs = OmegaConf.to_container(cfg, resolve=True)+ print(kwargs)++ scp_file_list = kwargs.get("scp_file_list", "/Users/zhifu/funasr1.0/data/list/train_wav.scp")+ # if isinstance(scp_file_list, str):+ # scp_file_list = eval(scp_file_list)+ data_type_list = kwargs.get("data_type_list", ("source",))+ jsonl_file_out = kwargs.get("jsonl_file_out", "/Users/zhifu/funasr1.0/data/list/wav_len.txt")+ gen_jsonl_from_wav_text_list(+ scp_file_list, data_type_list=data_type_list, jsonl_file_out=jsonl_file_out+ )+</code></pre></details></div>+<div class="api-detail" id="e557"><span class="dtag badge-f">function</span><h2><code>gen_jsonl_from_wav_text_list(path, data_type_list, jsonl_file_out, model_dir, **kwargs)</code></h2><div class="dmod">funasr.datasets.sensevoice2jsonl · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/datasets/audio_datasets/sensevoice2jsonl.py#L15" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Gen jsonl from wav text list.</p>+<h4>Args:</h4>+<ul>+<li><code>path</code> — TODO.</li>+<li><code>data_type_list</code> — TODO.</li>+<li><code>jsonl_file_out</code> — TODO.</li>+<li><code>model_dir</code> — TODO.</li>+<li><code>**kwargs</code> — Additional keyword arguments.</li>+</ul>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code>def gen_jsonl_from_wav_text_list(+ path, data_type_list=("source", "target"), jsonl_file_out: str = None, model_dir: str = "iic/SenseVoiceSmall", **kwargs+):+ """Gen jsonl from wav text list.++ Args:+ path: TODO.+ data_type_list: TODO.+ jsonl_file_out: TODO.+ model_dir: TODO.+ **kwargs: Additional keyword arguments.+ """+ try:+ rank = dist.get_rank()+ world_size = dist.get_world_size()+ except:+ rank = 0+ world_size = 1++ cpu_cores = os.cpu_count() or 1+ print(f"convert wav.scp text to jsonl, ncpu: {cpu_cores}")+ if rank == 0:+ json_dict = {}+ for data_type, data_file in zip(data_type_list, path):+ json_dict[data_type] = {}+ with open(data_file, "r") as f:++ data_file_lists = f.readlines()+ lines_for_each_th = (len(data_file_lists) - 1) // cpu_cores + 1+ task_num = cpu_cores if len(data_file_lists) > cpu_cores else 1+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/datasets/audio_datasets/sensevoice2jsonl.py#L15" target="_blank" class="src-more">View full source on GitHub →</a></details></div>+<div class="api-detail" id="e558"><span class="dtag badge-f">function</span><h2><code>contains_punctuation(s)</code></h2><div class="dmod">funasr.datasets.sensevoice2jsonl · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/datasets/audio_datasets/sensevoice2jsonl.py#L131" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Contains punctuation.</p>+<h4>Args:</h4>+<ul>+<li><code>s</code> — TODO.</li>+</ul>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code>def contains_punctuation(s):+ """Contains punctuation.++ Args:+ s: TODO.+ """+ punctuations = (+ string.punctuation ++ ',。、;:?!""''()【】《》〈〉「」『』〔〕[]{}~·…—–'+ )+ return any(char in punctuations for char in s)+</code></pre></details></div>+<div class="api-detail" id="e559"><span class="dtag badge-f">function</span><h2><code>parse_context_length(data_list, data_type, id)</code></h2><div class="dmod">funasr.datasets.sensevoice2jsonl · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/datasets/audio_datasets/sensevoice2jsonl.py#L143" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Parse context length.</p>+<h4>Args:</h4>+<ul>+<li><code>data_list</code> — TODO.</li>+<li><code>data_type</code> — TODO.</li>+<li><code>id</code> — TODO.</li>+</ul>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code>def parse_context_length(data_list: list, data_type: str, id=0):+ """Parse context length.++ Args:+ data_list: TODO.+ data_type: TODO.+ id: TODO.+ """+ pbar = tqdm(total=len(data_list), dynamic_ncols=True)+ res = {}+ for i, line in enumerate(data_list):+ pbar.update(1)+ pbar.set_description(f"cpu: {id}")+ lines = line.strip().split(maxsplit=1)+ key = lines[0]+ line = lines[1] if len(lines) > 1 else ""+ line = line.strip()+ if os.path.exists(line):+ waveform, _ = librosa.load(line, sr=16000)+ sample_num = len(waveform)+ context_len = int(sample_num / 16000 * 1000 / 10)+ else:+ context_len = len(line.split()) if " " in line else len(line)+ if data_type == "source":+ res[key] = {data_type: line, f"{data_type}_len": context_len}+ elif data_type == "target":+ punc = contains_punctuation(line)+ if punc:+ with_or_wo_itn = "<|withitn|>"+ else:+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/datasets/audio_datasets/sensevoice2jsonl.py#L143" target="_blank" class="src-more">View full source on GitHub →</a></details></div>+<div class="api-detail" id="e560"><span class="dtag badge-f">function</span><h2><code>main_hydra(cfg)</code></h2><div class="dmod">funasr.datasets.sensevoice2jsonl · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/datasets/audio_datasets/sensevoice2jsonl.py#L181" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Main hydra.</p>+<h4>Args:</h4>+<ul>+<li><code>cfg</code> — Configuration overrides.</li>+</ul>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code>def main_hydra(cfg: DictConfig):++ """Main hydra.++ Args:+ cfg: Configuration overrides.+ """+ kwargs = OmegaConf.to_container(cfg, resolve=True)+ print(kwargs)++ scp_file_list = kwargs.get(+ "scp_file_list",+ ("/Users/zhifu/funasr1.0/test_local/wav.scp", "/Users/zhifu/funasr1.0/test_local/text.txt"),+ )+ if isinstance(scp_file_list, str):+ scp_file_list = eval(scp_file_list)+ data_type_list = kwargs.get("data_type_list", ("source", "target"))+ jsonl_file_out = kwargs.get(+ "jsonl_file_out", "/Users/zhifu/funasr1.0/test_local/audio_datasets.jsonl"+ )+ model_dir = kwargs.get("model_dir", "iic/SenseVoiceSmall")+ gen_jsonl_from_wav_text_list(+ scp_file_list, data_type_list=data_type_list, jsonl_file_out=jsonl_file_out, model_dir=model_dir+ )+</code></pre></details></div>+<div class="api-detail" id="e561"><span class="dtag badge-f">function</span><h2><code>gen_scp_from_jsonl(jsonl_file, jsonl_file_out, ncpu)</code></h2><div class="dmod">funasr.datasets.update_jsonl · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/datasets/audio_datasets/update_jsonl.py#L15" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Gen scp from jsonl.</p>+<h4>Args:</h4>+<ul>+<li><code>jsonl_file</code> — TODO.</li>+<li><code>jsonl_file_out</code> — TODO.</li>+<li><code>ncpu</code> — TODO.</li>+</ul>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code>def gen_scp_from_jsonl(jsonl_file, jsonl_file_out, ncpu):+ """Gen scp from jsonl.++ Args:+ jsonl_file: TODO.+ jsonl_file_out: TODO.+ ncpu: TODO.+ """+ jsonl_file_out_f = open(jsonl_file_out, "w")+ with open(jsonl_file, encoding="utf-8") as fin:+ lines = fin.readlines()++ num_total = len(lines)+ if ncpu > 1:+ # 使用ThreadPoolExecutor限制并发线程数+ with ThreadPoolExecutor(max_workers=ncpu) as executor:+ # 提交任务到线程池+ futures = {executor.submit(update_data, lines, i) for i in tqdm(range(num_total))}++ # 等待所有任务完成,这会阻塞直到所有提交的任务完成+ for future in concurrent.futures.as_completed(futures):+ # 这里可以添加额外的逻辑来处理完成的任务,但在这个例子中我们只是等待+ pass+ else:+ for i in range(num_total):+ update_data(lines, i)+ logging.info("All audio durations have been processed.")++ for line in lines:++</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/datasets/audio_datasets/update_jsonl.py#L15" target="_blank" class="src-more">View full source on GitHub →</a></details></div>+<div class="api-detail" id="e562"><span class="dtag badge-f">function</span><h2><code>update_data(lines, i)</code></h2><div class="dmod">funasr.datasets.update_jsonl · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/datasets/audio_datasets/update_jsonl.py#L51" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Update data.</p>+<h4>Args:</h4>+<ul>+<li><code>lines</code> — TODO.</li>+<li><code>i</code> — TODO.</li>+</ul>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code>def update_data(lines, i):+ """Update data.++ Args:+ lines: TODO.+ i: TODO.+ """+ line = lines[i]+ data = json.loads(line.strip())++ wav_path = data["source"].replace("/cpfs01", "/cpfs_speech/data")+ if os.path.exists(wav_path):+ waveform, _ = librosa.load(wav_path, sr=16000)+ sample_num = len(waveform)+ source_len = int(sample_num / 16000 * 1000 / 10)+ source_len_old = data["source_len"]+ # if (source_len_old - source_len) > 100 or (source_len - source_len_old) > 100:+ # logging.info(f"old: {source_len_old}, new: {source_len}, wav: {wav_path}")+ data["source_len"] = source_len+ data["source"] = wav_path+ jsonl_line = json.dumps(data, ensure_ascii=False)+ lines[i] = jsonl_line+</code></pre></details></div>+<div class="api-detail" id="e563"><span class="dtag badge-f">function</span><h2><code>update_wav_len(jsonl_file_list_in, jsonl_file_out_dir, ncpu)</code></h2><div class="dmod">funasr.datasets.update_jsonl · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/datasets/audio_datasets/update_jsonl.py#L75" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Update wav len.</p>+<h4>Args:</h4>+<ul>+<li><code>jsonl_file_list_in</code> — TODO.</li>+<li><code>jsonl_file_out_dir</code> — TODO.</li>+<li><code>ncpu</code> — TODO.</li>+</ul>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code>def update_wav_len(jsonl_file_list_in, jsonl_file_out_dir, ncpu=1):++ """Update wav len.++ Args:+ jsonl_file_list_in: TODO.+ jsonl_file_out_dir: TODO.+ ncpu: TODO.+ """+ os.makedirs(jsonl_file_out_dir, exist_ok=True)+ with open(jsonl_file_list_in, "r") as f:+ data_file_lists = f.readlines()++ for i, jsonl in enumerate(data_file_lists):+ filename_with_extension = os.path.basename(jsonl.strip())+ jsonl_file_out = os.path.join(jsonl_file_out_dir, filename_with_extension)+ logging.info(f"{i}/{len(data_file_lists)}, jsonl: {jsonl}, {jsonl_file_out}")++ gen_scp_from_jsonl(jsonl.strip(), jsonl_file_out, ncpu)+</code></pre></details></div>+<div class="api-detail" id="e564"><span class="dtag badge-f">function</span><h2><code>main_hydra(cfg)</code></h2><div class="dmod">funasr.datasets.update_jsonl · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/datasets/audio_datasets/update_jsonl.py#L97" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Main hydra.</p>+<h4>Args:</h4>+<ul>+<li><code>cfg</code> — Configuration overrides.</li>+</ul>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code>def main_hydra(cfg: DictConfig):++ """Main hydra.++ Args:+ cfg: Configuration overrides.+ """+ kwargs = OmegaConf.to_container(cfg, resolve=True)+ logging.info(kwargs)++ jsonl_file_list_in = kwargs.get(+ "jsonl_file_list_in", "/Users/zhifu/funasr1.0/data/list/data_jsonl.list"+ )+ jsonl_file_out_dir = kwargs.get("jsonl_file_out_dir", "/Users/zhifu/funasr1.0/data_tmp")+ ncpu = kwargs.get("ncpu", 1)+ update_wav_len(jsonl_file_list_in, jsonl_file_out_dir, ncpu)+</code></pre></details></div>+<div class="api-detail" id="e565"><span class="dtag badge-f">function</span><h2><code>DataloaderMapStyle(frontend, tokenizer, **kwargs)</code></h2><div class="dmod">funasr.datasets.dataloader_entry · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/datasets/dataloader_entry.py#L8" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Dataloadermapstyle.</p>+<h4>Args:</h4>+<ul>+<li><code>frontend</code> — Audio frontend for feature extraction.</li>+<li><code>tokenizer</code> — Tokenizer instance for text encoding/decoding.</li>+<li><code>**kwargs</code> — Additional keyword arguments.</li>+</ul>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code>def DataloaderMapStyle(frontend=None, tokenizer=None, **kwargs):+ # dataset+ """Dataloadermapstyle.++ Args:+ frontend: Audio frontend for feature extraction.+ tokenizer: Tokenizer instance for text encoding/decoding.+ **kwargs: Additional keyword arguments.+ """+ logging.info("Build dataloader")+ dataset_class = tables.dataset_classes.get(kwargs.get("dataset", "AudioDataset"))+ dataset_tr = dataset_class(+ kwargs.get("train_data_set_list"),+ frontend=frontend,+ tokenizer=tokenizer,+ is_training=True,+ **kwargs.get("dataset_conf"),+ )+ dataset_val = dataset_class(+ kwargs.get("valid_data_set_list"),+ frontend=frontend,+ tokenizer=tokenizer,+ is_training=False,+ **kwargs.get("dataset_conf"),+ )++ # dataloader+ batch_sampler = kwargs["dataset_conf"].get("batch_sampler", "BatchSampler")+ batch_sampler_val = None+ if batch_sampler is not None:+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/datasets/dataloader_entry.py#L8" target="_blank" class="src-more">View full source on GitHub →</a></details></div>+<div class="api-detail" id="e566"><span class="dtag badge-c">class</span><h2>DataloaderMapStyle</h2><div class="dmod">funasr.datasets.dataloader_entry · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/datasets/dataloader_entry.py#L55" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p class="muted">No documentation yet.</p></div><details class="src-block"><summary>📄 Source code</summary><pre><code>class DataloaderMapStyle:+ def __init__(self, frontend=None, tokenizer=None, **kwargs):+ # dataset+ """Initialize DataloaderMapStyle.++ Args:+ frontend: Audio frontend for feature extraction.+ tokenizer: Tokenizer instance for text encoding/decoding.+ **kwargs: Additional keyword arguments.+ """+ logging.info("Build dataloader")++ dataset_class = tables.dataset_classes.get(kwargs.get("dataset", "AudioDataset"))+ dataset_tr = None+ # split dataset+ self.data_split_num = kwargs["dataset_conf"].get("data_split_num", 1)+ if self.data_split_num == 1:+ dataset_tr = dataset_class(+ kwargs.get("train_data_set_list"),+ frontend=frontend,+ tokenizer=tokenizer,+ is_training=True,+ **kwargs.get("dataset_conf"),+ )+ dataset_val = dataset_class(+ kwargs.get("valid_data_set_list"),+ frontend=frontend,+ tokenizer=tokenizer,+ is_training=False,+ **kwargs.get("dataset_conf"),+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/datasets/dataloader_entry.py#L55" target="_blank" class="src-more">View full source on GitHub →</a></details><h3 class="mt">Methods</h3><div class="mblk"><code>.build_iter(epoch, data_split_i, start_step, **kwargs)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/datasets/dataloader_entry.py#L96" target="_blank" class="gh-link">L96</a><div class="mdoc"><p>Build iter.</p>+<h4>Args:</h4>+<ul>+<li><code>epoch</code> — TODO.</li>+<li><code>data_split_i</code> — TODO.</li>+<li><code>start_step</code> — TODO.</li>+<li><code>**kwargs</code> — Additional keyword arguments.</li>+</ul>+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def build_iter(self, epoch=0, data_split_i=0, start_step=0, **kwargs):++ # reload dataset slice+ """Build iter.++ Args:+ epoch: TODO.+ data_split_i: TODO.+ start_step: TODO.+ **kwargs: Additional keyword arguments.+ """+ if self.data_split_num > 1:+ del self.dataset_tr+ self.dataset_tr = self.dataset_class(+ self.kwargs.get("train_data_set_list"),+ frontend=self.frontend,+ tokenizer=self.tokenizer,+ is_training=True,+ **self.kwargs.get("dataset_conf"),+ data_split_i=data_split_i,+ )++ # dataloader+ batch_sampler = self.kwargs["dataset_conf"].get("batch_sampler", "BatchSampler")+ batch_sampler_val = None+ if batch_sampler is not None:+ batch_sampler_class = tables.batch_sampler_classes.get(batch_sampler)+ batch_sampler = batch_sampler_class(+ self.dataset_tr, start_step=start_step, **self.kwargs.get("dataset_conf")+ )+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/datasets/dataloader_entry.py#L96" target="_blank" class="src-more">View full source →</a></details></div></div>+<div class="api-detail" id="e567"><span class="dtag badge-m">method</span><h2><code>DataloaderMapStyle.build_iter(epoch, data_split_i, start_step, **kwargs)</code></h2><div class="dmod">funasr.datasets.dataloader_entry.DataloaderMapStyle · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/datasets/dataloader_entry.py#L96" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Build iter.</p>+<h4>Args:</h4>+<ul>+<li><code>epoch</code> — TODO.</li>+<li><code>data_split_i</code> — TODO.</li>+<li><code>start_step</code> — TODO.</li>+<li><code>**kwargs</code> — Additional keyword arguments.</li>+</ul>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def build_iter(self, epoch=0, data_split_i=0, start_step=0, **kwargs):++ # reload dataset slice+ """Build iter.++ Args:+ epoch: TODO.+ data_split_i: TODO.+ start_step: TODO.+ **kwargs: Additional keyword arguments.+ """+ if self.data_split_num > 1:+ del self.dataset_tr+ self.dataset_tr = self.dataset_class(+ self.kwargs.get("train_data_set_list"),+ frontend=self.frontend,+ tokenizer=self.tokenizer,+ is_training=True,+ **self.kwargs.get("dataset_conf"),+ data_split_i=data_split_i,+ )++ # dataloader+ batch_sampler = self.kwargs["dataset_conf"].get("batch_sampler", "BatchSampler")+ batch_sampler_val = None+ if batch_sampler is not None:+ batch_sampler_class = tables.batch_sampler_classes.get(batch_sampler)+ batch_sampler = batch_sampler_class(+ self.dataset_tr, start_step=start_step, **self.kwargs.get("dataset_conf")+ )+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/datasets/dataloader_entry.py#L96" target="_blank" class="src-more">View full source on GitHub →</a></details></div>+<div class="api-detail" id="e568"><span class="dtag badge-f">function</span><h2><code>DataloaderIterable(frontend, tokenizer, **kwargs)</code></h2><div class="dmod">funasr.datasets.dataloader_entry · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/datasets/dataloader_entry.py#L143" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Dataloaderiterable.</p>+<h4>Args:</h4>+<ul>+<li><code>frontend</code> — Audio frontend for feature extraction.</li>+<li><code>tokenizer</code> — Tokenizer instance for text encoding/decoding.</li>+<li><code>**kwargs</code> — Additional keyword arguments.</li>+</ul>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code>def DataloaderIterable(frontend=None, tokenizer=None, **kwargs):+ """Dataloaderiterable.++ Args:+ frontend: Audio frontend for feature extraction.+ tokenizer: Tokenizer instance for text encoding/decoding.+ **kwargs: Additional keyword arguments.+ """+ logging.info("Build dataloader")+ dataset_class = tables.dataset_classes.get(kwargs.get("dataset", "LargeDataset"))+ dataset_tr = dataset_class(+ kwargs.get("train_data_set_list"),+ frontend=frontend,+ tokenizer=tokenizer,+ is_training=True,+ **kwargs.get("dataset_conf"),+ )+ dataset_val = dataset_class(+ kwargs.get("valid_data_set_list"),+ frontend=frontend,+ tokenizer=tokenizer,+ is_training=False,+ **kwargs.get("dataset_conf"),+ )++ return dataset_tr, dataset_val+</code></pre></details></div>+<div class="api-detail" id="e569"><span class="dtag badge-c">class</span><h2>LabelSmoothingLoss</h2><div class="dmod">funasr.losses.label_smoothing_loss · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/losses/label_smoothing_loss.py#L14" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><ul>+<li><code>Label</code> — smoothing loss.</li>+</ul>+<p>:param int size: the number of class</p>+<p>:param int padding_idx: ignored class id</p>+<p>:param float smoothing: smoothing rate (0.0 means the conventional CE)</p>+<p>:param bool normalize_length: normalize loss by sequence length if True</p>+<p>:param torch.nn.Module criterion: loss function to be smoothed</p>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code>class LabelSmoothingLoss(nn.Module):+ """Label-smoothing loss.++ :param int size: the number of class+ :param int padding_idx: ignored class id+ :param float smoothing: smoothing rate (0.0 means the conventional CE)+ :param bool normalize_length: normalize loss by sequence length if True+ :param torch.nn.Module criterion: loss function to be smoothed+ """++ def __init__(+ self,+ size,+ padding_idx,+ smoothing,+ normalize_length=False,+ criterion=nn.KLDivLoss(reduction="none"),+ ):+ """Construct an LabelSmoothingLoss object."""+ super(LabelSmoothingLoss, self).__init__()+ self.criterion = criterion+ self.padding_idx = padding_idx+ self.confidence = 1.0 - smoothing+ self.smoothing = smoothing+ self.size = size+ self.true_dist = None+ self.normalize_length = normalize_length++ def forward(self, x, target):+ """Compute loss between x and target.+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/losses/label_smoothing_loss.py#L14" target="_blank" class="src-more">View full source on GitHub →</a></details><h3 class="mt">Methods</h3><div class="mblk"><code>.forward(x, target)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/losses/label_smoothing_loss.py#L42" target="_blank" class="gh-link">L42</a><div class="mdoc"><p>Compute loss between x and target.</p>+<p>:param torch.Tensor x: prediction (batch, seqlen, class)</p>+<p>:param torch.Tensor target:</p>+<p>target signal masked with self.padding_id (batch, seqlen)</p>+<p>:return: scalar float value</p>+<p>:rtype torch.Tensor</p>+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def forward(self, x, target):+ """Compute loss between x and target.++ :param torch.Tensor x: prediction (batch, seqlen, class)+ :param torch.Tensor target:+ target signal masked with self.padding_id (batch, seqlen)+ :return: scalar float value+ :rtype torch.Tensor+ """+ assert x.size(2) == self.size+ batch_size = x.size(0)+ x = x.contiguous().view(-1, self.size)+ target = target.contiguous().view(-1)+ with torch.no_grad():+ true_dist = x.clone()+ true_dist.fill_(self.smoothing / (self.size - 1))+ ignore = target == self.padding_idx # (B,)+ total = len(target) - ignore.sum().item()+ target = target.masked_fill(ignore, 0) # avoid -1 index+ true_dist.scatter_(1, target.unsqueeze(1), self.confidence)+ kl = self.criterion(torch.log_softmax(x, dim=1), true_dist)+ denom = total if self.normalize_length else batch_size+ return kl.masked_fill(ignore.unsqueeze(1), 0).sum() / denom+</code></pre></details></div></div>+<div class="api-detail" id="e570"><span class="dtag badge-m">method</span><h2><code>LabelSmoothingLoss.forward(x, target)</code></h2><div class="dmod">funasr.losses.label_smoothing_loss.LabelSmoothingLoss · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/losses/label_smoothing_loss.py#L42" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Compute loss between x and target.</p>+<p>:param torch.Tensor x: prediction (batch, seqlen, class)</p>+<p>:param torch.Tensor target:</p>+<p>target signal masked with self.padding_id (batch, seqlen)</p>+<p>:return: scalar float value</p>+<p>:rtype torch.Tensor</p>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def forward(self, x, target):+ """Compute loss between x and target.++ :param torch.Tensor x: prediction (batch, seqlen, class)+ :param torch.Tensor target:+ target signal masked with self.padding_id (batch, seqlen)+ :return: scalar float value+ :rtype torch.Tensor+ """+ assert x.size(2) == self.size+ batch_size = x.size(0)+ x = x.contiguous().view(-1, self.size)+ target = target.contiguous().view(-1)+ with torch.no_grad():+ true_dist = x.clone()+ true_dist.fill_(self.smoothing / (self.size - 1))+ ignore = target == self.padding_idx # (B,)+ total = len(target) - ignore.sum().item()+ target = target.masked_fill(ignore, 0) # avoid -1 index+ true_dist.scatter_(1, target.unsqueeze(1), self.confidence)+ kl = self.criterion(torch.log_softmax(x, dim=1), true_dist)+ denom = total if self.normalize_length else batch_size+ return kl.masked_fill(ignore.unsqueeze(1), 0).sum() / denom+</code></pre></details></div>+<div class="api-detail" id="e571"><span class="dtag badge-c">class</span><h2>SequenceBinaryCrossEntropy</h2><div class="dmod">funasr.losses.label_smoothing_loss · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/losses/label_smoothing_loss.py#L67" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p class="muted">No documentation yet.</p></div><details class="src-block"><summary>📄 Source code</summary><pre><code>class SequenceBinaryCrossEntropy(nn.Module):+ def __init__(self, normalize_length=False, criterion=nn.BCEWithLogitsLoss(reduction="none")):+ """Initialize SequenceBinaryCrossEntropy.++ Args:+ normalize_length: TODO.+ criterion: TODO.+ """+ super().__init__()+ self.normalize_length = normalize_length+ self.criterion = criterion++ def forward(self, pred, label, lengths):+ """Forward pass for training.++ Args:+ pred: TODO.+ label: TODO.+ lengths: TODO.+ """+ pad_mask = make_pad_mask(lengths, maxlen=pred.shape[1]).to(pred.device)+ loss = self.criterion(pred, label)+ denom = (~pad_mask).sum() if self.normalize_length else pred.shape[0]+ return loss.masked_fill(pad_mask.unsqueeze(-1), 0).sum() / denom+</code></pre></details><h3 class="mt">Methods</h3><div class="mblk"><code>.forward(pred, label, lengths)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/losses/label_smoothing_loss.py#L79" target="_blank" class="gh-link">L79</a><div class="mdoc"><p>Forward pass for training.</p>+<h4>Args:</h4>+<ul>+<li><code>pred</code> — TODO.</li>+<li><code>label</code> — TODO.</li>+<li><code>lengths</code> — TODO.</li>+</ul>+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def forward(self, pred, label, lengths):+ """Forward pass for training.++ Args:+ pred: TODO.+ label: TODO.+ lengths: TODO.+ """+ pad_mask = make_pad_mask(lengths, maxlen=pred.shape[1]).to(pred.device)+ loss = self.criterion(pred, label)+ denom = (~pad_mask).sum() if self.normalize_length else pred.shape[0]+ return loss.masked_fill(pad_mask.unsqueeze(-1), 0).sum() / denom+</code></pre></details></div></div>+<div class="api-detail" id="e572"><span class="dtag badge-m">method</span><h2><code>SequenceBinaryCrossEntropy.forward(pred, label, lengths)</code></h2><div class="dmod">funasr.losses.label_smoothing_loss.SequenceBinaryCrossEntropy · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/losses/label_smoothing_loss.py#L79" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Forward pass for training.</p>+<h4>Args:</h4>+<ul>+<li><code>pred</code> — TODO.</li>+<li><code>label</code> — TODO.</li>+<li><code>lengths</code> — TODO.</li>+</ul>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def forward(self, pred, label, lengths):+ """Forward pass for training.++ Args:+ pred: TODO.+ label: TODO.+ lengths: TODO.+ """+ pad_mask = make_pad_mask(lengths, maxlen=pred.shape[1]).to(pred.device)+ loss = self.criterion(pred, label)+ denom = (~pad_mask).sum() if self.normalize_length else pred.shape[0]+ return loss.masked_fill(pad_mask.unsqueeze(-1), 0).sum() / denom+</code></pre></details></div>+<div class="api-detail" id="e573"><span class="dtag badge-c">class</span><h2>NllLoss</h2><div class="dmod">funasr.losses.label_smoothing_loss · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/losses/label_smoothing_loss.py#L93" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Nll loss.</p>+<p>:param int size: the number of class</p>+<p>:param int padding_idx: ignored class id</p>+<p>:param bool normalize_length: normalize loss by sequence length if True</p>+<p>:param torch.nn.Module criterion: loss function</p>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code>class NllLoss(nn.Module):+ """Nll loss.++ :param int size: the number of class+ :param int padding_idx: ignored class id+ :param bool normalize_length: normalize loss by sequence length if True+ :param torch.nn.Module criterion: loss function+ """++ def __init__(+ self,+ size,+ padding_idx,+ normalize_length=False,+ criterion=nn.NLLLoss(reduction="none"),+ ):+ """Construct an NllLoss object."""+ super(NllLoss, self).__init__()+ self.criterion = criterion+ self.padding_idx = padding_idx+ self.size = size+ self.true_dist = None+ self.normalize_length = normalize_length++ def forward(self, x, target):+ """Compute loss between x and target.++ :param torch.Tensor x: prediction (batch, seqlen, class)+ :param torch.Tensor target:+ target signal masked with self.padding_id (batch, seqlen)+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/losses/label_smoothing_loss.py#L93" target="_blank" class="src-more">View full source on GitHub →</a></details><h3 class="mt">Methods</h3><div class="mblk"><code>.forward(x, target)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/losses/label_smoothing_loss.py#L117" target="_blank" class="gh-link">L117</a><div class="mdoc"><p>Compute loss between x and target.</p>+<p>:param torch.Tensor x: prediction (batch, seqlen, class)</p>+<p>:param torch.Tensor target:</p>+<p>target signal masked with self.padding_id (batch, seqlen)</p>+<p>:return: scalar float value</p>+<p>:rtype torch.Tensor</p>+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def forward(self, x, target):+ """Compute loss between x and target.++ :param torch.Tensor x: prediction (batch, seqlen, class)+ :param torch.Tensor target:+ target signal masked with self.padding_id (batch, seqlen)+ :return: scalar float value+ :rtype torch.Tensor+ """+ assert x.size(2) == self.size+ batch_size = x.size(0)+ x = x.view(-1, self.size)+ target = target.view(-1)+ with torch.no_grad():+ ignore = target == self.padding_idx # (B,)+ total = len(target) - ignore.sum().item()+ target = target.masked_fill(ignore, 0) # avoid -1 index+ kl = self.criterion(x, target)+ denom = total if self.normalize_length else batch_size+ return kl.masked_fill(ignore, 0).sum() / denom+</code></pre></details></div></div>+<div class="api-detail" id="e574"><span class="dtag badge-m">method</span><h2><code>NllLoss.forward(x, target)</code></h2><div class="dmod">funasr.losses.label_smoothing_loss.NllLoss · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/losses/label_smoothing_loss.py#L117" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Compute loss between x and target.</p>+<p>:param torch.Tensor x: prediction (batch, seqlen, class)</p>+<p>:param torch.Tensor target:</p>+<p>target signal masked with self.padding_id (batch, seqlen)</p>+<p>:return: scalar float value</p>+<p>:rtype torch.Tensor</p>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def forward(self, x, target):+ """Compute loss between x and target.++ :param torch.Tensor x: prediction (batch, seqlen, class)+ :param torch.Tensor target:+ target signal masked with self.padding_id (batch, seqlen)+ :return: scalar float value+ :rtype torch.Tensor+ """+ assert x.size(2) == self.size+ batch_size = x.size(0)+ x = x.view(-1, self.size)+ target = target.view(-1)+ with torch.no_grad():+ ignore = target == self.padding_idx # (B,)+ total = len(target) - ignore.sum().item()+ target = target.masked_fill(ignore, 0) # avoid -1 index+ kl = self.criterion(x, target)+ denom = total if self.normalize_length else batch_size+ return kl.masked_fill(ignore, 0).sum() / denom+</code></pre></details></div>+<div class="api-detail" id="e575"><span class="dtag badge-c">class</span><h2>CustomLambdaLR</h2><div class="dmod">funasr.schedulers.lambdalr_cus · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/schedulers/lambdalr_cus.py#L19" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p class="muted">No documentation yet.</p></div><details class="src-block"><summary>📄 Source code</summary><pre><code>class CustomLambdaLR(_LRScheduler):+ def __init__(+ self,+ optimizer,+ warmup_steps: int = 25000,+ total_steps: int = 500000,+ last_epoch=-1,+ verbose=False,+ ):+ """Initialize CustomLambdaLR.++ Args:+ optimizer: TODO.+ warmup_steps: TODO.+ total_steps: TODO.+ last_epoch: TODO.+ verbose: TODO.+ """+ self.warmup_steps = warmup_steps+ self.total_steps = total_steps+ super().__init__(optimizer, last_epoch, verbose)++ def get_lr(self):++ """Get lr."""+ step = self.last_epoch + 1+ if step < self.warmup_steps:+ lr_scale = step / self.warmup_steps+ else:+ lr_scale = max(+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/schedulers/lambdalr_cus.py#L19" target="_blank" class="src-more">View full source on GitHub →</a></details><h3 class="mt">Methods</h3><div class="mblk"><code>.get_lr()</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/schedulers/lambdalr_cus.py#L41" target="_blank" class="gh-link">L41</a><div class="mdoc"><p>Get lr.</p>+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def get_lr(self):++ """Get lr."""+ step = self.last_epoch + 1+ if step < self.warmup_steps:+ lr_scale = step / self.warmup_steps+ else:+ lr_scale = max(+ 0.0, 1 - (step - self.warmup_steps) / (self.total_steps - self.warmup_steps)+ )+ return [base_lr * lr_scale for base_lr in self.base_lrs]+</code></pre></details></div></div>+<div class="api-detail" id="e576"><span class="dtag badge-m">method</span><h2><code>CustomLambdaLR.get_lr()</code></h2><div class="dmod">funasr.schedulers.lambdalr_cus.CustomLambdaLR · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/schedulers/lambdalr_cus.py#L41" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Get lr.</p>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def get_lr(self):++ """Get lr."""+ step = self.last_epoch + 1+ if step < self.warmup_steps:+ lr_scale = step / self.warmup_steps+ else:+ lr_scale = max(+ 0.0, 1 - (step - self.warmup_steps) / (self.total_steps - self.warmup_steps)+ )+ return [base_lr * lr_scale for base_lr in self.base_lrs]+</code></pre></details></div>+<div class="api-detail" id="e577"><span class="dtag badge-c">class</span><h2>NoamLR</h2><div class="dmod">funasr.schedulers.noam_lr · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/schedulers/noam_lr.py#L12" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>The LR scheduler proposed by Noam</p>+<p>Ref:</p>+<p>"Attention Is All You Need", https://arxiv.org/pdf/1706.03762.pdf</p>+<ul>+<li><code>FIXME(kamo)</code> — PyTorch doesn't provide _LRScheduler as public class,</li>+</ul>+<p>thus the behaviour isn't guaranteed at forward PyTorch version.</p>+<ul>+<li><code>NOTE(kamo)</code> — The "model_size" in original implementation is derived from</li>+</ul>+<p>the model, but in this implementation, this parameter is a constant value.</p>+<p>You need to change it if the model is changed.</p>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code>class NoamLR(_LRScheduler, AbsBatchStepScheduler):+ """The LR scheduler proposed by Noam++ Ref:+ "Attention Is All You Need", https://arxiv.org/pdf/1706.03762.pdf++ FIXME(kamo): PyTorch doesn't provide _LRScheduler as public class,+ thus the behaviour isn't guaranteed at forward PyTorch version.++ NOTE(kamo): The "model_size" in original implementation is derived from+ the model, but in this implementation, this parameter is a constant value.+ You need to change it if the model is changed.++ """++ def __init__(+ self,+ optimizer: torch.optim.Optimizer,+ model_size: Union[int, float] = 320,+ warmup_steps: Union[int, float] = 25000,+ last_epoch: int = -1,+ ):+ """Initialize NoamLR.++ Args:+ optimizer: TODO.+ model_size: Size/dimension parameter.+ warmup_steps: TODO.+ last_epoch: TODO.+ """+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/schedulers/noam_lr.py#L12" target="_blank" class="src-more">View full source on GitHub →</a></details><h3 class="mt">Methods</h3><div class="mblk"><code>.lr_for_WarmupLR(lr)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/schedulers/noam_lr.py#L56" target="_blank" class="gh-link">L56</a><div class="mdoc"><p>Lr for warmuplr.</p>+<h4>Args:</h4>+<ul>+<li><code>lr</code> — TODO.</li>+</ul>+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def lr_for_WarmupLR(self, lr: float) -> float:+ """Lr for warmuplr.++ Args:+ lr: TODO.+ """+ return lr / self.model_size**0.5 / self.warmup_steps**0.5+</code></pre></details></div><div class="mblk"><code>.get_lr()</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/schedulers/noam_lr.py#L71" target="_blank" class="gh-link">L71</a><div class="mdoc"><p>Get lr.</p>+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def get_lr(self):+ """Get lr."""+ step_num = self.last_epoch + 1+ return [+ lr * self.model_size**-0.5 * min(step_num**-0.5, step_num * self.warmup_steps**-1.5)+ for lr in self.base_lrs+ ]+</code></pre></details></div></div>+<div class="api-detail" id="e578"><span class="dtag badge-m">method</span><h2><code>NoamLR.lr_for_WarmupLR(lr)</code></h2><div class="dmod">funasr.schedulers.noam_lr.NoamLR · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/schedulers/noam_lr.py#L56" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Lr for warmuplr.</p>+<h4>Args:</h4>+<ul>+<li><code>lr</code> — TODO.</li>+</ul>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def lr_for_WarmupLR(self, lr: float) -> float:+ """Lr for warmuplr.++ Args:+ lr: TODO.+ """+ return lr / self.model_size**0.5 / self.warmup_steps**0.5+</code></pre></details></div>+<div class="api-detail" id="e579"><span class="dtag badge-m">method</span><h2><code>NoamLR.get_lr()</code></h2><div class="dmod">funasr.schedulers.noam_lr.NoamLR · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/schedulers/noam_lr.py#L71" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Get lr.</p>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def get_lr(self):+ """Get lr."""+ step_num = self.last_epoch + 1+ return [+ lr * self.model_size**-0.5 * min(step_num**-0.5, step_num * self.warmup_steps**-1.5)+ for lr in self.base_lrs+ ]+</code></pre></details></div>+<div class="api-detail" id="e580"><span class="dtag badge-c">class</span><h2>TriStageLR</h2><div class="dmod">funasr.schedulers.tri_stage_scheduler · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/schedulers/tri_stage_scheduler.py#L15" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p class="muted">No documentation yet.</p></div><details class="src-block"><summary>📄 Source code</summary><pre><code>class TriStageLR(_LRScheduler, AbsBatchStepScheduler):+ def __init__(+ self,+ optimizer: torch.optim.Optimizer,+ last_epoch: int = -1,+ phase_ratio: Optional[List[float]] = None,+ init_lr_scale: float = 0.01,+ final_lr_scale: float = 0.01,+ ):+ """Initialize TriStageLR.++ Args:+ optimizer: TODO.+ last_epoch: TODO.+ phase_ratio: TODO.+ init_lr_scale: TODO.+ final_lr_scale: TODO.+ """+ self.optimizer = optimizer+ self.last_epoch = last_epoch+ self.phase_ratio = phase_ratio+ self.init_lr_scale = init_lr_scale+ self.final_lr_scale = final_lr_scale+ self.optimizer_lr = self.optimizer.defaults["lr"]++ def init_tri_stage_scheudler(self, max_update):+ """Init tri stage scheudler.++ Args:+ max_update: TODO.+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/schedulers/tri_stage_scheduler.py#L15" target="_blank" class="src-more">View full source on GitHub →</a></details><h3 class="mt">Methods</h3><div class="mblk"><code>.init_tri_stage_scheudler(max_update)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/schedulers/tri_stage_scheduler.py#L40" target="_blank" class="gh-link">L40</a><div class="mdoc"><p>Init tri stage scheudler.</p>+<h4>Args:</h4>+<ul>+<li><code>max_update</code> — TODO.</li>+</ul>+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def init_tri_stage_scheudler(self, max_update):+ """Init tri stage scheudler.++ Args:+ max_update: TODO.+ """+ self.max_update = max_update+ self.peak_lr = self.optimizer_lr+ self.init_lr = self.init_lr_scale * self.optimizer_lr+ self.final_lr = self.final_lr_scale * self.optimizer_lr++ assert self.max_update > 0+ assert sum(self.phase_ratio) == 1, "phase ratios must add up to 1"+ assert len(self.phase_ratio) == 3+ self.warmup_steps = int(self.max_update * self.phase_ratio[0])+ self.hold_steps = int(self.max_update * self.phase_ratio[1])+ self.decay_steps = int(self.max_update * self.phase_ratio[2])++ self.warmup_rate = (+ (self.peak_lr - self.init_lr) / self.warmup_steps if self.warmup_steps != 0 else 0+ )+ self.decay_factor = -math.log(self.final_lr_scale) / self.decay_steps++ # initial learning rate+ self.lr = self.init_lr++ # __init__() must be invoked before setting field+ # because step() is also invoked in __init__()+ self.set_optimizer_lr(self.lr)+ super().__init__(self.optimizer, self.last_epoch)+</code></pre></details></div><div class="mblk"><code>.step_update(num_updates)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/schedulers/tri_stage_scheduler.py#L96" target="_blank" class="gh-link">L96</a><div class="mdoc"><p>Update the learning rate after each update.</p>+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def step_update(self, num_updates):+ """Update the learning rate after each update."""+ stage, steps_in_stage = self._decide_stage(num_updates)+ if stage == 0:+ self.lr = self.init_lr + self.warmup_rate * steps_in_stage+ elif stage == 1:+ self.lr = self.peak_lr+ elif stage == 2:+ self.lr = self.peak_lr * math.exp(-self.decay_factor * steps_in_stage)+ elif stage == 3:+ self.lr = self.final_lr+ else:+ raise ValueError("Undefined stage")+ self.set_optimizer_lr(self.lr)+</code></pre></details></div><div class="mblk"><code>.set_optimizer_lr(lr)</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/schedulers/tri_stage_scheduler.py#L111" target="_blank" class="gh-link">L111</a><div class="mdoc"><p>Set optimizer lr.</p>+<h4>Args:</h4>+<ul>+<li><code>lr</code> — TODO.</li>+</ul>+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def set_optimizer_lr(self, lr):+ """Set optimizer lr.++ Args:+ lr: TODO.+ """+ for param_group in self.optimizer.param_groups:+ param_group["lr"] = lr+</code></pre></details></div><div class="mblk"><code>.get_lr()</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/schedulers/tri_stage_scheduler.py#L120" target="_blank" class="gh-link">L120</a><div class="mdoc"><p>Get lr.</p>+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def get_lr(self):+ """Get lr."""+ step_num = self.last_epoch + 1+ self.step_update(step_num)+ return [self.lr]+</code></pre></details></div></div>+<div class="api-detail" id="e581"><span class="dtag badge-m">method</span><h2><code>TriStageLR.init_tri_stage_scheudler(max_update)</code></h2><div class="dmod">funasr.schedulers.tri_stage_scheduler.TriStageLR · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/schedulers/tri_stage_scheduler.py#L40" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Init tri stage scheudler.</p>+<h4>Args:</h4>+<ul>+<li><code>max_update</code> — TODO.</li>+</ul>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def init_tri_stage_scheudler(self, max_update):+ """Init tri stage scheudler.++ Args:+ max_update: TODO.+ """+ self.max_update = max_update+ self.peak_lr = self.optimizer_lr+ self.init_lr = self.init_lr_scale * self.optimizer_lr+ self.final_lr = self.final_lr_scale * self.optimizer_lr++ assert self.max_update > 0+ assert sum(self.phase_ratio) == 1, "phase ratios must add up to 1"+ assert len(self.phase_ratio) == 3+ self.warmup_steps = int(self.max_update * self.phase_ratio[0])+ self.hold_steps = int(self.max_update * self.phase_ratio[1])+ self.decay_steps = int(self.max_update * self.phase_ratio[2])++ self.warmup_rate = (+ (self.peak_lr - self.init_lr) / self.warmup_steps if self.warmup_steps != 0 else 0+ )+ self.decay_factor = -math.log(self.final_lr_scale) / self.decay_steps++ # initial learning rate+ self.lr = self.init_lr++ # __init__() must be invoked before setting field+ # because step() is also invoked in __init__()+ self.set_optimizer_lr(self.lr)+ super().__init__(self.optimizer, self.last_epoch)+</code></pre></details></div>+<div class="api-detail" id="e582"><span class="dtag badge-m">method</span><h2><code>TriStageLR.step_update(num_updates)</code></h2><div class="dmod">funasr.schedulers.tri_stage_scheduler.TriStageLR · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/schedulers/tri_stage_scheduler.py#L96" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Update the learning rate after each update.</p>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def step_update(self, num_updates):+ """Update the learning rate after each update."""+ stage, steps_in_stage = self._decide_stage(num_updates)+ if stage == 0:+ self.lr = self.init_lr + self.warmup_rate * steps_in_stage+ elif stage == 1:+ self.lr = self.peak_lr+ elif stage == 2:+ self.lr = self.peak_lr * math.exp(-self.decay_factor * steps_in_stage)+ elif stage == 3:+ self.lr = self.final_lr+ else:+ raise ValueError("Undefined stage")+ self.set_optimizer_lr(self.lr)+</code></pre></details></div>+<div class="api-detail" id="e583"><span class="dtag badge-m">method</span><h2><code>TriStageLR.set_optimizer_lr(lr)</code></h2><div class="dmod">funasr.schedulers.tri_stage_scheduler.TriStageLR · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/schedulers/tri_stage_scheduler.py#L111" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Set optimizer lr.</p>+<h4>Args:</h4>+<ul>+<li><code>lr</code> — TODO.</li>+</ul>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def set_optimizer_lr(self, lr):+ """Set optimizer lr.++ Args:+ lr: TODO.+ """+ for param_group in self.optimizer.param_groups:+ param_group["lr"] = lr+</code></pre></details></div>+<div class="api-detail" id="e584"><span class="dtag badge-m">method</span><h2><code>TriStageLR.get_lr()</code></h2><div class="dmod">funasr.schedulers.tri_stage_scheduler.TriStageLR · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/schedulers/tri_stage_scheduler.py#L120" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Get lr.</p>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def get_lr(self):+ """Get lr."""+ step_num = self.last_epoch + 1+ self.step_update(step_num)+ return [self.lr]+</code></pre></details></div>+<div class="api-detail" id="e585"><span class="dtag badge-c">class</span><h2>WarmupLR</h2><div class="dmod">funasr.schedulers.warmup_lr · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/schedulers/warmup_lr.py#L11" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>The WarmupLR scheduler</p>+<p>This scheduler is almost same as NoamLR Scheduler except for following difference:</p>+<p>NoamLR:</p>+<p>lr = optimizer.lr * model_size ** -0.5</p>+<p>* min(step ** -0.5, step * warmup_step ** -1.5)</p>+<p>WarmupLR:</p>+<p>lr = optimizer.lr * warmup_step ** 0.5</p>+<p>* min(step ** -0.5, step * warmup_step ** -1.5)</p>+<p>Note that the maximum lr equals to optimizer.lr in this scheduler.</p>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code>class WarmupLR(_LRScheduler, AbsBatchStepScheduler):+ """The WarmupLR scheduler++ This scheduler is almost same as NoamLR Scheduler except for following difference:++ NoamLR:+ lr = optimizer.lr * model_size ** -0.5+ * min(step ** -0.5, step * warmup_step ** -1.5)+ WarmupLR:+ lr = optimizer.lr * warmup_step ** 0.5+ * min(step ** -0.5, step * warmup_step ** -1.5)++ Note that the maximum lr equals to optimizer.lr in this scheduler.++ """++ def __init__(+ self,+ optimizer: torch.optim.Optimizer,+ warmup_steps: Union[int, float] = 25000,+ last_epoch: int = -1,+ ):+ """Initialize WarmupLR.++ Args:+ optimizer: TODO.+ warmup_steps: TODO.+ last_epoch: TODO.+ """+ self.warmup_steps = warmup_steps+</code></pre><a href="https://github.com/modelscope/FunASR/blob/main/funasr/schedulers/warmup_lr.py#L11" target="_blank" class="src-more">View full source on GitHub →</a></details><h3 class="mt">Methods</h3><div class="mblk"><code>.get_lr()</code> <a href="https://github.com/modelscope/FunASR/blob/main/funasr/schedulers/warmup_lr.py#L50" target="_blank" class="gh-link">L50</a><div class="mdoc"><p>Get lr.</p>+</div><details class="src-block"><summary>📄 Source</summary><pre><code> def get_lr(self):+ """Get lr."""+ step_num = self.last_epoch + 1+ return [+ lr * self.warmup_steps**0.5 * min(step_num**-0.5, step_num * self.warmup_steps**-1.5)+ for lr in self.base_lrs+ ]+</code></pre></details></div></div>+<div class="api-detail" id="e586"><span class="dtag badge-m">method</span><h2><code>WarmupLR.get_lr()</code></h2><div class="dmod">funasr.schedulers.warmup_lr.WarmupLR · <a href="https://github.com/modelscope/FunASR/blob/main/funasr/schedulers/warmup_lr.py#L50" target="_blank">View on GitHub ↗</a></div><div class="ddoc"><p>Get lr.</p>+</div><details class="src-block"><summary>📄 Source code</summary><pre><code> def get_lr(self):+ """Get lr."""+ step_num = self.last_epoch + 1+ return [+ lr * self.warmup_steps**0.5 * min(step_num**-0.5, step_num * self.warmup_steps**-1.5)+ for lr in self.base_lrs+ ]+</code></pre></details></div>++</div></div>+<script>+function toggleL1(el){el.closest('.l1').classList.toggle('open')}+function toggleL2(el){el.closest('.l2').classList.toggle('open')}+function showEntry(id){+document.querySelectorAll('.api-detail').forEach(e=>e.classList.remove('active'));+document.querySelectorAll('.l3-item').forEach(e=>e.classList.remove('active'));+document.getElementById('api-welcome').style.display='none';+var t=document.getElementById(id);if(t)t.classList.add('active');+var l=document.querySelector('[data-target="'+id+'"]');if(l)l.classList.add('active');+document.querySelector('.api-content').scrollTop=0;+}+function filterSidebar(q){+q=q.toLowerCase();+document.querySelectorAll('.l3-item').forEach(e=>e.classList.toggle('hidden',q&&!e.textContent.toLowerCase().includes(q)));+if(q)document.querySelectorAll('.l1,.l2').forEach(e=>e.classList.add('open'));+}+if(window.location.hash)showEntry(window.location.hash.slice(1));+</script></body></html>\ No newline at end of filediff --git a/training.html b/training.htmlnew file mode 100644index 000000000..86d44b152--- /dev/null+++ b/training.html@@ -0,0 +1,348 @@+<!DOCTYPE html>+<html lang="en"><head><meta charset="UTF-8"><meta name="viewport" content="width=device-width,initial-scale=1.0">+<title>FunASR Training Guide</title>+<link href="https://fonts.googleapis.com/css2?family=Inter:wght@400;500;600;700;800&family=JetBrains+Mono:wght@400;500&display=swap" rel="stylesheet">+<link rel="stylesheet" href="style.css">+<style>+.content{padding:48px 0 80px}.container.narrow{max-width:820px;margin:0 auto;padding:0 24px}+h1{font-size:2.2rem;font-weight:800;letter-spacing:-0.5px;margin-bottom:8px}+h1+p{color:var(--muted);margin-bottom:32px;font-size:1.05rem}+h2{font-size:1.4rem;font-weight:700;margin:56px 0 16px;padding-bottom:10px;border-bottom:2px solid var(--line)}+h3{font-size:1.1rem;font-weight:600;margin:32px 0 12px}+h4{font-size:0.95rem;font-weight:600;margin:24px 0 8px;color:var(--ink-soft)}+p{margin-bottom:12px;line-height:1.75;font-size:0.92rem}+ul,ol{margin:12px 0 12px 24px}li{margin-bottom:8px;line-height:1.6;font-size:0.9rem}+pre{background:var(--code-bg,#101828);color:#e2e8f0;border-radius:8px;padding:18px 22px;overflow-x:auto;margin:14px 0 22px;font-size:0.82rem;line-height:1.55}+code{font-family:'JetBrains Mono',monospace}+p code,li code,td code{background:#f1f5f9;padding:2px 6px;border-radius:4px;font-size:0.82rem;color:#b91c1c}+.note{background:#eff6ff;border-left:4px solid #2563eb;padding:14px 18px;border-radius:0 8px 8px 0;margin:16px 0;font-size:0.88rem}+.warning{background:#fffbeb;border-left:4px solid #f59e0b;padding:14px 18px;border-radius:0 8px 8px 0;margin:16px 0;font-size:0.88rem}+.tip{background:#f0fdf4;border-left:4px solid #059669;padding:14px 18px;border-radius:0 8px 8px 0;margin:16px 0;font-size:0.88rem}+table{width:100%;border-collapse:collapse;margin:16px 0;font-size:0.88rem}+th,td{padding:10px 14px;text-align:left;border-bottom:1px solid var(--line)}+th{background:#f8fafc;font-weight:600;font-size:0.78rem;text-transform:uppercase;letter-spacing:0.3px;color:var(--muted)}+.toc-grid{display:grid;grid-template-columns:repeat(auto-fit,minmax(180px,1fr));gap:8px;margin:20px 0 40px}+.toc-grid a{display:block;padding:10px 14px;background:var(--page,#f7f9fc);border:1px solid var(--line);border-radius:8px;text-decoration:none;color:var(--ink-soft);font-size:0.82rem;font-weight:500;transition:all 0.15s}+.toc-grid a:hover{border-color:#2563eb;color:#2563eb}+.output-box{background:#f8fafc;border:1px solid var(--line);border-radius:8px;padding:14px 18px;margin:12px 0;font-family:'JetBrains Mono',monospace;font-size:0.76rem;line-height:1.6;overflow-x:auto}+</style>+</head><body>+<nav class="nav"><div class="container">+<a href="index.html" class="nav-logo">FunASR</a>+<div class="nav-links">+<a href="index.html">Home</a>+<a href="tutorial.html">Tutorial</a>+<a href="training.html" class="active">Training</a>+<a href="model-registration.html">Develop</a>+<a href="api.html">API</a>+</div>+<a href="https://github.com/modelscope/FunASR" class="nav-github">GitHub</a>+</div></nav>++<div class="content"><div class="container narrow">+<h1>Training & Fine-tuning</h1>+<p>Fine-tune pretrained models on your own data using FunASR's training framework.</p>++<div class="toc-grid">+<a href="#overview">Overview</a>+<a href="#data">Data Preparation</a>+<a href="#paraformer">Fine-tune Paraformer</a>+<a href="#sensevoice">Fine-tune SenseVoice</a>+<a href="#nano">Fine-tune Fun-ASR-Nano</a>+<a href="#params">Parameter Reference</a>+<a href="#multi-gpu">Multi-GPU Training</a>+<a href="#deepspeed">DeepSpeed</a>+<a href="#monitor">Monitoring</a>+<a href="#inference-after">Use Fine-tuned Model</a>+<a href="#tips">Tips & Troubleshooting</a>+</div>++<!-- Overview -->+<h2 id="overview">Overview</h2>+<p>FunASR's training framework supports:</p>+<ul>+<li><strong>Fine-tuning</strong> any pretrained model on custom domain data</li>+<li><strong>Multi-GPU</strong> training with PyTorch DDP (single/multi-node)</li>+<li><strong>DeepSpeed</strong> ZeRO Stage 1/2/3 for large model training</li>+<li><strong>Dynamic batching</strong> by token count or example count</li>+<li><strong>Checkpoint averaging</strong> for best performance</li>+<li><strong>Resume training</strong> from interruption</li>+</ul>++<p>The training entry point is <code>funasr-train-ds</code> (or <code>funasr/bin/train_ds.py</code>), launched via <code>torchrun</code> for distributed training.</p>++<!-- Data -->+<h2 id="data">Data Preparation</h2>++<h3>Standard Format (Paraformer, SenseVoice)</h3>+<p>Training data uses JSONL format — one JSON object per line:</p>+<pre>{"key": "utt001", "source": "/path/to/audio.wav", "source_len": 90, "target": "这是转写文本", "target_len": 6}+{"key": "utt002", "source": "/path/to/audio2.wav", "source_len": 150, "target": "hello world", "target_len": 2}</pre>++<table>+<tr><th>Field</th><th>Type</th><th>Description</th></tr>+<tr><td><code>key</code></td><td>str</td><td>Unique utterance ID</td></tr>+<tr><td><code>source</code></td><td>str</td><td>Audio file path (local path or URL)</td></tr>+<tr><td><code>source_len</code></td><td>int</td><td>Audio length in fbank frames (1 frame = 10ms)</td></tr>+<tr><td><code>target</code></td><td>str</td><td>Transcription text</td></tr>+<tr><td><code>target_len</code></td><td>int</td><td>Number of text tokens</td></tr>+</table>++<h4>Generate from wav.scp + text.txt</h4>+<p>If you have Kaldi-style data files, convert them:</p>+<pre># train_wav.scp (tab-separated: id \t path)+utt001 /data/audio/001.wav+utt002 /data/audio/002.wav++# train_text.txt (tab-separated: id \t text)+utt001 这是转写文本+utt002 hello world</pre>++<pre># Convert to jsonl+scp2jsonl \+ ++scp_file_list='["/data/list/train_wav.scp", "/data/list/train_text.txt"]' \+ ++data_type_list='["source", "target"]' \+ ++jsonl_file_out="/data/list/train.jsonl"</pre>++<h3>ChatML Format (Fun-ASR-Nano)</h3>+<p>Fun-ASR-Nano uses ChatML conversation format:</p>+<pre>{"messages": [+ {"role": "system", "content": "You are a helpful assistant."},+ {"role": "user", "content": "语音转写:<|startofspeech|>!/path/to/audio.wav<|endofspeech|>"},+ {"role": "assistant", "content": "几点了?"}+], "speech_length": 145, "text_length": 3}</pre>++<table>+<tr><th>Field</th><th>Description</th></tr>+<tr><td><code>messages[0]</code></td><td>System prompt (fixed: "You are a helpful assistant.")</td></tr>+<tr><td><code>messages[1]</code></td><td>User: prompt + audio path wrapped in <code><|startofspeech|>!...<|endofspeech|></code></td></tr>+<tr><td><code>messages[2]</code></td><td>Assistant: transcription text</td></tr>+<tr><td><code>speech_length</code></td><td>Number of fbank frames (10ms each)</td></tr>+<tr><td><code>text_length</code></td><td>Number of tokens (tokenized by Qwen3-0.6B)</td></tr>+</table>++<div class="tip"><strong>Prompt variations:</strong><br>+• Chinese: <code>语音转写:</code><br>+• English: <code>Speech transcription:</code><br>+• Cross-language: <code>语音转写成英文:</code><br>+• No ITN: <code>语音转写,不进行文本规整:</code></div>++<p>Convert from wav.scp + text.txt:</p>+<pre>python tools/scp2jsonl.py \+ ++scp_file=data/train_wav.scp \+ ++transcript_file=data/train_text.txt \+ ++jsonl_file=data/train_example.jsonl</pre>++<!-- Paraformer -->+<h2 id="paraformer">Fine-tune Paraformer</h2>+<pre>cd examples/industrial_data_pretraining/paraformer+bash finetune.sh</pre>++<p>Or customize the key parameters:</p>+<pre>export CUDA_VISIBLE_DEVICES="0,1"+gpu_num=2++torchrun --nproc_per_node $gpu_num \+ funasr/bin/train_ds.py \+ ++model="iic/speech_paraformer-large_asr_nat-zh-cn-16k-common-vocab8404-pytorch" \+ ++train_data_set_list="data/train.jsonl" \+ ++valid_data_set_list="data/val.jsonl" \+ ++dataset_conf.batch_size=6000 \+ ++dataset_conf.batch_type="token" \+ ++dataset_conf.num_workers=4 \+ ++train_conf.max_epoch=50 \+ ++train_conf.validate_interval=2000 \+ ++train_conf.save_checkpoint_interval=2000 \+ ++train_conf.keep_nbest_models=20 \+ ++train_conf.avg_nbest_model=10 \+ ++optim_conf.lr=0.0002 \+ ++output_dir="./outputs"</pre>++<!-- SenseVoice -->+<h2 id="sensevoice">Fine-tune SenseVoice</h2>+<pre>cd examples/industrial_data_pretraining/sense_voice+bash finetune.sh</pre>++<p>Same data format as Paraformer (source/target JSONL). Key difference: SenseVoice uses its own dataset class internally.</p>++<!-- Fun-ASR-Nano -->+<h2 id="nano">Fine-tune Fun-ASR-Nano</h2>+<pre>cd examples/industrial_data_pretraining/fun_asr_nano+bash finetune.sh</pre>++<p>Key differences from Paraformer:</p>+<ul>+<li>Uses ChatML data format (see above)</li>+<li><code>++trust_remote_code=true</code> required</li>+<li>Supports <strong>selective freezing</strong>: freeze encoder/adaptor while training LLM decoder</li>+</ul>++<pre># Freeze encoder + adaptor, only train LLM (recommended for domain adaptation)+++audio_encoder_conf.freeze=true+++audio_adaptor_conf.freeze=true+++llm_conf.freeze=false++# Full fine-tune (all parameters)+++audio_encoder_conf.freeze=false+++audio_adaptor_conf.freeze=false+++llm_conf.freeze=false</pre>++<div class="note"><strong>Recommended strategy:</strong> Start with LLM-only fine-tuning (faster, less data needed). If results are insufficient, unfreeze adaptor. Only unfreeze encoder with very large datasets (>1000h).</div>++<!-- Parameters -->+<h2 id="params">Parameter Reference</h2>++<h3>Dataset Parameters</h3>+<table>+<tr><th>Parameter</th><th>Default</th><th>Description</th></tr>+<tr><td><code>dataset_conf.batch_type</code></td><td>"token"</td><td><code>"token"</code>: dynamic batch by total tokens. <code>"example"</code>: fixed batch count.</td></tr>+<tr><td><code>dataset_conf.batch_size</code></td><td>6000</td><td>Token mode: total frames per batch. Example mode: number of samples.</td></tr>+<tr><td><code>dataset_conf.sort_size</code></td><td>1024</td><td>Buffer size for length-based sorting (improves padding efficiency).</td></tr>+<tr><td><code>dataset_conf.num_workers</code></td><td>4</td><td>Data loading threads.</td></tr>+<tr><td><code>dataset_conf.data_split_num</code></td><td>1</td><td>Split data into N groups for large-scale training (reduces memory).</td></tr>+<tr><td><code>dataset_conf.max_token_length</code></td><td>—</td><td>Filter: skip samples longer than this (in frames/tokens).</td></tr>+<tr><td><code>dataset_conf.min_token_length</code></td><td>—</td><td>Filter: skip samples shorter than this.</td></tr>+</table>++<h3>Training Parameters</h3>+<table>+<tr><th>Parameter</th><th>Default</th><th>Description</th></tr>+<tr><td><code>train_conf.max_epoch</code></td><td>50</td><td>Total training epochs.</td></tr>+<tr><td><code>train_conf.log_interval</code></td><td>1</td><td>Print loss every N steps.</td></tr>+<tr><td><code>train_conf.validate_interval</code></td><td>2000</td><td>Run validation every N steps.</td></tr>+<tr><td><code>train_conf.save_checkpoint_interval</code></td><td>2000</td><td>Save model every N steps.</td></tr>+<tr><td><code>train_conf.keep_nbest_models</code></td><td>20</td><td>Keep top N models (by validation accuracy).</td></tr>+<tr><td><code>train_conf.avg_nbest_model</code></td><td>10</td><td>Average top N models for final checkpoint.</td></tr>+<tr><td><code>train_conf.resume</code></td><td>true</td><td>Resume from last checkpoint if exists.</td></tr>+<tr><td><code>train_conf.use_deepspeed</code></td><td>false</td><td>Enable DeepSpeed ZeRO optimization.</td></tr>+<tr><td><code>optim_conf.lr</code></td><td>0.0002</td><td>Learning rate.</td></tr>+</table>++<!-- Multi-GPU -->+<h2 id="multi-gpu">Multi-GPU Training</h2>++<h3>Single Machine, Multiple GPUs</h3>+<pre>export CUDA_VISIBLE_DEVICES="0,1,2,3"+gpu_num=4++torchrun --nnodes 1 --nproc_per_node $gpu_num \+ funasr/bin/train_ds.py ${train_args}</pre>++<h3>Multiple Machines</h3>+<pre># Machine 1 (master, IP=192.168.1.1)+torchrun --nnodes 2 --node_rank 0 --nproc_per_node 4 \+ --master_addr=192.168.1.1 --master_port=12345 \+ funasr/bin/train_ds.py ${train_args}++# Machine 2+torchrun --nnodes 2 --node_rank 1 --nproc_per_node 4 \+ --master_addr=192.168.1.1 --master_port=12345 \+ funasr/bin/train_ds.py ${train_args}</pre>++<!-- DeepSpeed -->+<h2 id="deepspeed">DeepSpeed</h2>+<p>For large models (Fun-ASR-Nano 800M params), enable DeepSpeed ZeRO:</p>+<pre>++train_conf.use_deepspeed=true+++train_conf.deepspeed_config=./deepspeed_conf/ds_stage1.json</pre>++<p>Stage 1 config (recommended starting point):</p>+<pre>{+ "train_micro_batch_size_per_gpu": 1,+ "gradient_accumulation_steps": 1,+ "bf16": {"enabled": true},+ "zero_optimization": {+ "stage": 1,+ "reduce_bucket_size": 5e8,+ "allgather_bucket_size": 5e8+ }+}</pre>++<div class="note"><strong>When to use which stage:</strong><br>+• Stage 1: Optimizer state partitioned. Good for most cases.<br>+• Stage 2: + Gradient partitioned. For larger models.<br>+• Stage 3: + Parameter partitioned. Maximum memory savings but slower.</div>++<!-- Monitor -->+<h2 id="monitor">Monitoring Training</h2>++<h4>Log file</h4>+<pre>tail -f outputs/log.txt++# Example output:+# train, rank: 0, epoch: 0/50, step: 6990, (loss_avg_rank: 0.327),+# (acc_avg_epoch: 0.795), (lr: 1.165e-04),+# GPU memory: usage: 3.8GB, peak: 18.3GB</pre>++<p>Key metrics to watch:</p>+<ul>+<li><code>loss_avg_epoch</code>: should decrease over time</li>+<li><code>acc_avg_epoch</code>: should increase (most important metric)</li>+<li><code>lr</code>: learning rate at current step</li>+<li><code>GPU memory</code>: peak should not exceed your GPU VRAM</li>+</ul>++<h4>TensorBoard</h4>+<pre>tensorboard --logdir outputs/log/tensorboard+# Open http://localhost:6006</pre>++<!-- Use Fine-tuned Model -->+<h2 id="inference-after">Use Your Fine-tuned Model</h2>++<h4>If outputs/ has configuration.json</h4>+<pre>from funasr import AutoModel+model = AutoModel(model="./outputs")+res = model.generate(input="test.wav")+print(res[0]["text"])</pre>++<h4>If no configuration.json</h4>+<pre>funasr ++model="./outputs" \+ ++config-path="./outputs" \+ ++config-name="config.yaml" \+ ++init_param="./outputs/model.pt" \+ ++input="test.wav"</pre>++<!-- Tips -->+<h2 id="tips">Tips & Troubleshooting</h2>++<h4>OOM during training</h4>+<ol>+<li>Reduce <code>dataset_conf.batch_size</code></li>+<li>Add <code>dataset_conf.max_token_length=2000</code> to filter long utterances</li>+<li>Enable DeepSpeed (partitions optimizer states)</li>+<li>Reduce <code>dataset_conf.num_workers</code></li>+</ol>++<h4>Training loss stuck / NaN gradients</h4>+<ul>+<li>Reduce learning rate (try 0.00005)</li>+<li>Check data quality — corrupted audio files cause NaN</li>+<li>For Fun-ASR-Nano: start with encoder frozen</li>+</ul>++<h4>Validation accuracy not improving</h4>+<ul>+<li>Increase training data (min ~10h for fine-tuning)</li>+<li>Check domain match — model may not generalize to very different domains</li>+<li>Try unfreezing more layers gradually</li>+</ul>++<h4>Large-scale data (>10,000 hours)</h4>+<p>Use data splitting to avoid memory issues:</p>+<pre># Split data into chunks, load 2 at a time+++dataset_conf.data_split_num=256+# data.list contains paths to split jsonl files:+# data/train.0.jsonl+# data/train.1.jsonl+# ...+++train_data_set_list="data/data.list"</pre>++<h4>Resume after crash</h4>+<p>Set <code>++train_conf.resume=true</code> (default). Training automatically restarts from the latest checkpoint in <code>output_dir</code>.</p>++</div></div>++<footer>+<p>FunASR · Tongyi Lab, Alibaba Group</p>+<p><a href="index.html">Home</a> · <a href="tutorial.html">Tutorial</a> · <a href="model-registration.html">Develop</a> · <a href="api.html">API</a> · <a href="https://github.com/modelscope/FunASR">GitHub</a></p>+</footer>+</body></html>