Free, open-source screen sharing for private live streams with friends. Watch together in a browser or self-host Piik. 免费开源的私密屏幕共享,支持游戏直播、一起看电影与自部署。
feat(native): add window-scoped process audio (#93)
AAA6cd99545e662
diff --git a/docs/adr/0008-window-scoped-audio-capture.md b/docs/adr/0008-window-scoped-audio-capture.mdindex 51396f39..229da1b6 100644--- a/docs/adr/0008-window-scoped-audio-capture.md+++ b/docs/adr/0008-window-scoped-audio-capture.md@@ -2,8 +2,8 @@Date: 2026-08-21-Status: Accepted for the browser hint; native Windows implementation is a-planned P1 input.+Status: Accepted. The browser hint and opt-in Native Windows P1 slice are+source-complete; Native remains unpackaged, default-off, and undeployed.## Context@@ -35,8 +35,9 @@ Windows 11 desktop (build 22000 or newer) and is not a Windows 10 fallback.3. The native sender's next audio slice is an explicit opt-in`window-process-audio` mode on Windows 11. The local capture session ownsone selected target PID/process tree, one WGC/DXGI video source, and one- WASAPI loopback audio source. Process IDs, titles, paths, device identity,- and PCM stay local; they do not enter signaling, URLs, logs, or storage.+ WASAPI loopback audio source. Process IDs, creation tokens, titles, paths,+ device identity, and PCM stay local; they do not enter signaling, URLs,+ logs, or storage.4. If the target process exits, produces no render stream, or the loopbackactivation is denied, the sender reports `audio unavailable` or `audiosilent` and stops/asks the user. It never widens to whole-system audio.@@ -48,7 +49,7 @@ Windows 11 desktop (build 22000 or newer) and is not a Windows 10 fallback.This is an interface boundary, not a new framework or a current wire change:```text-NativeCaptureTarget { pid: uint32, includeProcessTree: true }+NativeCaptureTarget { pid: uint32, creationTime: uint64, includeProcessTree: true }NativeCaptureSession.start(target)-> VideoFrames(WGC/DXGI) + AudioPcm(WASAPI process loopback)-> existing native sender timeline/encoder/fanout@@ -66,7 +67,8 @@ The first product-wiring slice is deliberately smaller than the completenative capture boundary. It adds an explicit Windows 11`window-process-audio` input while leaving the current Web video capture andcodec path unchanged. A small Windows helper resolves the selected target PID-locally and sends timestamped PCM through the authenticated loopback bridge.+and creation time locally, revalidates both immediately before activation, and+sends timestamped PCM through the authenticated loopback bridge.The existing Chrome/Edge sender UI uses `AudioEncoder` with `codec: "opus"`,48 kHz, two channels, and 20 ms frames, then returns the encoded Opus packets tothe existing Go sender. This reuses the current browser prerequisite and avoids@@ -84,8 +86,8 @@ QPC origin when native video replaces the Web source.Expected implementation size is about **500--800 new LOC**, excluding thebrowser/Windows SDK and existing video/MF fixture code: Windows loopbackcapture and local IPC (250--400), Go audio envelope/timeline/track (150--250),-and UI lifecycle/status plus focused tests (100--150). This is a planned P1-input, not current product behavior or a deployment switch.+and UI lifecycle/status plus focused tests (100--150). The landed slice keeps+that boundary and is not a deployment switch.The two source clocks are both converted from 100-ns QPC: WGC`SystemRelativeTime` and WASAPI `IAudioCaptureClient::GetBuffer`'s@@ -101,7 +103,8 @@ or `AUDCLNT_E_SERVICE_NOT_RUNNING`, protected-content silence, and a missingrender stream produce `audio unavailable` or `audio silent`. There is no retryto whole-system loopback. No administrator privilege is required, but thehelper must run in the interactive user's session and honor Windows consent and-privacy settings. PID, title, path, device identity, and PCM remain local.+privacy settings. PID, creation token, title, path, device identity, and PCM+remain local.The first smoke is one direct Viewer: a test window emits a known tone andvisual marker, while an independent voice process and notification emit@@ -109,7 +112,9 @@ different markers. Require an Opus inbound track and rendered video/audio,monotonic QPC timestamps, and a bounded A/V offset; verify the unrelatedmarkers are absent and that target exit/silence stops or asks without wideningcapture. SFU/TURN, second Viewer, endurance, and the native WGC video swap are-follow-up gates.+follow-up gates. On 2026-08-21, 100 20-ms chunks isolated a 440 Hz target from+an independent 880 Hz process by 4017.8x. Chrome 151 then received one audio+track and 495 inbound Opus packets while video decoded/rendered 296 frames.## Consequences@@ -122,10 +127,7 @@ follow-up gates.## Follow-Up TODO-- Add the Windows-only WGC + WASAPI process-loopback fixture beside the existing- Media Foundation H.264 fixture, then land the bounded audio-only P1 slice.-- Feed its timestamped PCM/video pair into the existing native sender only- after the isolation and sync checks pass; native WGC video replacement stays- a separate follow-up.-- Retain a short matrix result in `docs/research/`; do not enable the mode by- default or deploy it from this ADR alone.+- Package the helper beside the sender and keep explicit target selection+ default-off.+- Keep native WGC video, a second Viewer, SFU/TURN, endurance, and real-game+ A/V sync as separate follow-ups.diff --git a/docs/project-memory.md b/docs/project-memory.mdindex a06e0c70..4cffff93 100644--- a/docs/project-memory.md+++ b/docs/project-memory.md@@ -30,7 +30,7 @@ Last updated: 2026-08-21- C+B quality reparenting is edge-local and cooldown-bound; no score, timer, or global parent penalty.- Flagship media is UDP; HTTPS/WSS stays TLS/TCP; the old release remains rollback-only.- Treat settings as ceilings and degradation as unclassified. Use correlated Host A+B/Viewer C and one-variable evidence; never force AV1, infer by UA, or create a composite score.-- System capture can cause film-audio loss/self-echo; Web cannot isolate processes. Diagnose A/B/C and sync; Windows 11 native game-tree audio is a candidate, Windows 10 unresolved. See `docs/research/browser-screen-audio-quality.md`.+- Web capture cannot isolate process audio. ADR-0008 Windows 11 native audio is opt-in/source-only: opaque local selection, no system-mix fallback or exposed PID, and one shared Opus track; one direct Viewer received 495 packets. Packaging, game A/V sync, other routes, and Windows 10 remain open.- Viewer-local names and the opt-in Web participant roster remain control-plane-only. Web Host names are deployed in the current release and stay socket/localStorage-only; Native remains outside the capability boundary.- Do not add scene detection, dynamic-FPS control, or forced AV1 without negotiation, encode, game, CPU/GPU, and sender evidence.- ADR-0002 grant/public access, the v3 room-password migration, and Web Host display names are deployed. Names/presence remain session-only without account or roster tables.diff --git a/docs/research/browser-screen-audio-quality.md b/docs/research/browser-screen-audio-quality.mdindex 93452576..ec1ae1f3 100644--- a/docs/research/browser-screen-audio-quality.md+++ b/docs/research/browser-screen-audio-quality.md@@ -208,6 +208,24 @@ known target tone/visual marker and independent voice/notification markers;SFU/TURN, a second Viewer, endurance, and native WGC video replacement remainlater gates.+### Retained P1 Result (2026-08-21)++The source-complete Windows helper emits process-tree-only 48 kHz stereo s16 in+20-ms chunks. Chrome 151 encodes the chunks once with `AudioEncoder`/Opus, and+Go packetizes the same encoded stream once into a shared Pion track. The UI+receives an opaque memory-only target ID. PID and the enumeration-time process+creation token are revalidated locally and never enter public signaling, a URL,+storage, or diagnostics.++One bounded run played 440 Hz in the selected target and 880 Hz in an+independent process. Across 100 chunks, amplitudes were 0.240225 and 0.0000598+(4017.8x); no system-mix fallback was used. One direct Viewer then received one+audio track and 495 inbound Opus packets while video decoded/rendered 296+frames. There was no fatal or encoder error. The sole failed assertion was the+existing generic Pion two-second outbound snapshot, which also missed fresh+deltas in earlier video loopbacks. This is functional evidence, not packaging,+real-game sync, second-Viewer, SFU/TURN, or endurance evidence.+Run one bounded matrix rather than a full route Cartesian product: exactproduction and current `main` on Windows Chrome/Edge for tab/window/monitor,audio selected/unselected, and a simultaneous voice call; then the nativediff --git a/docs/status.md b/docs/status.mdindex 278c9f6f..1a9fc719 100644--- a/docs/status.md+++ b/docs/status.md@@ -42,6 +42,7 @@ Flagship; parallel; min run/smoke/rollback; benchmark later. Active UI/config/lo- Host A+B and authenticated P2P Viewer C are sanitized, generation-bound, read-only, and fail closed for stale, ambiguous, or SFU-fed evidence; no raw media metadata is retained.- The `a11a73d` built-in TURN canary is rejected: local allocation passed, but direct Host/Pion Viewer failed before forced relay. Full rollback restored STUN-only client ICE and zero allocations.- Chrome 151 loopbacks decoded VP8 299 and H.264 opt-in 299/298 rendered at 1280x720 without fatal/encoder errors. Hardware attribution, Pion timing, multi-viewer/endurance/public/native proof remain open; production stays VP8.+- Native Win11 audio is source-only/default-off. Smoke isolated target 4018x; one Viewer got 495 Opus packets. Packaging, game sync, Win10, and other routes remain open.- Access protocol/config/HTTP/storage/SQLite/signaling focused tests pass, including commit-first teardown and v1 rollback.- Source adds selected Host ingress and peer last-mile for every enabled room; focused routing/config tests pass with Peer ICE STUN-only.- `61a87ae` Host names, `d1a4fd4` window-audio hint, and `cf149df` all-room routing are active; the stale restart assertion is historical only.@@ -52,7 +53,6 @@ Flagship; parallel; min run/smoke/rollback; benchmark later. Active UI/config/lo- The final deployment proves process/config health, not a retained SFU frame, retry/failback, selected UDP pair, admission rescue, edge cap, quality, or resource deltas; game-share load/blur remains unclassified.- Android Chrome/iOS Safari Viewer leaves remain unverified and conservatively leaf-only. Mobile Web Host is unsupported; native senders are planned only.- Silent partitions can wait 30 to 60 seconds for heartbeat detection before the default 5-second grace; this remains unverified.-- Real audio and heterogeneous clients remain unverified. System capture can lose film audio or echo voice calls; Web has no process isolation. Diagnose A/B/C and sync; gate Windows 11 game audio; Windows 10 unresolved.- The former release reportedly sustained bandwidth-limited blur on a capable LAN; Host refresh recovered while Viewer refresh did not. The new deployment has not yet reproduced or cleared it.- ADR-0006 has one Viewer proof; two-edge/FIFO, physical hardware, endurance, public, packaged-native, and browser-diversity proof remain open.- Browser fanout is host two/viewer one; any accepted endpoint relay stays capped at two downstream edges.@@ -74,7 +74,7 @@ evacuation. Native source accepts increasing capture timestamps shorter thanduration; VP8 and explicit H.264 Viewer decode/render are recorded. Hardwarepreference is not physical proof. Next prove fresh Pionoutbound diagnostics before viewer 2/FIFO/hardware/endurance/deploy.-Audio A/B/C may proceed without displacing P0.+Package Native audio and test one real-game A/V pair later without displacing P0.Ordinary Peer ICE stays STUN-only; participant-wide TURN is removed. Theselected-edge tuple is configured in production, but Host ingress still needs oneforced-relay canary before any media-success claim. Retained performance/resourcediff --git a/native/process-audio-helper/README.md b/native/process-audio-helper/README.mdnew file mode 100644index 00000000..d843bced--- /dev/null+++ b/native/process-audio-helper/README.md@@ -0,0 +1,20 @@+# Windows Process-Audio Helper++This Windows 11 helper is the local input for Native sender process-tree audio.+It enumerates visible top-level windows locally and captures only the explicitly+selected process plus its children through WASAPI application loopback. It has+no whole-system fallback and no network code.++Build it outside the repository, then place both executables in the same output+directory (or set `SCREENER_PROCESS_AUDIO_HELPER` for a development run):++```powershell+$out = Join-Path ([IO.Path]::GetTempPath()) 'screener-native-audio'+./native/process-audio-helper/build.ps1 -OutputDirectory $out+```++The implementation follows Microsoft's MIT-licensed Application Loopback+sample contract without copying its WIL/Media Foundation fixture machinery:++- <https://github.com/microsoft/Windows-classic-samples/tree/main/Samples/ApplicationLoopback>+- <https://learn.microsoft.com/en-us/windows/win32/api/audioclientactivationparams/ns-audioclientactivationparams-audioclient_activation_params>diff --git a/native/process-audio-helper/build.ps1 b/native/process-audio-helper/build.ps1new file mode 100644index 00000000..59e14cac--- /dev/null+++ b/native/process-audio-helper/build.ps1@@ -0,0 +1,40 @@+[CmdletBinding()]+param(+ [Parameter(Mandatory = $true)]+ [string]$OutputDirectory+)++Set-StrictMode -Version Latest+$ErrorActionPreference = 'Stop'++$helperDirectory = Split-Path -Parent $MyInvocation.MyCommand.Path+$repositoryRoot = [IO.Path]::GetFullPath((Join-Path $helperDirectory '..\..'))+$outputPath = [IO.Path]::GetFullPath($OutputDirectory)+$repositoryPrefix = $repositoryRoot.TrimEnd([IO.Path]::DirectorySeparatorChar) + [IO.Path]::DirectorySeparatorChar+if ($outputPath.Equals($repositoryRoot, [StringComparison]::OrdinalIgnoreCase) -or+ $outputPath.StartsWith($repositoryPrefix, [StringComparison]::OrdinalIgnoreCase)) {+ throw 'OutputDirectory must be outside the repository.'+}++$vswhere = Join-Path ${env:ProgramFiles(x86)} 'Microsoft Visual Studio\Installer\vswhere.exe'+$installationPath = & $vswhere -latest -products * -requires Microsoft.VisualStudio.Component.VC.Tools.x86.x64 -property installationPath+if ([string]::IsNullOrWhiteSpace($installationPath)) {+ throw 'Visual Studio C++ Build Tools were not found.'+}++$developerCommand = Join-Path $installationPath 'Common7\Tools\VsDevCmd.bat'+$sourcePath = Join-Path $helperDirectory 'main.cpp'+$executablePath = Join-Path $outputPath 'screener-process-audio.exe'+$objectPath = Join-Path $outputPath 'main.obj'+New-Item -ItemType Directory -Path $outputPath -Force | Out-Null++$compile = @(+ 'call "{0}" -arch=x64 -host_arch=x64 >nul' -f $developerCommand+ 'cl.exe /nologo /std:c++20 /EHsc /W4 /WX /DUNICODE /D_UNICODE /DWIN32_LEAN_AND_MEAN /D_WIN32_WINNT=0x0A00 /DNTDDI_VERSION=0x0A00000A "{0}" /Fo:"{1}" /Fe:"{2}" /link ole32.lib mmdevapi.lib runtimeobject.lib user32.lib' -f $sourcePath, $objectPath, $executablePath+) -join ' && '++& cmd.exe /d /s /c $compile+if ($LASTEXITCODE -ne 0) {+ throw "Process-audio helper compilation failed with exit code $LASTEXITCODE."+}+Write-Output $executablePathdiff --git a/native/process-audio-helper/main.cpp b/native/process-audio-helper/main.cppnew file mode 100644index 00000000..95c5b6e7--- /dev/null+++ b/native/process-audio-helper/main.cpp@@ -0,0 +1,334 @@+#define NOMINMAX+#include <windows.h>++#include <audioclient.h>+#include <audioclientactivationparams.h>+#include <fcntl.h>+#include <io.h>+#include <mmdeviceapi.h>+#include <wrl.h>+#include <wrl/implements.h>++#include <array>+#include <cstdint>+#include <cstdlib>+#include <iostream>+#include <string>+#include <vector>++using Microsoft::WRL::ComPtr;+using Microsoft::WRL::FtmBase;+using Microsoft::WRL::Make;+using Microsoft::WRL::RuntimeClass;+using Microsoft::WRL::RuntimeClassFlags;+using Microsoft::WRL::ClassicCom;++namespace {+constexpr UINT32 kSampleRate = 48'000;+constexpr UINT16 kChannels = 2;+constexpr UINT16 kBytesPerSample = 2;+constexpr UINT32 kFramesPerChunk = 960;+constexpr DWORD kBytesPerChunk = kFramesPerChunk * kChannels * kBytesPerSample;+constexpr UINT64 kChunkDuration100ns = 200'000;+constexpr size_t kMaxWindows = 100;++class ActivationHandler final : public RuntimeClass<+ RuntimeClassFlags<ClassicCom>, FtmBase, IActivateAudioInterfaceCompletionHandler> {+ public:+ explicit ActivationHandler(HANDLE completed) : completed_(completed) {}++ STDMETHODIMP ActivateCompleted(IActivateAudioInterfaceAsyncOperation* operation) override {+ ComPtr<IUnknown> activated;+ HRESULT activationResult = E_UNEXPECTED;+ result_ = operation->GetActivateResult(&activationResult, &activated);+ if (SUCCEEDED(result_)) result_ = activationResult;+ if (SUCCEEDED(result_)) result_ = activated.As(&client_);+ SetEvent(completed_);+ return S_OK;+ }++ HRESULT Result() const { return result_; }+ ComPtr<IAudioClient> Client() const { return client_; }++ private:+ HANDLE completed_ = nullptr;+ HRESULT result_ = E_UNEXPECTED;+ ComPtr<IAudioClient> client_;+};++struct WindowTarget {+ DWORD pid;+ UINT64 creationTime;+ std::string title;+};++HRESULT ReadProcessCreationTime(HANDLE process, UINT64* creationTime) {+ FILETIME created{}, exited{}, kernel{}, user{};+ if (!GetProcessTimes(process, &created, &exited, &kernel, &user)) {+ return HRESULT_FROM_WIN32(GetLastError());+ }+ ULARGE_INTEGER value{};+ value.LowPart = created.dwLowDateTime;+ value.HighPart = created.dwHighDateTime;+ if (value.QuadPart == 0) return HRESULT_FROM_WIN32(ERROR_INVALID_DATA);+ *creationTime = value.QuadPart;+ return S_OK;+}++std::string Utf8(const std::wstring& value) {+ if (value.empty()) return {};+ const int size = WideCharToMultiByte(CP_UTF8, WC_ERR_INVALID_CHARS, value.data(),+ static_cast<int>(value.size()), nullptr, 0, nullptr, nullptr);+ if (size <= 0) return {};+ std::string result(static_cast<size_t>(size), '\0');+ if (WideCharToMultiByte(CP_UTF8, WC_ERR_INVALID_CHARS, value.data(),+ static_cast<int>(value.size()), result.data(), size, nullptr, nullptr) != size) {+ return {};+ }+ return result;+}++std::string JsonString(const std::string& value) {+ static constexpr char hex[] = "0123456789abcdef";+ std::string result = "\"";+ for (const unsigned char character : value) {+ switch (character) {+ case '\\': result += "\\\\"; break;+ case '"': result += "\\\""; break;+ case '\b': result += "\\b"; break;+ case '\f': result += "\\f"; break;+ case '\n': result += "\\n"; break;+ case '\r': result += "\\r"; break;+ case '\t': result += "\\t"; break;+ default:+ if (character < 0x20) {+ result += "\\u00";+ result += hex[character >> 4];+ result += hex[character & 0xf];+ } else {+ result.push_back(static_cast<char>(character));+ }+ }+ }+ result += '"';+ return result;+}++BOOL CALLBACK CollectWindow(HWND window, LPARAM parameter) {+ auto* targets = reinterpret_cast<std::vector<WindowTarget>*>(parameter);+ if (targets->size() >= kMaxWindows) return FALSE;+ if (!IsWindowVisible(window) || GetWindow(window, GW_OWNER) != nullptr) return TRUE;+ const int length = GetWindowTextLengthW(window);+ if (length <= 0 || length > 512) return TRUE;+ DWORD pid = 0;+ GetWindowThreadProcessId(window, &pid);+ if (pid == 0 || pid == GetCurrentProcessId()) return TRUE;+ std::wstring title(static_cast<size_t>(length) + 1, L'\0');+ const int copied = GetWindowTextW(window, title.data(), length + 1);+ if (copied <= 0) return TRUE;+ title.resize(static_cast<size_t>(copied));+ std::string utf8 = Utf8(title);+ if (utf8.empty()) return TRUE;+ HANDLE process = OpenProcess(PROCESS_QUERY_LIMITED_INFORMATION, FALSE, pid);+ if (process == nullptr) return TRUE;+ UINT64 creationTime = 0;+ const HRESULT result = ReadProcessCreationTime(process, &creationTime);+ CloseHandle(process);+ if (SUCCEEDED(result)) targets->push_back({pid, creationTime, std::move(utf8)});+ return TRUE;+}++int ListWindows() {+ std::vector<WindowTarget> targets;+ if (!EnumWindows(CollectWindow, reinterpret_cast<LPARAM>(&targets))) return 2;+ std::cout << '[';+ for (size_t index = 0; index < targets.size(); ++index) {+ if (index != 0) std::cout << ',';+ std::cout << "{\"pid\":" << targets[index].pid << ",\"creationTime\":"+ << targets[index].creationTime << ",\"title\":"+ << JsonString(targets[index].title) << '}';+ }+ std::cout << ']';+ return std::cout.good() ? 0 : 2;+}++void PutUint32BE(BYTE* output, UINT32 value) {+ for (int index = 3; index >= 0; --index) {+ output[index] = static_cast<BYTE>(value & 0xff);+ value >>= 8;+ }+}++void PutUint64BE(BYTE* output, UINT64 value) {+ for (int index = 7; index >= 0; --index) {+ output[index] = static_cast<BYTE>(value & 0xff);+ value >>= 8;+ }+}++HRESULT WriteAll(HANDLE output, const BYTE* data, DWORD size) {+ while (size > 0) {+ DWORD written = 0;+ if (!WriteFile(output, data, size, &written, nullptr) || written == 0) {+ return HRESULT_FROM_WIN32(GetLastError());+ }+ data += written;+ size -= written;+ }+ return S_OK;+}++HRESULT WritePCM(HANDLE output, UINT64 timestamp100ns, const BYTE* data) {+ std::array<BYTE, 28> header{};+ header[0] = 'S'; header[1] = 'P'; header[2] = 'C'; header[3] = 'M';+ header[4] = 1;+ PutUint64BE(header.data() + 8, timestamp100ns);+ PutUint64BE(header.data() + 16, kChunkDuration100ns);+ PutUint32BE(header.data() + 24, kBytesPerChunk);+ HRESULT result = WriteAll(output, header.data(), static_cast<DWORD>(header.size()));+ return SUCCEEDED(result) ? WriteAll(output, data, kBytesPerChunk) : result;+}++HRESULT ActivateProcessLoopback(DWORD pid, HANDLE completed, ComPtr<IAudioClient>* client) {+ AUDIOCLIENT_ACTIVATION_PARAMS parameters{};+ parameters.ActivationType = AUDIOCLIENT_ACTIVATION_TYPE_PROCESS_LOOPBACK;+ parameters.ProcessLoopbackParams.TargetProcessId = pid;+ parameters.ProcessLoopbackParams.ProcessLoopbackMode =+ PROCESS_LOOPBACK_MODE_INCLUDE_TARGET_PROCESS_TREE;+ PROPVARIANT variant{};+ variant.vt = VT_BLOB;+ variant.blob.cbSize = sizeof(parameters);+ variant.blob.pBlobData = reinterpret_cast<BYTE*>(¶meters);++ auto handler = Make<ActivationHandler>(completed);+ if (!handler) return E_OUTOFMEMORY;+ ComPtr<IActivateAudioInterfaceAsyncOperation> operation;+ HRESULT result = ActivateAudioInterfaceAsync(VIRTUAL_AUDIO_DEVICE_PROCESS_LOOPBACK,+ __uuidof(IAudioClient), &variant,+ handler.Get(), &operation);+ if (FAILED(result)) return result;+ if (WaitForSingleObject(completed, 10'000) != WAIT_OBJECT_0) return HRESULT_FROM_WIN32(WAIT_TIMEOUT);+ result = handler->Result();+ if (SUCCEEDED(result)) *client = handler->Client();+ return result;+}++HRESULT Capture(DWORD pid, UINT64 expectedCreationTime) {+ HANDLE process = OpenProcess(SYNCHRONIZE | PROCESS_QUERY_LIMITED_INFORMATION, FALSE, pid);+ if (process == nullptr) return HRESULT_FROM_WIN32(GetLastError());+ HANDLE completed = CreateEventW(nullptr, FALSE, FALSE, nullptr);+ HANDLE sampleReady = CreateEventW(nullptr, FALSE, FALSE, nullptr);+ if (completed == nullptr || sampleReady == nullptr) {+ const HRESULT result = HRESULT_FROM_WIN32(GetLastError());+ if (completed != nullptr) CloseHandle(completed);+ if (sampleReady != nullptr) CloseHandle(sampleReady);+ CloseHandle(process);+ return result;+ }+ UINT64 creationTime = 0;+ HRESULT result = ReadProcessCreationTime(process, &creationTime);+ if (SUCCEEDED(result) && creationTime != expectedCreationTime) {+ result = HRESULT_FROM_WIN32(ERROR_INVALID_DATA);+ }+ ComPtr<IAudioClient> client;+ if (SUCCEEDED(result)) result = ActivateProcessLoopback(pid, completed, &client);+ ComPtr<IAudioCaptureClient> capture;+ WAVEFORMATEX format{};+ format.wFormatTag = WAVE_FORMAT_PCM;+ format.nChannels = kChannels;+ format.nSamplesPerSec = kSampleRate;+ format.wBitsPerSample = kBytesPerSample * 8;+ format.nBlockAlign = kChannels * kBytesPerSample;+ format.nAvgBytesPerSec = kSampleRate * format.nBlockAlign;+ if (SUCCEEDED(result)) {+ result = client->Initialize(AUDCLNT_SHAREMODE_SHARED,+ AUDCLNT_STREAMFLAGS_LOOPBACK | AUDCLNT_STREAMFLAGS_EVENTCALLBACK |+ AUDCLNT_STREAMFLAGS_AUTOCONVERTPCM | AUDCLNT_STREAMFLAGS_SRC_DEFAULT_QUALITY,+ 0, 0, &format, nullptr);+ }+ if (SUCCEEDED(result)) result = client->GetService(IID_PPV_ARGS(&capture));+ if (SUCCEEDED(result)) result = client->SetEventHandle(sampleReady);+ if (SUCCEEDED(result)) result = client->Start();++ std::vector<BYTE> pending;+ size_t consumed = 0;+ UINT64 nextTimestamp = 0;+ HANDLE output = GetStdHandle(STD_OUTPUT_HANDLE);+ const HANDLE waits[] = {process, sampleReady};+ while (SUCCEEDED(result)) {+ const DWORD wait = WaitForMultipleObjects(2, waits, FALSE, 1'000);+ if (wait == WAIT_OBJECT_0) {+ result = HRESULT_FROM_WIN32(ERROR_PROCESS_ABORTED);+ break;+ }+ if (wait == WAIT_TIMEOUT) continue;+ if (wait != WAIT_OBJECT_0 + 1) {+ result = HRESULT_FROM_WIN32(GetLastError());+ break;+ }+ UINT32 frames = 0;+ while (SUCCEEDED(result = capture->GetNextPacketSize(&frames)) && frames > 0) {+ BYTE* data = nullptr;+ DWORD flags = 0;+ UINT64 devicePosition = 0;+ UINT64 qpcPosition = 0;+ result = capture->GetBuffer(&data, &frames, &flags, &devicePosition, &qpcPosition);+ if (FAILED(result)) break;+ const size_t bytes = static_cast<size_t>(frames) * format.nBlockAlign;+ if ((flags & AUDCLNT_BUFFERFLAGS_TIMESTAMP_ERROR) != 0) {+ capture->ReleaseBuffer(frames);+ continue;+ }+ if ((flags & AUDCLNT_BUFFERFLAGS_DATA_DISCONTINUITY) != 0) {+ pending.clear();+ consumed = 0;+ }+ if (pending.size() == consumed) {+ pending.clear();+ consumed = 0;+ nextTimestamp = qpcPosition;+ }+ if ((flags & AUDCLNT_BUFFERFLAGS_SILENT) != 0) {+ pending.insert(pending.end(), bytes, 0);+ } else {+ pending.insert(pending.end(), data, data + bytes);+ }+ capture->ReleaseBuffer(frames);+ while (pending.size() - consumed >= kBytesPerChunk) {+ result = WritePCM(output, nextTimestamp, pending.data() + consumed);+ if (FAILED(result)) break;+ consumed += kBytesPerChunk;+ nextTimestamp += kChunkDuration100ns;+ }+ if (FAILED(result)) break;+ if (consumed > kBytesPerChunk * 4) {+ pending.erase(pending.begin(), pending.begin() + static_cast<ptrdiff_t>(consumed));+ consumed = 0;+ }+ }+ }+ if (client) client->Stop();+ CloseHandle(sampleReady);+ CloseHandle(completed);+ CloseHandle(process);+ return result;+}+} // namespace++int wmain(int argc, wchar_t** argv) {+ if (argc == 2 && std::wstring(argv[1]) == L"--list") return ListWindows();+ if (argc != 4 || std::wstring(argv[1]) != L"--capture") return 2;+ wchar_t* pidEnd = nullptr;+ const unsigned long parsedPID = wcstoul(argv[2], &pidEnd, 10);+ wchar_t* creationEnd = nullptr;+ const unsigned long long parsedCreationTime = wcstoull(argv[3], &creationEnd, 10);+ if (parsedPID == 0 || pidEnd == argv[2] || *pidEnd != L'\0' ||+ parsedCreationTime == 0 || creationEnd == argv[3] || *creationEnd != L'\0') return 2;+ if (_setmode(_fileno(stdout), _O_BINARY) == -1) return 2;+ HRESULT result = CoInitializeEx(nullptr, COINIT_MULTITHREADED);+ if (FAILED(result)) return 2;+ result = Capture(static_cast<DWORD>(parsedPID), static_cast<UINT64>(parsedCreationTime));+ CoUninitialize();+ if (FAILED(result)) std::cerr << "process_audio_capture_failed hresult=0x" << std::hex << result << '\n';+ return SUCCEEDED(result) ? 0 : 2;+}diff --git a/native/sender/go.mod b/native/sender/go.modindex f8bbc89f..dc136cf9 100644--- a/native/sender/go.mod+++ b/native/sender/go.mod@@ -7,6 +7,7 @@ require (github.com/pion/rtcp v1.2.17github.com/pion/rtp v1.10.5github.com/pion/webrtc/v4 v4.2.18+ golang.org/x/sys v0.41.0)require (@@ -27,6 +28,5 @@ require (github.com/wlynxg/anet v0.0.5 // indirectgolang.org/x/crypto v0.48.0 // indirectgolang.org/x/net v0.50.0 // indirect- golang.org/x/sys v0.41.0 // indirectgolang.org/x/time v0.14.0 // indirect)diff --git a/native/sender/internal/app/server.go b/native/sender/internal/app/server.goindex a9b0dfa4..485a276a 100644--- a/native/sender/internal/app/server.go+++ b/native/sender/internal/app/server.go@@ -16,6 +16,7 @@ import ("github.com/TNTcraftHIM/Screener/native/sender/internal/media""github.com/TNTcraftHIM/Screener/native/sender/internal/remote"+ "github.com/TNTcraftHIM/Screener/native/sender/internal/windowaudio""github.com/coder/websocket""github.com/coder/websocket/wsjson")@@ -59,12 +60,16 @@ type App struct {configTimeout time.DurationactiveViewers intwaitingViewers int+ audioProvider windowaudio.Provider+ audioTargets map[string]windowaudio.Target+ audioTarget *windowaudio.Target}type startRequest struct {- ServerURL string `json:"serverUrl"`- Password string `json:"password"`- Codec string `json:"codec,omitempty"`+ ServerURL string `json:"serverUrl"`+ Password string `json:"password"`+ Codec string `json:"codec,omitempty"`+ AudioTargetID string `json:"audioTargetId,omitempty"`}type encoderConfig struct {@@ -75,6 +80,12 @@ type encoderConfig struct {FPS int `json:"fps"`Bitrate int `json:"bitrate"`EncoderInstances int `json:"encoderInstances"`+ Audio bool `json:"audio"`+}++type localAudioTarget struct {+ ID string `json:"id"`+ Title string `json:"title"`}func New() (*App, error) {@@ -88,6 +99,8 @@ func New() (*App, error) {attachTimeout: mediaAttachTimeout,configTimeout: mediaConfigTimeout,codec: media.CodecVP8,+ audioProvider: windowaudio.NewProvider(),+ audioTargets: make(map[string]windowaudio.Target),}, nil}@@ -144,6 +157,8 @@ func (app *App) ServeHTTP(response http.ResponseWriter, request *http.Request) {app.handleStart(response, request)case "/api/stop":app.handleStop(response, request)+ case "/api/audio-targets":+ app.handleAudioTargets(response, request)case "/media":app.handleMedia(response, request)default:@@ -188,6 +203,16 @@ func (app *App) handleStart(response http.ResponseWriter, request *http.Request)return}app.mu.Lock()+ var audioTarget *windowaudio.Target+ if input.AudioTargetID != "" {+ selected, ok := app.audioTargets[input.AudioTargetID]+ if !ok {+ app.mu.Unlock()+ writeJSON(response, http.StatusBadRequest, map[string]string{"error": "selected audio target is no longer available"})+ return+ }+ audioTarget = &selected+ }if app.starting || app.session != nil {app.mu.Unlock()writeJSON(response, http.StatusConflict, map[string]string{"error": "A sharing session is already active"})@@ -205,6 +230,7 @@ func (app *App) handleStart(response http.ResponseWriter, request *http.Request)BaseURL: baseURL,HostAdmissionPassword: strings.TrimSpace(input.Password),Codec: codec,+ EnableAudio: audioTarget != nil,OnEvent: func(event remote.Event) { app.emit(generation, event) },})app.mu.Lock()@@ -219,6 +245,7 @@ func (app *App) handleStart(response http.ResponseWriter, request *http.Request)app.sessionCancel = startCancelapp.room = roomapp.codec = codec+ app.audioTarget = audioTargetapp.attachTimer = time.AfterFunc(app.attachTimeout, func() {app.mu.Lock()expired := app.generation == generation && app.session == session && app.mediaConn == nil@@ -254,6 +281,35 @@ func (app *App) handleStop(response http.ResponseWriter, request *http.Request)response.WriteHeader(http.StatusNoContent)}+func (app *App) handleAudioTargets(response http.ResponseWriter, request *http.Request) {+ if !app.authorizeAPI(response, request, http.MethodPost) {+ return+ }+ targets, err := app.audioProvider.List(request.Context())+ if err != nil {+ writeJSON(response, http.StatusOK, map[string]any{+ "supported": false, "targets": []localAudioTarget{},+ "reason": "Windows 11 process audio helper is unavailable",+ })+ return+ }+ localTargets := make([]localAudioTarget, 0, len(targets))+ resolved := make(map[string]windowaudio.Target, len(targets))+ for _, target := range targets {+ id, tokenErr := newOpaqueToken()+ if tokenErr != nil {+ writeJSON(response, http.StatusInternalServerError, map[string]string{"error": "create local audio target identity failed"})+ return+ }+ resolved[id] = target+ localTargets = append(localTargets, localAudioTarget{ID: id, Title: target.Title})+ }+ app.mu.Lock()+ app.audioTargets = resolved+ app.mu.Unlock()+ writeJSON(response, http.StatusOK, map[string]any{"supported": true, "targets": localTargets})+}+func (app *App) handleMedia(response http.ResponseWriter, request *http.Request) {if !exactRequest(request, app.expectedHost, app.origin) || request.Method != http.MethodGet {http.Error(response, "forbidden", http.StatusForbidden)@@ -285,7 +341,7 @@ func (app *App) handleMedia(response http.ResponseWriter, request *http.Request)app.releaseMediaClaim(generation, claimID, nil)return}- connection.SetReadLimit(media.MaxFrameBytes + media.FrameHeaderBytes)+ connection.SetReadLimit(media.MaxMediaPayload + media.EnvelopeHeaderBytes)app.mu.Lock()if app.generation != generation || app.session != session || app.mediaClaimID != claimID {app.mu.Unlock()@@ -332,7 +388,8 @@ func (app *App) handleMedia(response http.ResponseWriter, request *http.Request)return}var config encoderConfig- if err = decodeLocalPayload(payload, &config); err != nil || !validEncoderConfigForCodec(config, session.Codec()) {+ if err = decodeLocalPayload(payload, &config); err != nil ||+ !validEncoderConfigForCodec(config, session.Codec()) || config.Audio != session.AudioEnabled() {session.Fail(errors.New("local encoder configuration is invalid"))_ = connection.Close(websocket.StatusPolicyViolation, "invalid encoder config")return@@ -341,6 +398,39 @@ func (app *App) handleMedia(response http.ResponseWriter, request *http.Request)session.Fail(fmt.Errorf("acknowledge local encoder configuration: %w", err))return}+ app.mu.Lock()+ target := app.audioTarget+ app.mu.Unlock()+ captureContext, cancelCapture := context.WithCancel(app.ctx)+ audioDone := make(chan struct{})+ if target != nil {+ _ = app.writeLocal(connection, map[string]string{"kind": "audio-state", "state": "starting"})+ go func(selected windowaudio.Target) {+ defer close(audioDone)+ captureErr := app.audioProvider.Capture(captureContext, selected, func(chunk windowaudio.PCMChunk) error {+ envelope, encodeErr := media.EncodePacket(media.Packet{+ Kind: media.KindPCM, Timestamp100ns: chunk.Timestamp100ns,+ Duration100ns: chunk.Duration100ns, Data: chunk.Data,+ })+ if encodeErr != nil {+ return encodeErr+ }+ return app.writeLocalBinary(connection, envelope)+ })+ if captureErr != nil && captureContext.Err() == nil {+ _ = app.writeLocal(connection, map[string]string{+ "kind": "audio-state", "state": "unavailable",+ "message": "Target audio stopped; system audio was not substituted",+ })+ }+ }(*target)+ } else {+ close(audioDone)+ }+ defer func() {+ cancelCapture()+ <-audioDone+ }()for {messageType, payload, err = connection.Read(app.ctx)if err != nil {@@ -354,13 +444,18 @@ func (app *App) handleMedia(response http.ResponseWriter, request *http.Request)_ = connection.Close(websocket.StatusUnsupportedData, "binary frames required")return}- frame, decodeErr := media.DecodeFrame(payload)+ packet, decodeErr := media.DecodePacket(payload)if decodeErr != nil {- session.Fail(fmt.Errorf("decode local encoded frame: %w", decodeErr))- _ = connection.Close(websocket.StatusPolicyViolation, "invalid encoded frame")+ session.Fail(fmt.Errorf("decode local encoded media: %w", decodeErr))+ _ = connection.Close(websocket.StatusPolicyViolation, "invalid encoded media")return}- if err = session.WriteFrame(frame); err != nil {+ if packet.Kind != media.KindVideo && packet.Kind != media.KindOpus {+ session.Fail(errors.New("local media bridge received an invalid encoded media kind"))+ _ = connection.Close(websocket.StatusPolicyViolation, "encoded media required")+ return+ }+ if err = session.WriteMedia(packet); err != nil {return}}@@ -426,6 +521,14 @@ func (app *App) writeLocal(connection *websocket.Conn, value any) error {return wsjson.Write(ctx, connection, value)}+func (app *App) writeLocalBinary(connection *websocket.Conn, payload []byte) error {+ app.writeMu.Lock()+ defer app.writeMu.Unlock()+ ctx, cancel := context.WithTimeout(app.ctx, localWriteTimeout)+ defer cancel()+ return connection.Write(ctx, websocket.MessageBinary, payload)+}+func (app *App) releaseMediaClaim(generation, claimID uint64, connection *websocket.Conn) {app.mu.Lock()defer app.mu.Unlock()@@ -465,6 +568,7 @@ func (app *App) stopActiveGeneration(expectedGeneration uint64) {}app.room = remote.Room{}app.codec = media.CodecVP8+ app.audioTarget = nilapp.activeViewers = 0app.waitingViewers = 0app.mu.Unlock()diff --git a/native/sender/internal/app/server_test.go b/native/sender/internal/app/server_test.goindex c6926f6c..8f4544a3 100644--- a/native/sender/internal/app/server_test.go+++ b/native/sender/internal/app/server_test.go@@ -3,7 +3,6 @@ package appimport ("bytes""context"- "encoding/binary""encoding/json""errors""net/http"@@ -15,6 +14,7 @@ import ("github.com/TNTcraftHIM/Screener/native/sender/internal/media""github.com/TNTcraftHIM/Screener/native/sender/internal/remote"+ "github.com/TNTcraftHIM/Screener/native/sender/internal/windowaudio""github.com/coder/websocket")@@ -62,6 +62,49 @@ func TestDecodeLocalPayloadIsStrictAndSingleValued(t *testing.T) {}}+func TestAudioTargetListingKeepsProcessIdentityBehindAnOpaqueLocalID(t *testing.T) {+ application, err := New()+ if err != nil {+ t.Fatal(err)+ }+ application.audioProvider = fakeAudioProvider{targets: []windowaudio.Target{{+ PID: 424_242, CreationTime: 987_654_321, Title: "Game Window",+ }}}+ if _, err = application.Start(); err != nil {+ t.Fatal(err)+ }+ defer application.Close()++ request, err := http.NewRequest(http.MethodPost, application.origin+"/api/audio-targets", nil)+ if err != nil {+ t.Fatal(err)+ }+ request.Header.Set("Origin", application.origin)+ request.Header.Set(processTokenHeader, application.token)+ response, err := http.DefaultClient.Do(request)+ if err != nil {+ t.Fatal(err)+ }+ defer response.Body.Close()+ var result struct {+ Supported bool `json:"supported"`+ Targets []localAudioTarget `json:"targets"`+ }+ if json.NewDecoder(response.Body).Decode(&result) != nil || !result.Supported || len(result.Targets) != 1 {+ t.Fatalf("audio target response = %+v", result)+ }+ if result.Targets[0].ID == "" || result.Targets[0].Title != "Game Window" ||+ strings.Contains(result.Targets[0].ID, "424242") || strings.Contains(result.Targets[0].ID, "987654321") {+ t.Fatalf("audio target was not locally anonymized: %+v", result.Targets[0])+ }+ application.mu.Lock()+ resolved := application.audioTargets[result.Targets[0].ID]+ application.mu.Unlock()+ if resolved.PID != 424_242 || resolved.CreationTime != 987_654_321 {+ t.Fatal("opaque target did not resolve in process memory")+ }+}+func TestStaleSessionGenerationCannotMutateOrStopTheCurrentSession(t *testing.T) {application := &App{generation: 7, activeViewers: 1, waitingViewers: 2}application.emit(6, remote.Event{Kind: "viewer-count", ActiveViewers: 9, WaitingViewers: 9})@@ -193,19 +236,26 @@ func TestConfiguredMediaFramesWithCaptureJitterReachFanout(t *testing.T) {application, events, terminal := startObservedTestApplication(t)connection := openConfiguredMedia(t, application)- payload := make([]byte, media.FrameHeaderBytes+4)- payload[0] = 1- binary.BigEndian.PutUint64(payload[1:9], 1_000_000)- binary.BigEndian.PutUint64(payload[9:17], 33_333)- copy(payload[media.FrameHeaderBytes:], []byte{0x10, 0x00, 0x00, 0x00})+ payload, err := media.EncodePacket(media.Packet{+ Kind: media.KindVideo, KeyFrame: true, Timestamp100ns: 10_000_000,+ Duration100ns: 333_330, Data: []byte{0x10, 0x00, 0x00, 0x00},+ })+ if err != nil {+ t.Fatal(err)+ }writeContext, cancelWrite := context.WithTimeout(context.Background(), time.Second)- err := connection.Write(writeContext, websocket.MessageBinary, payload)+ err = connection.Write(writeContext, websocket.MessageBinary, payload)cancelWrite()if err != nil {t.Fatalf("write encoded frame: %v", err)}- payload[0] = 2- binary.BigEndian.PutUint64(payload[1:9], 1_033_000)+ payload, err = media.EncodePacket(media.Packet{+ Kind: media.KindVideo, Timestamp100ns: 10_330_000,+ Duration100ns: 333_330, Data: []byte{0x10, 0x00, 0x00, 0x00},+ })+ if err != nil {+ t.Fatal(err)+ }writeContext, cancelWrite = context.WithTimeout(context.Background(), time.Second)err = connection.Write(writeContext, websocket.MessageBinary, payload)cancelWrite()@@ -638,3 +688,13 @@ func appRoomResponse(origin string) map[string]any {"viewerPolicy": "private-link", "viewerGrantExpiresAt": expiresAt.Format(time.RFC3339), "expiresAt": expiresAt.Format(time.RFC3339),}}++type fakeAudioProvider struct{ targets []windowaudio.Target }++func (provider fakeAudioProvider) List(context.Context) ([]windowaudio.Target, error) {+ return provider.targets, nil+}++func (fakeAudioProvider) Capture(context.Context, windowaudio.Target, func(windowaudio.PCMChunk) error) error {+ return nil+}diff --git a/native/sender/internal/app/ui/app.js b/native/sender/internal/app/ui/app.jsindex 3215f5fc..a5a8ffca 100644--- a/native/sender/internal/app/ui/app.js+++ b/native/sender/internal/app/ui/app.js@@ -12,6 +12,10 @@ const FRAME_DURATION = Math.round(1_000_000 / FPS);const MAX_ENCODER_QUEUE = 2;const MAX_SOCKET_BUFFER = 512 * 1024;const MAX_FRAME_BYTES = 1 << 20;+const ENVELOPE_HEADER_BYTES = 20;+const MEDIA_KIND_VIDEO = 1;+const MEDIA_KIND_PCM = 2;+const MEDIA_KIND_OPUS = 3;const fragment = new URLSearchParams(location.hash.slice(1));const processToken = fragment.get("token") || "";@@ -21,6 +25,8 @@ const elements = {serverURL: document.querySelector("#server-url"),password: document.querySelector("#password"),codec: document.querySelector("#codec"),+ audioTarget: document.querySelector("#audio-target"),+ refreshAudio: document.querySelector("#refresh-audio"),start: document.querySelector("#start"),stop: document.querySelector("#stop"),preview: document.querySelector("#preview"),@@ -36,11 +42,14 @@ const elements = {bridgeDiagnostic: document.querySelector("#bridge-diagnostic"),helperDiagnostic: document.querySelector("#helper-diagnostic"),edgeDiagnostic: document.querySelector("#edge-diagnostic"),+ audioStatus: document.querySelector("#audio-status"),};let stream = null;let processorReader = null;let encoder = null;+let audioEncoder = null;+let silentAudioChunks = 0;let mediaSocket = null;let sharing = false;let stopping = false;@@ -71,6 +80,8 @@ elements.copy.addEventListener("click", async () => {showError("无法复制链接,请直接打开邀请链接");}});+elements.refreshAudio.addEventListener("click", () => void loadAudioTargets());+if (processToken) void loadAudioTargets();async function startSharing() {if (sharing || stopping || startInFlight) return;@@ -83,6 +94,13 @@ async function startSharing() {elements.start.disabled = true;const generation = ++startGeneration;const codec = selectedCodec();+ const audioTargetId = elements.audioTarget.value;+ if (audioTargetId && (!("AudioEncoder" in window) || !("AudioData" in window))) {+ showError("当前浏览器缺少 WebCodecs Opus 编码支持,请关闭目标进程音频或使用当前版 Chrome / Edge");+ startInFlight = false;+ elements.start.disabled = false;+ return;+ }setStatus("选择屏幕");try {const capturedStream = await navigator.mediaDevices.getDisplayMedia({@@ -111,6 +129,7 @@ async function startSharing() {serverUrl: elements.serverURL.value,password: elements.password.value,codec,+ audioTargetId,});assertCurrentStart(generation);if (pendingStartGeneration === generation) pendingStartGeneration = 0;@@ -120,7 +139,8 @@ async function startSharing() {elements.inviteLink.textContent = room.inviteUrl;elements.invite.hidden = false;- await connectMediaSocket(generation, codec);+ if (audioTargetId) await startAudioEncoder(generation);+ await connectMediaSocket(generation, codec, Boolean(audioTargetId));assertCurrentStart(generation);const activeEncoder = await startEncoder(generation, codec);assertCurrentStart(generation);@@ -139,7 +159,7 @@ async function startSharing() {}}-async function connectMediaSocket(generation, codec) {+async function connectMediaSocket(generation, codec, audio) {const url = new URL("/media", location.href);url.protocol = "ws:";const socket = new WebSocket(url, [`screener.token.${processToken}`]);@@ -166,6 +186,7 @@ async function connectMediaSocket(generation, codec) {fps: FPS,bitrate: BITRATE,encoderInstances: 1,+ audio,}));await accepted;assertCurrentStart(generation, socket);@@ -275,12 +296,13 @@ function sendEncodedChunk(generation, activeEncoder, chunk) {}const data = new Uint8Array(chunk.byteLength);chunk.copyTo(data);- const message = new ArrayBuffer(17 + data.byteLength);- const view = new DataView(message);- view.setUint8(0, chunk.type === "key" ? 1 : 2);- view.setBigUint64(1, BigInt(Math.max(0, chunk.timestamp)), false);- view.setBigUint64(9, BigInt(chunk.duration || FRAME_DURATION), false);- new Uint8Array(message, 17).set(data);+ const message = mediaEnvelope(+ MEDIA_KIND_VIDEO,+ chunk.type === "key" ? 1 : 0,+ BigInt(Math.max(0, chunk.timestamp)) * 10n,+ BigInt(chunk.duration || FRAME_DURATION) * 10n,+ data,+ );mediaSocket.send(message);metrics.encoderOutputs += 1;metrics.encodedBytes += data.byteLength;@@ -293,6 +315,10 @@ function sendEncodedChunk(generation, activeEncoder, chunk) {function handleLocalEvent(generation, socket, event) {if (generation !== startGeneration || socket !== mediaSocket) return;+ if (event.data instanceof ArrayBuffer) {+ handlePCM(generation, event.data);+ return;+ }if (typeof event.data !== "string") return;const message = parseLocalMessage(event.data);if (!message) return;@@ -305,6 +331,11 @@ function handleLocalEvent(generation, socket, event) {updateDiagnostics();}if (message.kind === "warning" && message.message) showError(message.message);+ if (message.kind === "audio-state") {+ elements.audioStatus.textContent = message.state === "unavailable"+ ? "不可用(未回退系统声音)"+ : "正在连接目标进程";+ }if (message.kind === "fatal") {void failSharing(generation, message.message || "分享会话已失败并停止");}@@ -347,6 +378,12 @@ async function stopSharing(expectedGeneration = startGeneration) {rememberError(error);}encoder = null;+ try {+ if (audioEncoder && audioEncoder.state !== "closed") audioEncoder.close();+ } catch (error) {+ rememberError(error);+ }+ audioEncoder = null;const socket = mediaSocket;mediaSocket = null;try {@@ -365,6 +402,7 @@ async function stopSharing(expectedGeneration = startGeneration) {} finally {frameNumber = 0;forceKeyFrame = true;+ silentAudioChunks = 0;captureSettings = null;nativeDiagnostics = null;metrics = emptyMetrics();@@ -379,6 +417,7 @@ async function stopSharing(expectedGeneration = startGeneration) {elements.start.disabled = startInFlight;showViewerCounts(0, 0);setStatus("未开始");+ elements.audioStatus.textContent = "关闭";stopping = false;}}@@ -446,6 +485,100 @@ async function localRequest(path, body) {return result;}+async function loadAudioTargets() {+ elements.refreshAudio.disabled = true;+ const previous = elements.audioTarget.value;+ try {+ const result = await localRequest("/api/audio-targets");+ elements.audioTarget.replaceChildren(new Option("关闭", ""));+ for (const target of Array.isArray(result.targets) ? result.targets : []) {+ if (target?.id && target?.title) elements.audioTarget.add(new Option(target.title, target.id));+ }+ if ([...elements.audioTarget.options].some((option) => option.value === previous)) {+ elements.audioTarget.value = previous;+ }+ elements.audioTarget.disabled = !result.supported;+ elements.audioStatus.textContent = result.supported ? "关闭" : "Windows 11 helper 不可用";+ } catch (error) {+ elements.audioTarget.disabled = true;+ elements.audioStatus.textContent = readableError(error, "窗口列表不可用");+ } finally {+ elements.refreshAudio.disabled = false;+ }+}++async function startAudioEncoder(generation) {+ const config = { codec: "opus", sampleRate: 48_000, numberOfChannels: 2, bitrate: 128_000 };+ const support = await AudioEncoder.isConfigSupported(config);+ assertCurrentStart(generation);+ if (!support.supported) throw new Error("当前浏览器不支持 WebCodecs Opus 编码");+ let active;+ active = new AudioEncoder({+ output: (chunk) => sendEncodedAudio(generation, active, chunk),+ error: (error) => void failSharing(generation, `音频编码器失败:${error.message}`),+ });+ audioEncoder = active;+ active.configure(support.config);+ elements.audioStatus.textContent = "等待目标进程音频";+}++function handlePCM(generation, message) {+ if (generation !== startGeneration || !audioEncoder || audioEncoder.state !== "configured") return;+ if (audioEncoder.encodeQueueSize >= 4) return;+ try {+ const view = new DataView(message);+ if (message.byteLength <= ENVELOPE_HEADER_BYTES || view.getUint8(0) !== 1 ||+ view.getUint8(1) !== MEDIA_KIND_PCM || view.getUint8(2) !== 0 || view.getUint8(3) !== 0) {+ throw new Error("本地 PCM envelope 无效");+ }+ const data = new Uint8Array(message, ENVELOPE_HEADER_BYTES);+ if (data.byteLength % 4 !== 0) throw new Error("本地 PCM 帧长度无效");+ let audible = false;+ for (const byte of data) if (byte !== 0) { audible = true; break; }+ silentAudioChunks = audible ? 0 : silentAudioChunks + 1;+ elements.audioStatus.textContent = silentAudioChunks >= 100+ ? "目标进程当前无声(未回退系统声音)"+ : "目标进程树音频";+ const audio = new AudioData({+ format: "s16", sampleRate: 48_000, numberOfChannels: 2,+ numberOfFrames: data.byteLength / 4,+ timestamp: Number(view.getBigUint64(4, false) / 10n), data,+ });+ audioEncoder.encode(audio);+ audio.close();+ } catch (error) {+ void failSharing(generation, readableError(error, "处理目标进程音频失败"));+ }+}++function sendEncodedAudio(generation, activeEncoder, chunk) {+ if (generation !== startGeneration || stopping || audioEncoder !== activeEncoder ||+ !mediaSocket || mediaSocket.readyState !== WebSocket.OPEN) return;+ if (chunk.byteLength < 1 || chunk.byteLength > 64 * 1024 ||+ mediaSocket.bufferedAmount > MAX_SOCKET_BUFFER) {+ void failSharing(generation, "本地音频桥接超过安全上限");+ return;+ }+ const data = new Uint8Array(chunk.byteLength);+ chunk.copyTo(data);+ mediaSocket.send(mediaEnvelope(+ MEDIA_KIND_OPUS, 0, BigInt(Math.max(0, chunk.timestamp)) * 10n,+ BigInt(chunk.duration || 20_000) * 10n, data,+ ));+}++function mediaEnvelope(kind, flags, timestamp100ns, duration100ns, data) {+ const message = new ArrayBuffer(ENVELOPE_HEADER_BYTES + data.byteLength);+ const view = new DataView(message);+ view.setUint8(0, 1);+ view.setUint8(1, kind);+ view.setUint8(2, flags);+ view.setBigUint64(4, timestamp100ns, false);+ view.setBigUint64(12, duration100ns, false);+ new Uint8Array(message, ENVELOPE_HEADER_BYTES).set(data);+ return message;+}+function showViewerCounts(activeValue, waitingValue) {const active = boundedCount(activeValue);const waiting = boundedCount(waitingValue);diff --git a/native/sender/internal/app/ui/index.html b/native/sender/internal/app/ui/index.htmlindex 199e897d..2a75ffaa 100644--- a/native/sender/internal/app/ui/index.html+++ b/native/sender/internal/app/ui/index.html@@ -45,8 +45,15 @@<option value="h264">H.264 (hardware preference)</option></select></label>+ <label>+ <span>目标进程音频</span>+ <span class="select-row">+ <select id="audio-target"><option value="">关闭</option></select>+ <button id="refresh-audio" class="icon-button" type="button" title="刷新窗口列表" aria-label="刷新窗口列表">↻</button>+ </span>+ </label>+ <p class="audio-note">请在浏览器画面选择器中选择同一目标。音频只包含所选进程树,失败时不会改用系统混音。</p><button id="start" class="button primary" type="button">开始分享</button>- <p class="notice">当前 canary 仅发送画面,不含音频。</p><p id="error" class="error" role="alert"></p><details><summary>技术信息</summary>@@ -57,7 +64,7 @@<div><dt>本地桥接</dt><dd id="bridge-diagnostic">尚无数据</dd></div><div><dt>Helper</dt><dd id="helper-diagnostic">尚无数据</dd></div><div><dt>出边</dt><dd id="edge-diagnostic">无活动媒体边</dd></div>- <div><dt>音频</dt><dd>当前 canary 未启用,电影与视频音质仍待验收</dd></div>+ <div><dt>音频</dt><dd id="audio-status">关闭</dd></div></dl></details></aside>diff --git a/native/sender/internal/app/ui/styles.css b/native/sender/internal/app/ui/styles.cssindex 062d57a9..7d6da525 100644--- a/native/sender/internal/app/ui/styles.css+++ b/native/sender/internal/app/ui/styles.css@@ -16,8 +16,8 @@* { box-sizing: border-box; }html, body { min-width: 320px; min-height: 100%; margin: 0; }body { min-height: 100vh; background: #101310; }-button, input { font: inherit; letter-spacing: 0; }-button:focus-visible, input:focus-visible, a:focus-visible, summary:focus-visible { outline: 2px solid var(--green); outline-offset: 2px; }+button, input, select { font: inherit; letter-spacing: 0; }+button:focus-visible, input:focus-visible, select:focus-visible, a:focus-visible, summary:focus-visible { outline: 2px solid var(--green); outline-offset: 2px; }.app-header {display: flex;@@ -44,7 +44,9 @@ video { display: block; width: 100%; height: 100%; object-fit: contain; }.controls { display: grid; align-content: start; gap: 18px; padding: 32px 24px; border-left: 1px solid var(--border); background: var(--surface); }label { display: grid; gap: 8px; color: var(--muted); font-size: 13px; }-input { width: 100%; min-height: 44px; padding: 10px 11px; border: 1px solid var(--border); border-radius: 6px; color: #f1f3f1; background: #101310; }+input, select { width: 100%; min-width: 0; min-height: 44px; padding: 10px 11px; border: 1px solid var(--border); border-radius: 6px; color: #f1f3f1; background: #101310; }+.select-row { display: grid; grid-template-columns: minmax(0, 1fr) 44px; gap: 8px; }+.audio-note { margin: -8px 0 0; color: var(--muted); font-size: 12px; line-height: 1.45; }.button, .icon-button { min-height: 44px; border: 1px solid transparent; border-radius: 6px; color: inherit; cursor: pointer; }.button:disabled { cursor: not-allowed; opacity: .5; }.primary { color: #07150c; font-weight: 700; background: var(--green); }diff --git a/native/sender/internal/media/audio.go b/native/sender/internal/media/audio.gonew file mode 100644index 00000000..513816d6--- /dev/null+++ b/native/sender/internal/media/audio.go@@ -0,0 +1,178 @@+package media++import (+ "context"+ "errors"+ "fmt"+ "math"+ "sync"++ "github.com/pion/rtp"+ "github.com/pion/rtp/codecs"+ "github.com/pion/webrtc/v4"+)++const (+ audioClockRate = 48_000+ audioQueueDepth = 16+ OpusSDPFmtpLine = "minptime=10;useinbandfec=1"+)++type AudioMetrics struct {+ PacketsWritten uint64 `json:"packetsWritten"`+ SourceRTPPacketsWritten uint64 `json:"sourceRtpPacketsWritten"`+ SourceRTPBytesWritten uint64 `json:"sourceRtpBytesWritten"`+ LastSourceTimestamp100ns uint64 `json:"lastSourceTimestamp100ns"`+}++type AudioFanout struct {+ track *webrtc.TrackLocalStaticRTP+ writer rtpWriter+ queue chan Packet+ ctx context.Context+ cancel context.CancelFunc+ done chan struct{}+ closeOnce sync.Once+ fatalOnce sync.Once+ onFatal func(error)+ metricsMu sync.Mutex+ metrics AudioMetrics+}++func NewAudioFanout(onFatal func(error)) (*AudioFanout, error) {+ capability := webrtc.RTPCodecCapability{+ MimeType: webrtc.MimeTypeOpus, ClockRate: audioClockRate, Channels: 2, SDPFmtpLine: OpusSDPFmtpLine,+ }+ track, err := webrtc.NewTrackLocalStaticRTP(capability, "audio", "screener")+ if err != nil {+ return nil, fmt.Errorf("create shared Opus track: %w", err)+ }+ return newAudioFanout(track, track, onFatal), nil+}++func newAudioFanout(track *webrtc.TrackLocalStaticRTP, writer rtpWriter, onFatal func(error)) *AudioFanout {+ ctx, cancel := context.WithCancel(context.Background())+ fanout := &AudioFanout{+ track: track, writer: writer, queue: make(chan Packet, audioQueueDepth),+ ctx: ctx, cancel: cancel, done: make(chan struct{}), onFatal: onFatal,+ }+ go fanout.writeLoop()+ return fanout+}++func (fanout *AudioFanout) Track() *webrtc.TrackLocalStaticRTP { return fanout.track }++func (fanout *AudioFanout) Push(packet Packet) error {+ if packet.Kind != KindOpus {+ return errors.New("shared audio fanout requires Opus")+ }+ if err := validatePacket(packet); err != nil {+ return err+ }+ select {+ case <-fanout.ctx.Done():+ return errors.New("shared audio fanout is closed")+ case fanout.queue <- packet:+ return nil+ default:+ return errors.New("shared audio queue exceeded its bound")+ }+}++func (fanout *AudioFanout) Snapshot() AudioMetrics {+ fanout.metricsMu.Lock()+ defer fanout.metricsMu.Unlock()+ return fanout.metrics+}++func (fanout *AudioFanout) Close() {+ fanout.stop()+ <-fanout.done+}++func (fanout *AudioFanout) writeLoop() {+ defer close(fanout.done)+ timeline := newAudioTimeline()+ for {+ select {+ case <-fanout.ctx.Done():+ return+ case packet := <-fanout.queue:+ packets, err := timeline.Packetize(packet)+ if err != nil {+ fanout.fail(err)+ return+ }+ var sourceBytes uint64+ for _, rtpPacket := range packets {+ if err = fanout.writer.WriteRTP(rtpPacket); err != nil {+ fanout.fail(errors.New("write shared Opus RTP packet failed"))+ return+ }+ sourceBytes += uint64(rtpPacket.MarshalSize())+ }+ fanout.metricsMu.Lock()+ fanout.metrics.PacketsWritten+++ fanout.metrics.SourceRTPPacketsWritten += uint64(len(packets))+ fanout.metrics.SourceRTPBytesWritten += sourceBytes+ fanout.metrics.LastSourceTimestamp100ns = packet.Timestamp100ns+ fanout.metricsMu.Unlock()+ }+ }+}++func (fanout *AudioFanout) fail(err error) {+ fanout.fatalOnce.Do(func() {+ fanout.stop()+ if fanout.onFatal != nil {+ go func() { <-fanout.done; fanout.onFatal(err) }()+ }+ })+}++func (fanout *AudioFanout) stop() { fanout.closeOnce.Do(fanout.cancel) }++type audioTimeline struct {+ packetizer rtp.Packetizer+ hasPrevious bool+ previousTimestamp uint64+}++func newAudioTimeline() *audioTimeline {+ return &audioTimeline{packetizer: rtp.NewPacketizer(+ 1200, 111, 0, &codecs.OpusPayloader{}, rtp.NewRandomSequencer(), audioClockRate,+ )}+}++func (timeline *audioTimeline) Packetize(packet Packet) ([]*rtp.Packet, error) {+ if timeline.hasPrevious {+ if packet.Timestamp100ns <= timeline.previousTimestamp {+ return nil, errors.New("encoded audio timestamp did not increase")+ }+ delta, err := audioSamples(packet.Timestamp100ns - timeline.previousTimestamp)+ if err != nil {+ return nil, err+ }+ if delta == 0 {+ delta = 1+ }+ timeline.packetizer.SkipSamples(delta)+ }+ packets := timeline.packetizer.Packetize(packet.Data, 0)+ if len(packets) == 0 {+ return nil, errors.New("Opus access unit exceeded the RTP payload bound")+ }+ timeline.hasPrevious = true+ timeline.previousTimestamp = packet.Timestamp100ns+ return packets, nil+}++func audioSamples(value100ns uint64) (uint32, error) {+ seconds := value100ns / 10_000_000+ remainder := value100ns % 10_000_000+ samples := seconds*audioClockRate + (remainder*audioClockRate+5_000_000)/10_000_000+ if samples > math.MaxUint32 {+ return 0, errors.New("encoded audio timestamp gap is too large")+ }+ return uint32(samples), nil+}diff --git a/native/sender/internal/media/audio_test.go b/native/sender/internal/media/audio_test.gonew file mode 100644index 00000000..3c63ff6b--- /dev/null+++ b/native/sender/internal/media/audio_test.go@@ -0,0 +1,56 @@+package media++import (+ "testing"+ "time"++ "github.com/pion/rtp"+ "github.com/pion/webrtc/v4"+)++func TestAudioFanoutUsesOneOpusTimeline(t *testing.T) {+ written := make(chan *rtp.Packet, 2)+ fanout := newAudioFanout(nil, rtpWriterFunc(func(packet *rtp.Packet) error {+ written <- packet.Clone()+ return nil+ }), nil)+ defer fanout.Close()++ for _, timestamp := range []uint64{100_000, 300_000} {+ if err := fanout.Push(Packet{+ Kind: KindOpus, Timestamp100ns: timestamp, Duration100ns: 200_000,+ Data: []byte{0xf8, 0xff, 0xfe},+ }); err != nil {+ t.Fatal(err)+ }+ }+ first := awaitAudioPacket(t, written)+ second := awaitAudioPacket(t, written)+ if delta := second.Timestamp - first.Timestamp; delta != 960 {+ t.Fatalf("Opus RTP timestamp delta = %d, want 960", delta)+ }+}++func TestOpusTrackCapabilityIsBrowserCompatible(t *testing.T) {+ fanout, err := NewAudioFanout(nil)+ if err != nil {+ t.Fatal(err)+ }+ defer fanout.Close()+ codec := fanout.Track().Codec()+ if codec.MimeType != webrtc.MimeTypeOpus || codec.ClockRate != 48_000 || codec.Channels != 2 ||+ codec.SDPFmtpLine != OpusSDPFmtpLine {+ t.Fatalf("Opus capability = %+v", codec)+ }+}++func awaitAudioPacket(t *testing.T, packets <-chan *rtp.Packet) *rtp.Packet {+ t.Helper()+ select {+ case packet := <-packets:+ return packet+ case <-time.After(time.Second):+ t.Fatal("timed out waiting for shared Opus RTP")+ return nil+ }+}diff --git a/native/sender/internal/media/envelope.go b/native/sender/internal/media/envelope.gonew file mode 100644index 00000000..87cb53ab--- /dev/null+++ b/native/sender/internal/media/envelope.go@@ -0,0 +1,104 @@+package media++import (+ "encoding/binary"+ "errors"+ "time"+)++const (+ EnvelopeVersion = 1+ EnvelopeHeaderBytes = 20+ MaxMediaPayload = 1 << 20+ MaxPCMBytes = 48_000 * 2 * 2 / 10 // 100 ms of 48 kHz stereo s16.++ flagKeyFrame = 1 << 0+)++type Kind uint8++const (+ KindVideo Kind = 1+ KindPCM Kind = 2+ KindOpus Kind = 3+)++type Packet struct {+ Kind Kind+ KeyFrame bool+ Timestamp100ns uint64+ Duration100ns uint64+ Data []byte+}++func EncodePacket(packet Packet) ([]byte, error) {+ if err := validatePacket(packet); err != nil {+ return nil, err+ }+ payload := make([]byte, EnvelopeHeaderBytes+len(packet.Data))+ payload[0] = EnvelopeVersion+ payload[1] = byte(packet.Kind)+ if packet.KeyFrame {+ payload[2] = flagKeyFrame+ }+ binary.BigEndian.PutUint64(payload[4:12], packet.Timestamp100ns)+ binary.BigEndian.PutUint64(payload[12:20], packet.Duration100ns)+ copy(payload[EnvelopeHeaderBytes:], packet.Data)+ return payload, nil+}++func DecodePacket(payload []byte) (Packet, error) {+ if len(payload) <= EnvelopeHeaderBytes || len(payload)-EnvelopeHeaderBytes > MaxMediaPayload {+ return Packet{}, errors.New("media envelope is empty, truncated, or too large")+ }+ if payload[0] != EnvelopeVersion || payload[3] != 0 || payload[2]&^byte(flagKeyFrame) != 0 {+ return Packet{}, errors.New("media envelope header is invalid")+ }+ packet := Packet{+ Kind: Kind(payload[1]),+ KeyFrame: payload[2]&flagKeyFrame != 0,+ Timestamp100ns: binary.BigEndian.Uint64(payload[4:12]),+ Duration100ns: binary.BigEndian.Uint64(payload[12:20]),+ Data: append([]byte(nil), payload[EnvelopeHeaderBytes:]...),+ }+ if err := validatePacket(packet); err != nil {+ return Packet{}, err+ }+ return packet, nil+}++func (packet Packet) VideoFrame() (Frame, error) {+ if packet.Kind != KindVideo {+ return Frame{}, errors.New("media envelope is not encoded video")+ }+ return Frame{+ KeyFrame: packet.KeyFrame,+ TimestampMicros: packet.Timestamp100ns / 10,+ DurationMicros: packet.Duration100ns / 10,+ Data: packet.Data,+ }, nil+}++func validatePacket(packet Packet) error {+ if len(packet.Data) == 0 || len(packet.Data) > MaxMediaPayload || packet.Duration100ns == 0 ||+ packet.Duration100ns > uint64(time.Second/(100*time.Nanosecond)) {+ return errors.New("media envelope violates the bounded media contract")+ }+ switch packet.Kind {+ case KindVideo:+ if packet.Duration100ns < 10 || packet.Duration100ns%10 != 0 {+ return errors.New("video envelope timestamp unit is invalid")+ }+ case KindPCM:+ if packet.KeyFrame || len(packet.Data) > MaxPCMBytes || len(packet.Data)%4 != 0 {+ return errors.New("PCM envelope is invalid")+ }+ case KindOpus:+ if packet.KeyFrame || len(packet.Data) > 64<<10 {+ return errors.New("Opus envelope is invalid")+ }+ default:+ return errors.New("media envelope kind is invalid")+ }+ return nil+}diff --git a/native/sender/internal/media/fanout.go b/native/sender/internal/media/fanout.goindex 598cbf7b..ed613467 100644--- a/native/sender/internal/media/fanout.go+++ b/native/sender/internal/media/fanout.go@@ -2,12 +2,10 @@ package mediaimport ("context"- "encoding/binary""errors""fmt""math""sync"- "time""github.com/pion/rtp""github.com/pion/rtp/codecs"@@ -15,10 +13,8 @@ import ()const (- FrameHeaderBytes = 17- MaxFrameBytes = 1 << 20- queueCapacity = 8- videoClockRate = 90_000+ queueCapacity = 8+ videoClockRate = 90_000)// Codec selects the encoded representation carried by the native bridge.@@ -135,7 +131,7 @@ func (fanout *Fanout) Track() *webrtc.TrackLocalStaticRTP {}func (fanout *Fanout) Push(frame Frame) error {- if len(frame.Data) == 0 || len(frame.Data) > MaxFrameBytes || frame.DurationMicros == 0 {+ if len(frame.Data) == 0 || len(frame.Data) > MaxMediaPayload || frame.DurationMicros == 0 {return errors.New("encoded frame violates the bounded media contract")}result, err := fanout.queue.Push(frame)@@ -283,25 +279,3 @@ func microsToSamples(micros uint64) (uint32, error) {}return uint32(samples), nil}--func DecodeFrame(payload []byte) (Frame, error) {- if len(payload) <= FrameHeaderBytes || len(payload)-FrameHeaderBytes > MaxFrameBytes {- return Frame{}, errors.New("encoded frame is empty, truncated, or too large")- }- frame := Frame{- TimestampMicros: binary.BigEndian.Uint64(payload[1:9]),- DurationMicros: binary.BigEndian.Uint64(payload[9:17]),- Data: append([]byte(nil), payload[FrameHeaderBytes:]...),- }- switch payload[0] {- case 1:- frame.KeyFrame = true- case 2:- default:- return Frame{}, errors.New("encoded frame type is invalid")- }- if frame.DurationMicros == 0 || frame.DurationMicros > uint64(time.Second/time.Microsecond) {- return Frame{}, errors.New("encoded frame duration is invalid")- }- return frame, nil-}diff --git a/native/sender/internal/media/fanout_test.go b/native/sender/internal/media/fanout_test.goindex e80e05d9..9285b95a 100644--- a/native/sender/internal/media/fanout_test.go+++ b/native/sender/internal/media/fanout_test.go@@ -1,7 +1,6 @@package mediaimport (- "encoding/binary""errors""runtime""sync"@@ -207,14 +206,19 @@ func TestFanoutCloseDropsBacklogWithoutLateWritesOrFatal(t *testing.T) {}}-func TestDecodeFrame(t *testing.T) {- payload := make([]byte, FrameHeaderBytes+3)- payload[0] = 1- binary.BigEndian.PutUint64(payload[1:9], 123)- binary.BigEndian.PutUint64(payload[9:17], 33_333)- copy(payload[17:], []byte{1, 2, 3})-- frame, err := DecodeFrame(payload)+func TestTypedEnvelopePreservesVideoTimeline(t *testing.T) {+ payload, err := EncodePacket(Packet{+ Kind: KindVideo, KeyFrame: true, Timestamp100ns: 1_230,+ Duration100ns: 333_330, Data: []byte{1, 2, 3},+ })+ if err != nil {+ t.Fatal(err)+ }+ packet, err := DecodePacket(payload)+ if err != nil {+ t.Fatal(err)+ }+ frame, err := packet.VideoFrame()if err != nil {t.Fatal(err)}diff --git a/native/sender/internal/remote/peer.go b/native/sender/internal/remote/peer.goindex e0f46cf2..acfe31fc 100644--- a/native/sender/internal/remote/peer.go+++ b/native/sender/internal/remote/peer.go@@ -97,20 +97,37 @@ func newPeer(session *Session, peerID string, slot int, configuration webrtc.Con}}})- sender, err := connection.AddTrack(session.fanout.Track())+ videoSender, err := connection.AddTrack(session.fanout.Track())if err != nil {_ = connection.Close()return nil, errors.New("bind shared encoded track failed")}+ var audioSender *webrtc.RTPSender+ if session.audioFanout != nil {+ audioSender, err = connection.AddTrack(session.audioFanout.Track())+ if err != nil {+ _ = connection.Close()+ return nil, errors.New("bind shared Opus track failed")+ }+ }if session.codec == media.CodecH264 {- transceivers := connection.GetTransceivers()- if len(transceivers) != 1 || len(session.codecPreferences) == 0 ||- transceivers[0].SetCodecPreferences(session.codecPreferences) != nil {+ var videoTransceiver *webrtc.RTPTransceiver+ for _, transceiver := range connection.GetTransceivers() {+ if transceiver.Kind() == webrtc.RTPCodecTypeVideo {+ videoTransceiver = transceiver+ break+ }+ }+ if videoTransceiver == nil || len(session.codecPreferences) == 0 ||+ videoTransceiver.SetCodecPreferences(session.codecPreferences) != nil {_ = connection.Close()return nil, errors.New("configure H.264 codec preferences failed")}}- go peer.readRTCP(sender)+ go peer.readRTCP(videoSender)+ if audioSender != nil {+ go peer.readRTCP(audioSender)+ }return peer, nil}diff --git a/native/sender/internal/remote/session.go b/native/sender/internal/remote/session.goindex af2d9404..8738ed07 100644--- a/native/sender/internal/remote/session.go+++ b/native/sender/internal/remote/session.go@@ -42,18 +42,20 @@ var ()type Event struct {- Kind string `json:"kind"`- Message string `json:"message,omitempty"`- ActiveViewers int `json:"activeViewers,omitempty"`- WaitingViewers int `json:"waitingViewers,omitempty"`- Media *media.Metrics `json:"media,omitempty"`- Peers []PeerDiagnostics `json:"peers,omitempty"`+ Kind string `json:"kind"`+ Message string `json:"message,omitempty"`+ ActiveViewers int `json:"activeViewers,omitempty"`+ WaitingViewers int `json:"waitingViewers,omitempty"`+ Media *media.Metrics `json:"media,omitempty"`+ Audio *media.AudioMetrics `json:"audio,omitempty"`+ Peers []PeerDiagnostics `json:"peers,omitempty"`}type StartOptions struct {BaseURL *url.URLHostAdmissionPassword stringCodec media.Codec+ EnableAudio boolOnEvent func(Event)}@@ -82,6 +84,7 @@ type Session struct {peers map[string]*peeradmission *viewerAdmissionfanout *media.Fanout+ audioFanout *media.AudioFanoutcodec media.CodecpeerAPI *webrtc.APIcodecPreferences []webrtc.RTPCodecParameters@@ -185,6 +188,13 @@ func Start(parent context.Context, options StartOptions) (*Session, Room, error)cancel()return nil, Room{}, err}+ if options.EnableAudio {+ session.audioFanout, err = media.NewAudioFanout(func(fatalErr error) { session.fail(fatalErr) })+ if err != nil {+ session.close()+ return nil, Room{}, err+ }+ }session.peerAPI, session.codecPreferences, err = newPeerAPI(codec)if err != nil {session.close()@@ -228,18 +238,38 @@ func (session *Session) resetSignalingConnection() {}}-func (session *Session) WriteFrame(frame media.Frame) error {- if err := session.fanout.Push(frame); err != nil {- session.fail(fmt.Errorf("write encoded frame: %w", err))- return err+func (session *Session) WriteMedia(packet media.Packet) error {+ switch packet.Kind {+ case media.KindVideo:+ frame, err := packet.VideoFrame()+ if err == nil {+ err = session.fanout.Push(frame)+ }+ if err != nil {+ session.fail(fmt.Errorf("write encoded video: %w", err))+ return err+ }+ return nil+ case media.KindOpus:+ if session.audioFanout == nil {+ return errors.New("encoded audio was not enabled for this session")+ }+ if err := session.audioFanout.Push(packet); err != nil {+ session.fail(fmt.Errorf("write encoded audio: %w", err))+ return err+ }+ return nil+ default:+ return errors.New("local sender supplied an unsupported encoded media kind")}- return nil}func (session *Session) Codec() media.Codec {return session.codec}+func (session *Session) AudioEnabled() bool { return session.audioFanout != nil }+func (session *Session) Stop() {session.close()}@@ -587,7 +617,12 @@ func (session *Session) diagnosticsLoop() {return diagnostics[left].Slot < diagnostics[right].Slot})metrics := session.fanout.Snapshot()- session.emit(Event{Kind: "diagnostics", Media: &metrics, Peers: diagnostics})+ var audio *media.AudioMetrics+ if session.audioFanout != nil {+ value := session.audioFanout.Snapshot()+ audio = &value+ }+ session.emit(Event{Kind: "diagnostics", Media: &metrics, Audio: audio, Peers: diagnostics})}}}@@ -664,6 +699,9 @@ func (session *Session) close() {session.peers = make(map[string]*peer)session.mu.Unlock()session.fanout.Close()+ if session.audioFanout != nil {+ session.audioFanout.Close()+ }for _, peer := range peers {peer.close()}diff --git a/native/sender/internal/windowaudio/provider.go b/native/sender/internal/windowaudio/provider.gonew file mode 100644index 00000000..0704e735--- /dev/null+++ b/native/sender/internal/windowaudio/provider.go@@ -0,0 +1,25 @@+package windowaudio++import (+ "context"+ "errors"+)++var ErrUnavailable = errors.New("Windows 11 process audio is unavailable")++type Target struct {+ PID uint32+ CreationTime uint64+ Title string+}++type PCMChunk struct {+ Timestamp100ns uint64+ Duration100ns uint64+ Data []byte+}++type Provider interface {+ List(context.Context) ([]Target, error)+ Capture(context.Context, Target, func(PCMChunk) error) error+}diff --git a/native/sender/internal/windowaudio/provider_other.go b/native/sender/internal/windowaudio/provider_other.gonew file mode 100644index 00000000..7fb74cf6--- /dev/null+++ b/native/sender/internal/windowaudio/provider_other.go@@ -0,0 +1,15 @@+//go:build !windows++package windowaudio++import "context"++type unsupportedProvider struct{}++func NewProvider() Provider { return unsupportedProvider{} }++func (unsupportedProvider) List(context.Context) ([]Target, error) { return nil, ErrUnavailable }++func (unsupportedProvider) Capture(context.Context, Target, func(PCMChunk) error) error {+ return ErrUnavailable+}diff --git a/native/sender/internal/windowaudio/provider_windows.go b/native/sender/internal/windowaudio/provider_windows.gonew file mode 100644index 00000000..3fa6433f--- /dev/null+++ b/native/sender/internal/windowaudio/provider_windows.go@@ -0,0 +1,145 @@+//go:build windows++package windowaudio++import (+ "context"+ "encoding/binary"+ "encoding/json"+ "errors"+ "io"+ "os"+ "os/exec"+ "path/filepath"+ "strconv"+ "strings"++ "golang.org/x/sys/windows"+)++const (+ maxListBytes = 64 << 10+ maxTargets = 100+ helperHeaderSize = 28+)++type windowsProvider struct{ helper string }++type helperTarget struct {+ PID uint32 `json:"pid"`+ CreationTime uint64 `json:"creationTime"`+ Title string `json:"title"`+}++func NewProvider() Provider {+ helper := strings.TrimSpace(os.Getenv("SCREENER_PROCESS_AUDIO_HELPER"))+ if helper == "" {+ if executable, err := os.Executable(); err == nil {+ helper = filepath.Join(filepath.Dir(executable), "screener-process-audio.exe")+ }+ }+ return &windowsProvider{helper: helper}+}++func (provider *windowsProvider) available() bool {+ version := windows.RtlGetVersion()+ if version.MajorVersion < 10 || version.BuildNumber < 22_000 || provider.helper == "" {+ return false+ }+ info, err := os.Stat(provider.helper)+ return err == nil && !info.IsDir()+}++func (provider *windowsProvider) List(ctx context.Context) ([]Target, error) {+ if !provider.available() {+ return nil, ErrUnavailable+ }+ command := exec.CommandContext(ctx, provider.helper, "--list")+ stdout, err := command.StdoutPipe()+ if err != nil {+ return nil, ErrUnavailable+ }+ command.Stderr = io.Discard+ if err = command.Start(); err != nil {+ return nil, ErrUnavailable+ }+ payload, readErr := io.ReadAll(io.LimitReader(stdout, maxListBytes+1))+ if (readErr != nil || len(payload) > maxListBytes) && ctx.Err() == nil {+ _ = command.Process.Kill()+ }+ waitErr := command.Wait()+ if readErr != nil || waitErr != nil || len(payload) > maxListBytes {+ return nil, ErrUnavailable+ }+ var listed []helperTarget+ if json.Unmarshal(payload, &listed) != nil || len(listed) > maxTargets {+ return nil, ErrUnavailable+ }+ targets := make([]Target, 0, len(listed))+ for _, candidate := range listed {+ title := strings.TrimSpace(candidate.Title)+ if candidate.PID == 0 || candidate.CreationTime == 0 || title == "" || len(title) > 512 {+ continue+ }+ targets = append(targets, Target{+ PID: candidate.PID, CreationTime: candidate.CreationTime, Title: title,+ })+ }+ return targets, nil+}++func (provider *windowsProvider) Capture(ctx context.Context, target Target, onChunk func(PCMChunk) error) error {+ if !provider.available() || target.PID == 0 || target.CreationTime == 0 || onChunk == nil {+ return ErrUnavailable+ }+ command := exec.CommandContext(ctx, provider.helper, "--capture",+ strconv.FormatUint(uint64(target.PID), 10), strconv.FormatUint(target.CreationTime, 10))+ stdout, err := command.StdoutPipe()+ if err != nil {+ return ErrUnavailable+ }+ command.Stderr = io.Discard+ if err = command.Start(); err != nil {+ return ErrUnavailable+ }+ started := false+ for {+ header := make([]byte, helperHeaderSize)+ _, readErr := io.ReadFull(stdout, header)+ if readErr != nil {+ err = readErr+ break+ }+ length := binary.BigEndian.Uint32(header[24:28])+ if string(header[:4]) != "SPCM" || header[4] != 1 || header[5] != 0 ||+ header[6] != 0 || header[7] != 0 || length == 0 || length > 48_000*2*2/10 || length%4 != 0 {+ err = errors.New("process audio helper returned an invalid PCM header")+ break+ }+ data := make([]byte, length)+ if _, readErr = io.ReadFull(stdout, data); readErr != nil {+ err = readErr+ break+ }+ started = true+ if err = onChunk(PCMChunk{+ Timestamp100ns: binary.BigEndian.Uint64(header[8:16]),+ Duration100ns: binary.BigEndian.Uint64(header[16:24]),+ Data: data,+ }); err != nil {+ _ = command.Process.Kill()+ break+ }+ }+ if err != nil && ctx.Err() == nil {+ _ = command.Process.Kill()+ }+ _ = command.Wait()+ if ctx.Err() != nil {+ return nil+ }+ if err != nil || !started {+ return errors.New("target-process audio capture stopped")+ }+ return errors.New("target process exited")+}diff --git a/native/sender/internal/windowaudio/provider_windows_test.go b/native/sender/internal/windowaudio/provider_windows_test.gonew file mode 100644index 00000000..7c7a6360--- /dev/null+++ b/native/sender/internal/windowaudio/provider_windows_test.go@@ -0,0 +1,100 @@+//go:build windows++package windowaudio++import (+ "context"+ "encoding/binary"+ "flag"+ "os"+ "testing"+ "time"+)++const testCreationTime = uint64(123456)++var (+ captureHelperPID = flag.String("capture", "", "run as the process-audio capture test helper")+ listHelper = flag.Bool("list", false, "run as the process-audio list test helper")+)++func TestMain(main *testing.M) {+ flag.Parse()+ if *listHelper {+ _, _ = os.Stdout.Write(make([]byte, maxListBytes+1))+ select {}+ }+ if *captureHelperPID != "" {+ if len(flag.Args()) != 1 || flag.Args()[0] != "123456" {+ os.Exit(3)+ }+ header := make([]byte, helperHeaderSize)+ copy(header, "SPCM")+ header[4] = 1+ binary.BigEndian.PutUint64(header[16:24], 200_000)+ binary.BigEndian.PutUint32(header[24:28], 4)+ _, _ = os.Stdout.Write(header)+ _, _ = os.Stdout.Write(make([]byte, 4))+ _, _ = os.Stdout.Write(make([]byte, helperHeaderSize))+ select {}+ }+ os.Exit(main.Run())+}++func TestCapturePassesIdentityAndKillsHelperBeforeWaitingOnInvalidHeader(t *testing.T) {+ provider := &windowsProvider{helper: os.Args[0]}+ if !provider.available() {+ t.Skip("requires Windows 11")+ }++ ctx, cancel := context.WithCancel(context.Background())+ result := make(chan error, 1)+ chunks := 0+ go func() {+ result <- provider.Capture(ctx, Target{PID: 1, CreationTime: testCreationTime}, func(PCMChunk) error {+ chunks+++ return nil+ })+ }()++ select {+ case err := <-result:+ if err == nil {+ t.Fatal("Capture returned nil after an invalid helper header")+ }+ if chunks != 1 {+ t.Fatalf("Capture delivered %d chunks; process identity was not passed to the helper", chunks)+ }+ case <-time.After(2 * time.Second):+ cancel()+ <-result+ t.Fatal("Capture waited on a running helper after an invalid header")+ }+ cancel()+}++func TestListKillsHelperBeforeWaitingOnOversizeOutput(t *testing.T) {+ provider := &windowsProvider{helper: os.Args[0]}+ if !provider.available() {+ t.Skip("requires Windows 11")+ }++ ctx, cancel := context.WithCancel(context.Background())+ result := make(chan error, 1)+ go func() {+ _, err := provider.List(ctx)+ result <- err+ }()++ select {+ case err := <-result:+ if err == nil {+ t.Fatal("List accepted oversized helper output")+ }+ case <-time.After(2 * time.Second):+ cancel()+ <-result+ t.Fatal("List waited on a running helper after oversized output")+ }+ cancel()+}diff --git a/scripts/native-one-viewer-gate.ts b/scripts/native-one-viewer-gate.tsindex 56f613b7..8515965c 100644--- a/scripts/native-one-viewer-gate.ts+++ b/scripts/native-one-viewer-gate.ts@@ -119,6 +119,9 @@ interface ViewerSnapshot {videoHeight: number;videoCurrentTime: number;renderedFrames: number;+ audioTrackCount: number;+ audioPacketsReceived: number;+ audioBytesReceived: number;}class CdpConnection {@@ -274,6 +277,7 @@ async function main(): Promise<void> {const chromePath = requiredEnvironmentPath("SCREENER_NATIVE_GATE_CHROME");const goPath = process.env.SCREENER_NATIVE_GATE_GO?.trim() || "go";const gateCodec = process.env.SCREENER_NATIVE_GATE_CODEC?.trim() === "h264" ? "h264" : "vp8";+ const audioTargetTitle = process.env.SCREENER_NATIVE_GATE_AUDIO_TARGET_TITLE?.trim() || "";const accessKey = randomBytes(9).toString("base64url");const report: GateReport = {schemaVersion: 1,@@ -283,7 +287,9 @@ async function main(): Promise<void> {limitations: ["The source is a real animated Chrome tab captured through getDisplayMedia, not a game workload.","This run proves one WebCodecs object, not one physical or hardware encoder.",- "The loopback run does not prove public STUN, restrictive networks, TURN, audio, packaging, or endurance.",+ audioTargetTitle+ ? "The loopback run does not prove public STUN, restrictive networks, TURN, packaging, or endurance."+ : "The loopback run does not prove public STUN, restrictive networks, TURN, audio, packaging, or endurance.",],};let server: ScreenerServer | null = null;@@ -366,10 +372,22 @@ async function main(): Promise<void> {await createPage(cdp, animatedSourceUrl());const activeSenderPage = await createPage(cdp, launchUrl, senderProbe());senderPage = activeSenderPage;+ let audioTargetId = "";+ if (audioTargetTitle) {+ await waitForSample(+ (deadline) => evaluate<string>(cdp!, activeSenderPage,+ `([...document.querySelector('#audio-target').options].find((option) => option.text.includes(${JSON.stringify(audioTargetTitle)}))?.value || '')`, deadline),+ Boolean,+ 5_000,+ );+ audioTargetId = await evaluate<string>(cdp, activeSenderPage,+ `([...document.querySelector('#audio-target').options].find((option) => option.text.includes(${JSON.stringify(audioTargetTitle)}))?.value || '')`, Date.now() + 2_000);+ }await evaluate<void>(cdp, activeSenderPage, `(() => {document.querySelector('#server-url').value = ${JSON.stringify(baseUrl)};document.querySelector('#password').value = ${JSON.stringify(accessKey)};document.querySelector('#codec').value = ${JSON.stringify(gateCodec)};+ document.querySelector('#audio-target').value = ${JSON.stringify(audioTargetId)};document.querySelector('#start').click();})()`, Date.now() + 5_000);await waitForSenderStart(@@ -464,6 +482,10 @@ async function main(): Promise<void> {sameBridge: retainsFirstBridgeSend(sender),oneEncoderObject: sender.encoderInstances === 1,criticalErrors: sender.fatalEvents === 0 && sender.encoderErrors === 0,+ ...(audioTargetTitle ? {+ audioTrack: viewer.audioTrackCount === 1,+ audioInbound: viewer.audioPacketsReceived > beforeViewer.audioPacketsReceived,+ } : {}),};report.stages["viewer-media"] = {...checks,@@ -471,6 +493,7 @@ async function main(): Promise<void> {byteDelta: viewer.bytesReceived - beforeViewer.bytesReceived,decodedDelta: viewer.framesDecoded - beforeViewer.framesDecoded,renderedDelta: viewer.renderedFrames - beforeViewer.renderedFrames,+ audioPacketDelta: viewer.audioPacketsReceived - beforeViewer.audioPacketsReceived,pionPacketDelta: (sender.peers[0]?.packetsSent ?? 0) - (beforeSender.peers[0]?.packetsSent ?? 0),pcOrdinal: viewer.identity.pcOrdinal,pionSlot: sender.peers[0]?.slot ?? -1,@@ -648,6 +671,7 @@ function viewerProbe(): string {consistent: true, pendingOffer: null,pcs: [], boundPcOrdinal: 0, boundConnectionOrdinal: 0, boundSocketOrdinal: 0,boundAuthGeneration: 0, trackCount: 0, trackOrdinal: 0, trackPcOrdinal: 0,+ audioTrackCount: 0,renderedFrames: 0, observedVideo: null };const connectionOrdinal = (value) => {let index = state.connectionIds.indexOf(value);@@ -733,6 +757,7 @@ function viewerProbe(): string {state.pcs.push(this);this.__gatePcOrdinal = state.pcs.length;this.addEventListener('track', (event) => {+ if (event.track?.kind === 'audio') { state.audioTrackCount += 1; return; }if (event.track?.kind !== 'video') return;state.trackCount += 1; state.trackOrdinal = state.trackCount;state.trackPcOrdinal = this.__gatePcOrdinal;@@ -762,6 +787,7 @@ function viewerProbe(): string {}const pc = state.pcs[state.boundPcOrdinal - 1];let packetsReceived = 0, bytesReceived = 0, framesDecoded = 0;+ let audioPacketsReceived = 0, audioBytesReceived = 0;if (pc) {const stats = await pc.getStats();stats.forEach((entry) => {@@ -770,6 +796,10 @@ function viewerProbe(): string {bytesReceived += finite(entry.bytesReceived);framesDecoded += finite(entry.framesDecoded);}+ if (entry.type === 'inbound-rtp' && entry.kind === 'audio' && entry.isRemote !== true) {+ audioPacketsReceived += finite(entry.packetsReceived);+ audioBytesReceived += finite(entry.bytesReceived);+ }});}const activeOrdinal = state.activeConnectionId === null ? 0 :@@ -787,7 +817,8 @@ function viewerProbe(): string {iceConnectionState: pc?.iceConnectionState || 'none', packetsReceived, bytesReceived, framesDecoded,videoReadyState: finite(video?.readyState), videoWidth: finite(video?.videoWidth),videoHeight: finite(video?.videoHeight), videoCurrentTime: finite(video?.currentTime),- renderedFrames: state.renderedFrames };+ renderedFrames: state.renderedFrames, audioTrackCount: state.audioTrackCount,+ audioPacketsReceived, audioBytesReceived };}const finite = (value) => Number.isFinite(value) ? value : 0;Object.defineProperty(globalThis, '__NATIVE_VIEWER_GATE__', {