Free, open-source screen sharing for private live streams with friends. Watch together in a browser or self-host Piik. 免费开源的私密屏幕共享,支持游戏直播、一起看电影与自部署。
docs(audio): scope native Windows audio slice
AAA7949efac2d54
diff --git a/docs/adr/0008-window-scoped-audio-capture.md b/docs/adr/0008-window-scoped-audio-capture.mdindex bf6973f5..51396f39 100644--- a/docs/adr/0008-window-scoped-audio-capture.md+++ b/docs/adr/0008-window-scoped-audio-capture.md@@ -16,9 +16,9 @@ application is open can leak that conversation back to Viewers.Windows has a stricter primitive: WASAPI application loopback through`ActivateAudioInterfaceAsync` with`AUDIOCLIENT_PROCESS_LOOPBACK_PARAMS.INCLUDE_TARGET_PROCESS_TREE`. It can be-paired with a selected window's PID and a WGC/DXGI video capture. The API is-available on Windows 11 build 20348 and later; this is not a Windows 10-fallback.+paired with a selected window's PID and a WGC/DXGI video capture. Microsoft's+process-loopback contract starts at build 20348; this product slice gates on+Windows 11 desktop (build 22000 or newer) and is not a Windows 10 fallback.## Decision@@ -60,6 +60,57 @@ independent voice process, a notification, process restart, no render stream,and A/V sync on one Viewer. It should use Microsoft's Application Loopbacksample and process-loopback parameter contract rather than a custom mixer.+## P1 First Slice: Audio Only++The first product-wiring slice is deliberately smaller than the complete+native capture boundary. It adds an explicit Windows 11+`window-process-audio` input while leaving the current Web video capture and+codec path unchanged. A small Windows helper resolves the selected target PID+locally and sends timestamped PCM through the authenticated loopback bridge.+The existing Chrome/Edge sender UI uses `AudioEncoder` with `codec: "opus"`,+48 kHz, two channels, and 20 ms frames, then returns the encoded Opus packets to+the existing Go sender. This reuses the current browser prerequisite and avoids+adding a native Opus library or a second package toolchain in this slice.++The bridge is one typed local media envelope (audio kind, flags, QPC-derived+timestamp, duration, bounded payload); it does not add a public signaling field.+The Go session owns one shared Opus `TrackLocalStaticRTP` and adds it before the+offer on every native PeerConnection. Audio packetization uses the 48 kHz RTP+clock and the source timestamp delta, so one encoded packet stream can feed the+existing two-edge cap. The WGC/DXGI video adapter remains the next small slice;+its `SystemRelativeTime` is retained in the fixture so it can share the same+QPC origin when native video replaces the Web source.++Expected implementation size is about **500--800 new LOC**, excluding the+browser/Windows SDK and existing video/MF fixture code: Windows loopback+capture and local IPC (250--400), Go audio envelope/timeline/track (150--250),+and UI lifecycle/status plus focused tests (100--150). This is a planned P1+input, not current product behavior or a deployment switch.++The two source clocks are both converted from 100-ns QPC: WGC+`SystemRelativeTime` and WASAPI `IAudioCaptureClient::GetBuffer`'s+`pu64QPCPosition`. Keep timestamps monotonic and derive audio duration from+the captured frame count (960 samples at 48 kHz for each 20 ms packet), not+from bridge arrival time. `TrackLocalStaticSample.Timestamp` is not the source+timeline control in the current Pion path; the audio fanout therefore uses the+same explicit RTP packetizer/timeline pattern as native video.++The slice fails closed. Windows 10, unsupported WGC, picker cancellation,+activation denial, an exited/inaccessible target, `AUDCLNT_E_DEVICE_INVALIDATED`+or `AUDCLNT_E_SERVICE_NOT_RUNNING`, protected-content silence, and a missing+render stream produce `audio unavailable` or `audio silent`. There is no retry+to whole-system loopback. No administrator privilege is required, but the+helper must run in the interactive user's session and honor Windows consent and+privacy settings. PID, title, path, device identity, and PCM remain local.++The first smoke is one direct Viewer: a test window emits a known tone and+visual marker, while an independent voice process and notification emit+different markers. Require an Opus inbound track and rendered video/audio,+monotonic QPC timestamps, and a bounded A/V offset; verify the unrelated+markers are absent and that target exit/silence stops or asks without widening+capture. SFU/TURN, second Viewer, endurance, and the native WGC video swap are+follow-up gates.+## Consequences- Current Web users get a useful Chromium hint and an honest status without a@@ -71,9 +122,10 @@ sample and process-loopback parameter contract rather than a custom mixer.## Follow-Up TODO-- Add a Windows-only WGC + WASAPI process-loopback fixture beside the existing- Media Foundation H.264 fixture.+- Add the Windows-only WGC + WASAPI process-loopback fixture beside the existing+ Media Foundation H.264 fixture, then land the bounded audio-only P1 slice.- Feed its timestamped PCM/video pair into the existing native sender only- after the isolation and sync checks pass.+ after the isolation and sync checks pass; native WGC video replacement stays+ a separate follow-up.- Retain a short matrix result in `docs/research/`; do not enable the mode bydefault or deploy it from this ADR alone.diff --git a/docs/research/browser-screen-audio-quality.md b/docs/research/browser-screen-audio-quality.mdindex 17e70de2..93452576 100644--- a/docs/research/browser-screen-audio-quality.md+++ b/docs/research/browser-screen-audio-quality.md@@ -180,6 +180,34 @@ classifiable error; the sender reports silence or unknown and never bypassesprotection. PID, path, window title, device identity, and raw audio remain localand are not retained or uploaded.+### P1 Audio-Only Wiring Slice++The first native implementation is intentionally an audio-only input and is+estimated at 500--800 new lines. It keeps the current Web video capture and+codec path, while a Windows 11 helper supplies process-tree PCM through the+existing authenticated local bridge. Chrome/Edge's already-required WebCodecs+surface encodes that PCM with `AudioEncoder` (`opus`, 48 kHz, two channels,+20 ms), so the helper does not add a native Opus dependency or a second package+toolchain. The Go sender packetizes the returned Opus access units once into a+shared `audio/opus` RTP track before adding it to each native PeerConnection.++The local envelope carries only media kind, flags, QPC-derived timestamp,+duration, and bounded payload; no PID or source label crosses the bridge. WGC+video remains a fixture/next adapter in this slice. Its+`SystemRelativeTime`, like WASAPI `GetBuffer`'s `pu64QPCPosition`, is a 100-ns+QPC value and is retained as the future A/V synchronization origin. Audio+timestamps are converted to the 48 kHz RTP clock and remain monotonic; packet+duration comes from the captured frame count rather than arrival time.++This is an opt-in P1 candidate, not a default or production switch. Windows 10,+picker/activation denial, an exited or inaccessible target, no render stream,+protected-content silence, and WASAPI device/service errors report+`unavailable`/`silent` and stop or ask. The implementation never widens to+whole-system loopback. The first acceptance smoke is one direct Viewer with a+known target tone/visual marker and independent voice/notification markers;+SFU/TURN, a second Viewer, endurance, and native WGC video replacement remain+later gates.+Run one bounded matrix rather than a full route Cartesian product: exactproduction and current `main` on Windows Chrome/Edge for tab/window/monitor,audio selected/unselected, and a simultaneous voice call; then the native@@ -241,6 +269,7 @@ Add a user-facing audio setting only when all of these are true:- [W3C WebRTC](https://www.w3.org/TR/webrtc/)- [W3C WebRTC Statistics](https://www.w3.org/TR/webrtc-stats/)- [W3C Web Audio](https://www.w3.org/TR/webaudio-1.1/)+- [W3C WebCodecs Opus registration](https://www.w3.org/TR/webcodecs-opus-codec-registration/)- [RFC 7587: RTP Payload Format for Opus](https://www.rfc-editor.org/rfc/rfc7587.html)- [RFC 7874: WebRTC Audio Codec and Processing Requirements](https://www.rfc-editor.org/rfc/rfc7874.html)- [MDN `getDisplayMedia()`](https://developer.mozilla.org/en-US/docs/Web/API/MediaDevices/getDisplayMedia)@@ -251,7 +280,10 @@ Add a user-facing audio setting only when all of these are true:- [Chrome 141 window-audio release note](https://developer.chrome.com/release-notes/141)- [Chromium media-capture architecture index](https://chromium.googlesource.com/chromium/src/+/HEAD/docs/media/capture/)- [Microsoft Application Loopback sample](https://learn.microsoft.com/en-us/samples/microsoft/windows-classic-samples/applicationloopbackaudio-sample/)+- [Microsoft `ActivateAudioInterfaceAsync`](https://learn.microsoft.com/en-us/windows/win32/api/mmdeviceapi/nf-mmdeviceapi-activateaudiointerfaceasync)- [Windows process-loopback parameters](https://learn.microsoft.com/en-us/windows/win32/api/audioclientactivationparams/ns-audioclientactivationparams-audioclient_process_loopback_params)+- [Microsoft Windows Graphics Capture](https://learn.microsoft.com/en-us/windows/apps/develop/media-authoring-processing/screen-capture)+- [Microsoft `IAudioCaptureClient::GetBuffer`](https://learn.microsoft.com/en-us/windows/win32/api/audioclient/nf-audioclient-iaudiocaptureclient-getbuffer)- [Windows 10 release information](https://learn.microsoft.com/en-us/windows/release-health/release-information)- [Microsoft WASAPI loopback recording](https://learn.microsoft.com/en-us/windows/win32/coreaudio/loopback-recording)- [Android capture video and audio playback](https://developer.android.com/media/platform/av-capture)