Skip to content

armgit / Piik

Public Imported from GitHub
0

Free, open-source screen sharing for private live streams with friends. Watch together in a browser or self-host Piik. 免费开源的私密屏幕共享,支持游戏直播、一起看电影与自部署。

docs(audio): scope native Windows audio slice

AAA7949efac2d54
diff --git a/docs/adr/0008-window-scoped-audio-capture.md b/docs/adr/0008-window-scoped-audio-capture.md
index bf6973f5..51396f39 100644
--- a/docs/adr/0008-window-scoped-audio-capture.md
+++ b/docs/adr/0008-window-scoped-audio-capture.md
@@ -16,9 +16,9 @@ application is open can leak that conversation back to Viewers.
Windows has a stricter primitive: WASAPI application loopback through
`ActivateAudioInterfaceAsync` with
`AUDIOCLIENT_PROCESS_LOOPBACK_PARAMS.INCLUDE_TARGET_PROCESS_TREE`. It can be
-paired with a selected window's PID and a WGC/DXGI video capture. The API is
-available on Windows 11 build 20348 and later; this is not a Windows 10
-fallback.
+paired with a selected window's PID and a WGC/DXGI video capture. Microsoft's
+process-loopback contract starts at build 20348; this product slice gates on
+Windows 11 desktop (build 22000 or newer) and is not a Windows 10 fallback.
## Decision
@@ -60,6 +60,57 @@ independent voice process, a notification, process restart, no render stream,
and A/V sync on one Viewer. It should use Microsoft's Application Loopback
sample and process-loopback parameter contract rather than a custom mixer.
+## P1 First Slice: Audio Only
+
+The first product-wiring slice is deliberately smaller than the complete
+native capture boundary. It adds an explicit Windows 11
+`window-process-audio` input while leaving the current Web video capture and
+codec path unchanged. A small Windows helper resolves the selected target PID
+locally and sends timestamped PCM through the authenticated loopback bridge.
+The existing Chrome/Edge sender UI uses `AudioEncoder` with `codec: "opus"`,
+48 kHz, two channels, and 20 ms frames, then returns the encoded Opus packets to
+the existing Go sender. This reuses the current browser prerequisite and avoids
+adding a native Opus library or a second package toolchain in this slice.
+
+The bridge is one typed local media envelope (audio kind, flags, QPC-derived
+timestamp, duration, bounded payload); it does not add a public signaling field.
+The Go session owns one shared Opus `TrackLocalStaticRTP` and adds it before the
+offer on every native PeerConnection. Audio packetization uses the 48 kHz RTP
+clock and the source timestamp delta, so one encoded packet stream can feed the
+existing two-edge cap. The WGC/DXGI video adapter remains the next small slice;
+its `SystemRelativeTime` is retained in the fixture so it can share the same
+QPC origin when native video replaces the Web source.
+
+Expected implementation size is about **500--800 new LOC**, excluding the
+browser/Windows SDK and existing video/MF fixture code: Windows loopback
+capture and local IPC (250--400), Go audio envelope/timeline/track (150--250),
+and UI lifecycle/status plus focused tests (100--150). This is a planned P1
+input, not current product behavior or a deployment switch.
+
+The two source clocks are both converted from 100-ns QPC: WGC
+`SystemRelativeTime` and WASAPI `IAudioCaptureClient::GetBuffer`'s
+`pu64QPCPosition`. Keep timestamps monotonic and derive audio duration from
+the captured frame count (960 samples at 48 kHz for each 20 ms packet), not
+from bridge arrival time. `TrackLocalStaticSample.Timestamp` is not the source
+timeline control in the current Pion path; the audio fanout therefore uses the
+same explicit RTP packetizer/timeline pattern as native video.
+
+The slice fails closed. Windows 10, unsupported WGC, picker cancellation,
+activation denial, an exited/inaccessible target, `AUDCLNT_E_DEVICE_INVALIDATED`
+or `AUDCLNT_E_SERVICE_NOT_RUNNING`, protected-content silence, and a missing
+render stream produce `audio unavailable` or `audio silent`. There is no retry
+to whole-system loopback. No administrator privilege is required, but the
+helper must run in the interactive user's session and honor Windows consent and
+privacy settings. PID, title, path, device identity, and PCM remain local.
+
+The first smoke is one direct Viewer: a test window emits a known tone and
+visual marker, while an independent voice process and notification emit
+different markers. Require an Opus inbound track and rendered video/audio,
+monotonic QPC timestamps, and a bounded A/V offset; verify the unrelated
+markers are absent and that target exit/silence stops or asks without widening
+capture. SFU/TURN, second Viewer, endurance, and the native WGC video swap are
+follow-up gates.
+
## Consequences
- Current Web users get a useful Chromium hint and an honest status without a
@@ -71,9 +122,10 @@ sample and process-loopback parameter contract rather than a custom mixer.
## Follow-Up TODO
-- Add a Windows-only WGC + WASAPI process-loopback fixture beside the existing
- Media Foundation H.264 fixture.
+- Add the Windows-only WGC + WASAPI process-loopback fixture beside the existing
+ Media Foundation H.264 fixture, then land the bounded audio-only P1 slice.
- Feed its timestamped PCM/video pair into the existing native sender only
- after the isolation and sync checks pass.
+ after the isolation and sync checks pass; native WGC video replacement stays
+ a separate follow-up.
- Retain a short matrix result in `docs/research/`; do not enable the mode by
default or deploy it from this ADR alone.
diff --git a/docs/research/browser-screen-audio-quality.md b/docs/research/browser-screen-audio-quality.md
index 17e70de2..93452576 100644
--- a/docs/research/browser-screen-audio-quality.md
+++ b/docs/research/browser-screen-audio-quality.md
@@ -180,6 +180,34 @@ classifiable error; the sender reports silence or unknown and never bypasses
protection. PID, path, window title, device identity, and raw audio remain local
and are not retained or uploaded.
+### P1 Audio-Only Wiring Slice
+
+The first native implementation is intentionally an audio-only input and is
+estimated at 500--800 new lines. It keeps the current Web video capture and
+codec path, while a Windows 11 helper supplies process-tree PCM through the
+existing authenticated local bridge. Chrome/Edge's already-required WebCodecs
+surface encodes that PCM with `AudioEncoder` (`opus`, 48 kHz, two channels,
+20 ms), so the helper does not add a native Opus dependency or a second package
+toolchain. The Go sender packetizes the returned Opus access units once into a
+shared `audio/opus` RTP track before adding it to each native PeerConnection.
+
+The local envelope carries only media kind, flags, QPC-derived timestamp,
+duration, and bounded payload; no PID or source label crosses the bridge. WGC
+video remains a fixture/next adapter in this slice. Its
+`SystemRelativeTime`, like WASAPI `GetBuffer`'s `pu64QPCPosition`, is a 100-ns
+QPC value and is retained as the future A/V synchronization origin. Audio
+timestamps are converted to the 48 kHz RTP clock and remain monotonic; packet
+duration comes from the captured frame count rather than arrival time.
+
+This is an opt-in P1 candidate, not a default or production switch. Windows 10,
+picker/activation denial, an exited or inaccessible target, no render stream,
+protected-content silence, and WASAPI device/service errors report
+`unavailable`/`silent` and stop or ask. The implementation never widens to
+whole-system loopback. The first acceptance smoke is one direct Viewer with a
+known target tone/visual marker and independent voice/notification markers;
+SFU/TURN, a second Viewer, endurance, and native WGC video replacement remain
+later gates.
+
Run one bounded matrix rather than a full route Cartesian product: exact
production and current `main` on Windows Chrome/Edge for tab/window/monitor,
audio selected/unselected, and a simultaneous voice call; then the native
@@ -241,6 +269,7 @@ Add a user-facing audio setting only when all of these are true:
- [W3C WebRTC](https://www.w3.org/TR/webrtc/)
- [W3C WebRTC Statistics](https://www.w3.org/TR/webrtc-stats/)
- [W3C Web Audio](https://www.w3.org/TR/webaudio-1.1/)
+- [W3C WebCodecs Opus registration](https://www.w3.org/TR/webcodecs-opus-codec-registration/)
- [RFC 7587: RTP Payload Format for Opus](https://www.rfc-editor.org/rfc/rfc7587.html)
- [RFC 7874: WebRTC Audio Codec and Processing Requirements](https://www.rfc-editor.org/rfc/rfc7874.html)
- [MDN `getDisplayMedia()`](https://developer.mozilla.org/en-US/docs/Web/API/MediaDevices/getDisplayMedia)
@@ -251,7 +280,10 @@ Add a user-facing audio setting only when all of these are true:
- [Chrome 141 window-audio release note](https://developer.chrome.com/release-notes/141)
- [Chromium media-capture architecture index](https://chromium.googlesource.com/chromium/src/+/HEAD/docs/media/capture/)
- [Microsoft Application Loopback sample](https://learn.microsoft.com/en-us/samples/microsoft/windows-classic-samples/applicationloopbackaudio-sample/)
+- [Microsoft `ActivateAudioInterfaceAsync`](https://learn.microsoft.com/en-us/windows/win32/api/mmdeviceapi/nf-mmdeviceapi-activateaudiointerfaceasync)
- [Windows process-loopback parameters](https://learn.microsoft.com/en-us/windows/win32/api/audioclientactivationparams/ns-audioclientactivationparams-audioclient_process_loopback_params)
+- [Microsoft Windows Graphics Capture](https://learn.microsoft.com/en-us/windows/apps/develop/media-authoring-processing/screen-capture)
+- [Microsoft `IAudioCaptureClient::GetBuffer`](https://learn.microsoft.com/en-us/windows/win32/api/audioclient/nf-audioclient-iaudiocaptureclient-getbuffer)
- [Windows 10 release information](https://learn.microsoft.com/en-us/windows/release-health/release-information)
- [Microsoft WASAPI loopback recording](https://learn.microsoft.com/en-us/windows/win32/coreaudio/loopback-recording)
- [Android capture video and audio playback](https://developer.android.com/media/platform/av-capture)