Free, open-source screen sharing for private live streams with friends. Watch together in a browser or self-host Piik. 免费开源的私密屏幕共享,支持游戏直播、一起看电影与自部署。
feat(native): add WGC hardware H264 source (#109)
AAAa560f54bcc3c
diff --git a/.github/workflows/native-sender-package.yml b/.github/workflows/native-sender-package.ymlindex 69af246a..2dadd58c 100644--- a/.github/workflows/native-sender-package.yml+++ b/.github/workflows/native-sender-package.yml@@ -5,7 +5,7 @@ on:pull_request:paths:- "native/sender/**"- - "native/process-audio-helper/**"+ - "native/window-capture-helper/**"- "scripts/package-native-sender.ps1"- ".github/workflows/native-sender-package.yml"push:@@ -13,7 +13,7 @@ on:- mainpaths:- "native/sender/**"- - "native/process-audio-helper/**"+ - "native/window-capture-helper/**"- "scripts/package-native-sender.ps1"- ".github/workflows/native-sender-package.yml"diff --git a/docs/adr/0008-window-scoped-audio-capture.md b/docs/adr/0008-window-scoped-audio-capture.mdindex e089d448..a344e67c 100644--- a/docs/adr/0008-window-scoped-audio-capture.md+++ b/docs/adr/0008-window-scoped-audio-capture.md@@ -2,9 +2,10 @@Date: 2026-08-21-Status: Accepted. The browser hint, opt-in Native Windows P1 slice, and bounded-Windows x64 evaluation-package path are source-complete; Native remains-default-off, formally unreleased, and undeployed.+Status: Accepted. The browser hint and opt-in Native Windows audio slice are+source-complete, including the bounded Windows x64 evaluation-package path.+The paired WGC/MF hardware-video source passed its one-Viewer acceptance smoke;+all Native sources remain default-off, formally unreleased, and undeployed.## Context@@ -33,12 +34,12 @@ Windows 11 desktop (build 22000 or newer) and is not a Windows 10 fallback.2. The default Web route remains unchanged in topology, codecs, and relaybehavior. No Web Audio mixer, SDP rewrite, or app-owned audio bitrate knobis introduced by this ADR.-3. The native sender's next audio slice is an explicit opt-in- `window-process-audio` mode on Windows 11. The local capture session owns- one selected target PID/process tree, one WGC/DXGI video source, and one- WASAPI loopback audio source. Process IDs, creation tokens, titles, paths,- device identity, and PCM stay local; they do not enter signaling, URLs,- logs, or storage.+3. The native sender has an explicit Windows 11 local window target. Browser+ VP8/H.264 may use that target only for `window-process-audio`; the separate+ `native-window-h264` source uses the same bound HWND/PID/creation time for+ WGC/DXGI video and WASAPI process-tree audio. Process IDs, HWNDs, creation+ tokens, titles, paths, device identity, PCM, and local hardware status stay+ local; they do not enter signaling, URLs, logs, or storage.4. If the target process exits, produces no render stream, or the loopbackactivation is denied, the sender reports `audio unavailable` or `audiosilent` and stops/asks the user. It never widens to whole-system audio.@@ -50,7 +51,7 @@ Windows 11 desktop (build 22000 or newer) and is not a Windows 10 fallback.This is an interface boundary, not a new framework or a current wire change:```text-NativeCaptureTarget { pid: uint32, creationTime: uint64, includeProcessTree: true }+NativeCaptureTarget { hwnd: uint64, pid: uint32, creationTime: uint64, includeProcessTree: true }NativeCaptureSession.start(target)-> VideoFrames(WGC/DXGI) + AudioPcm(WASAPI process loopback)-> existing native sender timeline/encoder/fanout@@ -80,9 +81,9 @@ timestamp, duration, bounded payload); it does not add a public signaling field.The Go session owns one shared Opus `TrackLocalStaticRTP` and adds it before theoffer on every native PeerConnection. Audio packetization uses the 48 kHz RTPclock and the source timestamp delta, so one encoded packet stream can feed the-existing two-edge cap. The WGC/DXGI video adapter remains the next small slice;-its `SystemRelativeTime` is retained in the fixture so it can share the same-QPC origin when native video replaces the Web source.+existing two-edge cap. In this historical audio-only phase, the WGC/DXGI video+adapter was the next small slice; its retained `SystemRelativeTime` let the+completed native-video path share the same QPC origin.Expected implementation size is about **500--800 new LOC**, excluding thebrowser/Windows SDK and existing video/MF fixture code: Windows loopback@@ -112,11 +113,29 @@ visual marker, while an independent voice process and notification emitdifferent markers. Require an Opus inbound track and rendered video/audio,monotonic QPC timestamps, and a bounded A/V offset; verify the unrelatedmarkers are absent and that target exit/silence stops or asks without widening-capture. SFU/TURN, second Viewer, endurance, and the native WGC video swap are-follow-up gates. On 2026-08-21, 100 20-ms chunks isolated a 440 Hz target from+capture. SFU/TURN, second Viewer, and endurance remain follow-up gates; paired+WGC video completion is recorded below. On 2026-08-21, 100 20-ms chunks isolated a 440 Hz target froman independent 880 Hz process by 4017.8x. Chrome 151 then received one audiotrack and 495 inbound Opus packets while video decoded/rendered 296 frames.+## P1 WGC/MF Completion++The paired native-video source reuses the proven hardware fixture rather than+adding a second encoder framework. WGC creates a capture item directly from the+bound HWND on the selected D3D11 device. The frame pool is free-threaded and is+recreated on content-size changes. D3D11 VideoProcessor performs aspect-fit+BGRA-to-NV12 conversion at fixed 1280x720/30, then the existing adapter-bound,+hardware-only asynchronous MF H.264 path emits Annex-B `42c01f` access units.+There is no software MFT, alternate codec, monitor, or browser-video fallback.++The helper multiplexes process-tree PCM, H.264 access units, and bounded local+status. Go writes H.264 directly to the existing shared fanout; the browser UI+receives only a one-way local copy for `VideoDecoder` preview. PCM retains the+existing WebCodecs Opus bridge. PLI/FIR recovery is generation-bound and+coalesced into a helper key-frame command. Normal browser VP8/H.264 sources+remain explicit alternatives, with VP8 as the default; neither is an implicit+runtime downgrade.+## Consequences- Current Web users get a useful Chromium hint and an honest status without a@@ -131,8 +150,8 @@ track and 495 inbound Opus packets while video decoded/rendered 296 frames.## Follow-Up TODO+- Keep a second Viewer, SFU/TURN, endurance, and real-game A/V sync as separate+ follow-ups; the one-Viewer H.264 and attributed `VideoEncode` gate passed.- Download and run the short-lived Windows x64 evaluation artifact; keepexplicit target selection default-off. A formal release remains blocked onthe project license, signing, and its own acceptance boundary.-- Keep native WGC video, a second Viewer, SFU/TURN, endurance, and real-game- A/V sync as separate follow-ups.diff --git a/docs/project-memory.md b/docs/project-memory.mdindex 388f6458..3565c421 100644--- a/docs/project-memory.md+++ b/docs/project-memory.md@@ -5,7 +5,7 @@ Last updated: 2026-08-21## Confirmed Intent- Build low-latency game sharing for one broadcaster and trusted friends; public or large broadcasts belong on OBS/Twitch-class services.-- Viewers join from normal desktop/mobile browsers. A later native sender may share encoding and improve capture/audio without changing the Web viewer requirement.+- Viewers join from normal desktop/mobile browsers. Native senders may improve capture/audio without changing the Web viewer requirement.- Use a small central service for access, rooms, signaling, deterministic topology, STUN, observability, and bounded SFU-root capacity.- Minimize server bandwidth. Ordinary peer ICE is STUN-only. Failure tries restart, rebuild, alternate peer and bounded SFU/UDP roots; only then may the controller authorize short-lived TURN for one selected exceptional edge before clear failure. TURN is transport, not topology.- Non-server nodes have at most two downstream edges; browser relays stay at one until resource gates pass. SFU normally feeds one or two roots that retain peer descendants; separately capped server edges may serve exceptional viewers that cannot attach behind a healthy root.@@ -23,7 +23,7 @@ Last updated: 2026-08-21- Use direct host P2P for one or two viewers, then the bounded controller for every room when `PEER_ASSISTED_MEDIA=true`. Room `1` is historical smoke, not a runtime gate; resource, quality, recovery, SFU/UDP, bounded-failure, and browser/mobile evidence remain separate.- Browser relays resend remote `MediaStreamTrack` values and re-encode at each hop; WebRTC does not guarantee a shared encoder across peer connections, so measure the cost.- Keep experiments bounded: the standard representation sequence is below; native RTP relay, encoded-object striping, and FEC remain separate.-- ADR-0006 remains no-go. VP8 is default; H.264 loopback rendered 298/299. Windows x64 evaluation packaging is source-complete; downloaded use, Pion timing, hardware, multi-viewer/endurance/public proof remain open.+- ADR-0006's historical browser-bridge canary remains no-go; VP8 stays default. Browser H.264 rendered 298/299; Native WGC/MF rendered 203 with PID/LUID-correlated `VideoEncode`. Downloaded use, multi-viewer/endurance/public proof remain open.- Mobile Web Host unsupported. Android 14+ sender is direct-child video-only; device, audio, rotation, and SFU remain open. iOS deferred; TV output is local-only P2.- ADR-0005 accepts direct/peer UDP, bounded SFU roots, then optional selected-edge TURN. Production now enables the one-shot selected-edge tuple after SFU; participant-wide TURN remains rejected and ordinary peer ICE stays STUN-only. No real TURN/SFU media canary has yet been run.- The controller is process-enabled, not room-allowlisted: `PEER_ASSISTED_MEDIA=true` gives every normal room the same direct/peer -> SFU -> selected-edge path with per-room state. Room `1` is historical smoke. Preserve sticky P2P, mobile leaves and break-before-make; recovery gets one attempt per layer (ICE restart, same-parent rebuild, alternate peer, then SFU), never three identical retries.@@ -34,7 +34,6 @@ Last updated: 2026-08-21- Viewer-local names and the opt-in Web participant roster remain control-plane-only. Web Host names are deployed in the current release and stay socket/localStorage-only; Native remains outside the capability boundary.- Do not add scene detection, dynamic-FPS control, or forced AV1 without negotiation, encode, game, CPU/GPU, and sender evidence.- ADR-0002 grant/public access, the v3 room-password migration, and Web Host display names are deployed. Names/presence remain session-only without account or roster tables.-- Deferred architecture audit: `docs/maintenance.md`.## Current Execution Principle@@ -53,7 +52,7 @@ Last updated: 2026-08-21- Chrome 151 synthetic topology/quality runs kept fanout 2/1 and decoding; one relay close recovered in 5.32 seconds. Control evidence only.- Room `1` publishes `HIGH+LOW` with Dynacast off and subscriber `HIGH` ceilings, and now has bounded local quality reparenting. Both remain unverified on real media; test zero-child leaves and calibrated C+B loss before broad rollout.- C+B uses three hard-bad pairs, one-use samples and guarded intent/cooldown. Deployed admission rescue moves the oldest childless zero-capacity Host leaf below an unassigned one-slot relay in one revision; no score/timer/global rebalance. Deployment passed, but no real room triggered it.-- Web Host name/presence and window-audio hint are deployed. Native source accepts authenticated peer-assisted direct-child assignments, ignores presence as an edge authority, and reports unsupported SFU/selected ingress as bounded route failure; it does not implement LiveKit or TURN media. Native H.264 source and Windows x64 evaluation packaging are present, but no Native release/deployment exists; VP8 remains the Web/production default. See `docs/research/native-h264-opt-in-path.md`.+- Web Host name/presence and window-audio hint are deployed. Native accepts bounded peer-assisted direct children and fails unsupported SFU/selected ingress; it has no LiveKit/TURN media. Its WGC/MF H.264 direct-child smoke rendered 203 frames plus 500 Opus packets and matched helper PID/LUID `VideoEncode`; packaging is source-only, unreleased and undeployed. VP8 remains the Web/production default. See `docs/research/native-h264-opt-in-path.md`.## Provisional Quality Targets@@ -69,7 +68,7 @@ These are measurement gates, not performance claims.- Sustainable count by hardware, quality, network, and route: finish instrumented `1/3/5/8`, then pass a 20-viewer matrix before changing the accepted target default to 20.- Whether ADR-0004 passes fanout, re-encoding, depth-four latency, reparenting, silent-partition, and mobile-leaf gates.- Whether source-complete selected-edge passes forced relay, expiry/failure, mobile, 1-GiB resource and relay-bandwidth gates. Coturn's bearer is non-revocable until expiry; application guards and TTL only bound reuse. Performance comparison and the retained matrix follow functional landing; Media TCP is out of scope.-- Exact mobile lifecycle behavior and whether Windows per-application audio is required for the first release.+- Exact mobile lifecycle behavior.- Project license and distribution model, which determines whether GPL/AGPL sources can move beyond study-only use.- Initial deployment regions and expected mainland China, Hong Kong, and overseas network mix.- Whether voice chat ever enters scope or Screener stays complementary to an existing voice application.diff --git a/docs/research/native-h264-hardware-decision.md b/docs/research/native-h264-hardware-decision.mdindex 6b81e4a3..85a3ece9 100644--- a/docs/research/native-h264-hardware-decision.md+++ b/docs/research/native-h264-hardware-decision.md@@ -2,16 +2,21 @@Date: 2026-08-20-Status: `no-go-native-h264-hardware-pinned-fmtp`+Status: Historical gate. Its `no-go-native-h264-hardware-pinned-fmtp`+stop resolved when the explicit opt-in Pion path registered exact `42c01f`;+the product WGC/MF path later passed its attributed one-Viewer smoke.The later source-only functional slice in[`native-h264-opt-in-path.md`](./native-h264-opt-in-path.md) registers the-fixture profile explicitly and proves one H.264 Host/Viewer loopback. It does-not change this spike's physical-hardware decision: WebCodecs hardware-attribution remains unverified and production remains VP8/default.--This spike does not amend ADR-0006 or authorize another product, room, or-production run. It answers one narrow question: can a bounded Windows H.264+fixture profile explicitly and proves one browser H.264 Host/Viewer loopback.+The native-window source now compiles this fixture's MF encoder source into the+product helper, retaining hardware-only enumeration and exact readback. That+path rendered 203 frames for one Chrome Viewer while the exact helper PID and+adapter LUID matched nonzero Windows `VideoEncode`; offline fixture telemetry+remains separate evidence.++This spike did not amend ADR-0006 or authorize another product, room, or+production run. It answered one narrow question: can a bounded Windows H.264path satisfy the project's physical hardware-encode and existing default Pionfmtp contracts before product integration?@@ -20,7 +25,8 @@ emitted SPS differed from the requested codec string. A subsequenthardware-only Media Foundation run did prove the selected NVIDIA encoder andGPU engine, but emitted exact `profile-level-id=42c01f`, which is absent fromthe default Pion mode-1 fmtp set. It therefore stopped before Pion, a Viewer,-rooms, or product wiring. Neither result authorizes an H.264 product switch.+rooms, or product wiring. The later opt-in source resolved that exact+registration mismatch without changing the browser VP8 default.## Decision Summary@@ -31,8 +37,8 @@ rooms, or product wiring. Neither result authorizes an H.264 product switch.| Did the requested codec string describe the emitted SPS exactly? | No. The requested `42e01f` produced SPS bytes `42 04 1f`. || Did forced key frames carry in-band recovery data? | Yes in this fixture: all six key chunks contained AUD, SPS, PPS, and IDR NAL units. || Can Pion v1.10.5 packetize the resulting Annex-B access units? | Yes statically: its `H264Payloader` splits 3/4-byte start codes, suppresses AUD/filler, emits SPS/PPS as STAP-A, and fragments slices as FU-A. |-| Is unmodified browser Viewer decode proven? | No. No PeerConnection, room, Viewer, or production service participated. |-| Should the small WebCodecs/Pion codec diff be implemented? | No. It would preserve the two unresolved product gates: physical encoder identity and exact emitted-fmtp registration plus Viewer interoperability. |+| Did this isolated spike prove unmodified browser Viewer decode? | No. No PeerConnection, room, Viewer, or production service participated. A later opt-in loopback records that evidence separately. |+| Did this spike authorize the small WebCodecs/Pion codec diff? | No. The later exact-`42c01f` opt-in reopened and proved that boundary separately. || Did the Media Foundation fixture prove physical H.264 hardware encode? | Yes for one local RTX 4070 SUPER run: adapter-LUID-bound hardware enumeration, D3D11 awareness, the NVIDIA H.264 Encoder MFT, and process-plus-LUID `VideoEncode` activity agreed. || Did that hardware MFT emit an exact default Pion mode-1 fmtp string? | No. It emitted `42c01f`; the default set is `42001f`, `42e01f`, `4d001f`, and `64001f`. The fixture failed closed without changing the request or adding a fallback. This is not a standards-level profile incompatibility. |@@ -178,10 +184,10 @@ preference, not physical hardware evidence.## Media Foundation Hardware FixtureThe follow-up fixture is retained at-`native/fixtures/mf-h264-hardware/`. It is a Windows-only offline C++ program,-not a sender component. Its generated object and executable stay outside the-repository. It has one H.264 path and no software MFT, codec, adapter, room,-capture, audio, signaling, Pion, or Viewer fallback.+`native/fixtures/mf-h264-hardware/`. Its build compiles the shared encoder+source from `native/window-capture-helper/` with fixture-only synthetic input+and telemetry. The generated fixture executable stays outside the repository+and has no room, capture, audio, signaling, Pion, or Viewer fallback.### Fixed Contractdiff --git a/docs/research/native-h264-opt-in-path.md b/docs/research/native-h264-opt-in-path.mdindex 7d331359..c46beb5c 100644--- a/docs/research/native-h264-opt-in-path.md+++ b/docs/research/native-h264-opt-in-path.md@@ -1,27 +1,79 @@# Native H.264 Opt-In Path-## 2026-08-21 00:15+## 2026-08-21 Native Window Source-The native sender now has a small, explicit H.264 mode. VP8 remains the default-and the Web sender is unchanged. A sender chooses H.264 in its local Codec-control; the choice is carried only through the loopback start/media handshake.+The Native sender now has two explicit local video sources. Browser VP8 remains+the default; browser H.264 remains available as the earlier opt-in. The new+`native-window-h264` source is Windows 11-only and requires the user to select+one local opaque window target. It never silently replaces a browser source or+falls back to software encode, another codec, monitor capture, or system audio.-The H.264 path is:+The native-window path is:```text-Chrome WebCodecs H.264 Annex-B- -> existing 17-byte local frame envelope+opaque local target -> HWND + PID + process creation time+ -> WGC BGRA surface on one selected D3D11 adapter+ -> D3D11 VideoProcessor 1280x720 NV12+ -> adapter-bound hardware-only MF H.264 42c01f Annex-B+ -> generation-bound Go Session.WriteMedia-> one Go queue and Pion H264Payloader-> one TrackLocalStaticRTP-> at most two independent PeerConnections```-The Go session creates a private Pion `MediaEngine` only for H.264 sessions. It-registers the fixture-derived Constrained Baseline `profile-level-id=42c01f`,-`packetization-mode=1`, and a dedicated dynamic payload type, then restricts-the sender transceiver to that H.264 capability. The shared fanout and keyframe-feedback remain unchanged. The default VP8 session still uses Pion's default-API and payloader.+The helper revalidates HWND-to-PID and PID creation time immediately before WGC+activation. The same process identity feeds WASAPI application loopback with+`INCLUDE_TARGET_PROCESS_TREE`. One private stdout protocol multiplexes bounded+PCM, H.264 access units, and local hardware status. PID, HWND, creation time,+adapter/MFT identity, PCM, and status never enter signaling. Go sends H.264+directly to the existing fanout; a second local copy travels only to the sender+UI's `VideoDecoder` preview and never returns to Go. PCM retains the existing+local WebCodecs Opus bridge.++PLI/FIR and fanout recovery requests enter the generation-bound helper stdin as+one coalesced `K` command. The helper forces an in-band SPS/PPS/IDR recovery+unit, while the existing Pion `H264Payloader`, exact `42c01f` MediaEngine, one+shared track, and two-edge cap remain unchanged. Browser VP8/H.264 use their+existing `getDisplayMedia` and `VideoEncoder` path.++Both the product helper and offline fixture compile the same MF encoder source.+The product build and the fixture build pass MSVC `/W4 /WX`; focused Go tests+cover bound target arguments, multiplexed protocol bounds, key-frame command+forwarding, generation/source validation, and the peer-assisted Host contract.++## Native Window Acceptance Evidence++One bounded Chrome 151 run used a real animated top-level window, process-tree+audio, the explicit Native source, and the peer-assisted direct-child wire. It+created no browser capture or `VideoEncoder`; the ordinary Viewer received+1,313 packets/1,431,696 bytes during the sample, decoded and rendered 203+1280x720 frames, and received 500 Opus packets without a fatal/encoder error.+The helper wrote 30 frames and 174 source RTP packets before the retained+sender-start snapshot.++The exact helper process was PID 15060. Its Windows GPU Engine instance+`pid_15060_luid_0x00000000_0x0001a496_phys_0_eng_6_engtype_videoencode`+reached 2.98028% utilization; the helper's independently validated adapter+status was `0x00000000:0x0001a496`. This is process-and-adapter-correlated+hardware evidence, not a throughput or performance claim. The generic gate+reported failure only because its asynchronous Pion diagnostics delta stayed+zero after the Viewer counters advanced; the run was not repeated for that+known sampling residue.++Primary sources checked 2026-08-21:++- [Windows Graphics Capture](https://learn.microsoft.com/en-us/windows/uwp/audio-video-camera/screen-capture)+- [`IGraphicsCaptureItemInterop::CreateForWindow`](https://learn.microsoft.com/en-us/windows/win32/api/windows.graphics.capture.interop/nf-windows-graphics-capture-interop-igraphicscaptureiteminterop-createforwindow)+- [`Direct3D11CaptureFramePool::CreateFreeThreaded`](https://learn.microsoft.com/en-us/uwp/api/windows.graphics.capture.direct3d11captureframepool.createfreethreaded)+- [`CreateDirect3D11DeviceFromDXGIDevice`](https://learn.microsoft.com/en-us/windows/win32/api/windows.graphics.directx.direct3d11.interop/nf-windows-graphics-directx-direct3d11-interop-createdirect3d11devicefromdxgidevice)+- [Microsoft MIT HWND capture sample](https://github.com/microsoft/Windows.UI.Composition-Win32-Samples/tree/master/cpp/ScreenCaptureforHWND)+- [`ID3D11VideoDevice::CreateVideoProcessorInputView`](https://learn.microsoft.com/en-us/windows/win32/api/d3d11/nf-d3d11-id3d11videodevice-createvideoprocessorinputview)+- [`ID3D11VideoContext::VideoProcessorSetOutputBackgroundColor`](https://learn.microsoft.com/en-us/windows/win32/api/d3d11/nf-d3d11-id3d11videocontext-videoprocessorsetoutputbackgroundcolor)++## 2026-08-21 Browser H.264 Source++The earlier browser H.264 source remains an explicit option. Its loopback path+is Chrome WebCodecs Annex-B -> the local envelope -> the same Go/Pion fanout.## Bounded Loopback Evidence@@ -47,21 +99,22 @@ loopback, not a public-network, multi-viewer, endurance, or codec benchmark.## Hardware Boundary-The Media Foundation/NVIDIA fixture remains the physical hardware evidence and-informed the exact `42c01f` profile and in-band SPS/PPS/keyframe contract. The-product sender requests WebCodecs H.264 hardware acceleration, but browser-`isConfigSupported()` and the preference string do not prove which encoder was-selected. The UI therefore reports hardware preference separately from hardware-evidence, which remains unverified. Physical hardware attribution and direct-Media Foundation capture integration are later work; this change does not-silently claim either one.+The Media Foundation/NVIDIA fixture established the physical hardware gate and+the exact `42c01f` in-band recovery contract. Browser H.264 still has only a+hardware preference. The native-window source instead enumerates only an+adapter-LUID-bound hardware MFT, requires D3D11 awareness and exact codec+readback, and has no software retry. The Native acceptance run above adds the+required product-process `VideoEncode` attribution; fixture evidence remains a+separate encoder-contract measurement.## Scope-- No production deployment was made.+- Browser VP8 remains the default and no production deployment switch is made.- No SFU, TURN, Web viewer, access protocol, or default VP8 behavior changed.- No codec matrix, benchmark, second viewer, or endurance run is required forthis landing slice.+- The Native source reuses existing controller compatibility and evaluation+ packaging changes; it adds no topology.- H.264 distribution still carries the existing patent/license review boundary.---diff --git a/docs/status.md b/docs/status.mdindex dcc93d6a..b3db49b9 100644--- a/docs/status.md+++ b/docs/status.md@@ -7,7 +7,7 @@ Last updated: 2026-08-21`https://share.bonfire.icu` serves exact `fd76277b05d491af8840b28f3132b7ff445d3cbe`; the 915,377-byte artifact SHA-256 is `8da4b8b2b1ae4b82add615f4367ee447b5ade86c4e58a938e51085940b9867d3`.The cutover held the lock 8,389 ms and returned local health 520 ms after stop. SQLite v3 has five rooms; Screener, LiveKit, coturn and nginx are healthy with zero restarts.Ordinary ICE is STUN-only; all rooms use the bounded peer/SFU-UDP controller and selected-edge UDP TTL 120. Room `1` is historical smoke and no real TURN/SFU media canary has run.-Web Host names/audio hint are deployed. Native H.264 and Windows x64 packaging are source-only; Web defaults VP8.+Web names/audio hint are deployed. Native WGC/MF/package is source-only; Web defaults VP8.## Execution Principle@@ -18,7 +18,7 @@ Flagship; parallel; min run/smoke/rollback; benchmark later. Active UI/config/lo- Capture precedes room creation; source/quality changes preserve peers, while pause disables audio/video tracks together and keeps connections. Quality defaults to `maintain-resolution` with balanced/fluid options.- Production access is `screener-v2`: a valid private fragment grant enters directly; every code-only Viewer must first hold site access, after which public-watch accepts the code and private rooms still require their room password. `SITE_ACCESS_PASSWORD`, `/api/site-access`, and the site-access cookie are deployed atomically; raw credentials are never stored.- Web Hosts can set their local/session display name, and the current release advertises it through the existing presence wire. The name is not an account identity.-- Native v2 remains source-only (memory rooms, 300s reclaim); its Host now follows authenticated peer-assisted direct-child assignments with a two-edge cap and bounded SFU/selected-ingress failure, but production does not run it.+- Native v2 is source-only (memory rooms, 300s reclaim); it follows authenticated direct-child assignments up to two and fails unsupported SFU/selected ingress boundedly.- PR #44's STUN-only/SFU-UDP router and token-free fallback prewarm are process-enabled for all normal rooms: roots <=2, browser relay <=1, bounded failure. Room `1` is historical smoke; mobile/iPad viewers are leaves, healthy edges sticky, and HTTPS/WSS stays TLS/TCP.- After an answer, a generation-bound 15s initial-connect deadline enters ICE restart; success/replacement/disposal cancels it. It is deployed but not mobile-verified.- Room `1` can publish exactly `HIGH+LOW` with Dynacast/backup codec off and subscriber `HIGH` ceilings; it has no retained real media frame.@@ -30,10 +30,11 @@ Flagship; parallel; min run/smoke/rollback; benchmark later. Active UI/config/lo- Chrome 151 synthetic `1/3/5/8` and 720p30 quality-change runs kept 2/1 fanout and decoding; slowest first frame was 1.05 seconds and one relay close recovered in 5.32 seconds. This is control evidence only.- Host A+B and authenticated P2P Viewer C are sanitized, generation-bound, read-only, and fail closed for stale, ambiguous, or SFU-fed evidence; no raw media metadata is retained.- The `a11a73d` built-in TURN canary is rejected: local allocation passed, but direct Host/Pion Viewer failed before forced relay. Full rollback restored STUN-only client ICE and zero allocations.-- Chrome 151 loopbacks decoded VP8 299 and H.264 opt-in 299/298 rendered at 1280x720 without fatal/encoder errors. Hardware attribution, Pion timing, multi-viewer/endurance/public/native proof remain open; production stays VP8.-- Native Win11 audio is source-only/default-off. Target isolation was 4018x; one Viewer got 495 Opus packets. Package build passes; download, game sync, Win10, and other routes are open.+- Chrome 151 browser loopbacks decoded VP8 299 and H.264 opt-in 299/298 rendered at 1280x720 without fatal/encoder errors. Browser hardware attribution, multi-viewer/endurance and public proof remain open; production stays VP8.+- Native WGC/MF used no browser capture/encoder. One Chrome Viewer rendered 203 1280x720 frames and received 500 Opus packets; process/LUID-correlated `VideoEncode` was nonzero, with no fatal/encoder errors.+- Native Win11 audio is source-only/default-off: target isolation was 4018x and one Viewer got 495 Opus packets. Download, game sync, Win10, and other routes remain open.- Access protocol/config/HTTP/storage/SQLite/signaling focused tests pass, including commit-first teardown and v1 rollback.-- Native peer-assisted wire/session focused tests pass: authoritative direct children reconcile at no more than two, presence cannot create a third edge, stale revisions do not rebuild, and unsupported SFU/selected ingress fails once per current revision.+- Native wire/session tests enforce at most two authoritative children, no presence-created edge, stale-revision ignore, and one bounded unsupported-route failure per revision.- Source adds selected Host ingress and peer last-mile for every enabled room; focused routing/config tests pass with Peer ICE STUN-only.- Host names, the window-audio hint, all-room routing, and all-room Host selected ingress are active; the stale restart assertion is historical only.- The `fd76277` gate preserved the environment and SQLite hashes, v3/five rooms/room `1`, selected-edge/SFU/ordinary-ICE settings, services, and local/public health plus route/asset 200s. Site access returned the unauthenticated status and the retired endpoint returned 404; no TURN/SFU session ran. Exact `5e4a3679076a9ea2fe7a41fadf4be65a439db450` is the rollback release.@@ -55,7 +56,7 @@ Flagship; parallel; min run/smoke/rollback; benchmark later. Active UI/config/loRepeat room `1` and fix only the identified SFU connect/source/publish/audio/transport layer; retain selected UDP, a rendered frame, edge caps and resource deltas.Recovery uses one ICE restart, one same-parent rebuild, one alternate peer, then SFU. Next gate exactly-two/Dynacast-off BWE on a zero-child SFU leaf, then root evacuation.-Package one-Viewer VP8/H.264/Opus first; WGC/MF hardware then Viewer2/FIFO are functional slices. Real-game A/V and resource benchmarks follow functional landing.+Native WGC/MF passed one-Viewer hardware; package download, Viewer2/FIFO and game A/V remain. Benchmark later.Ordinary Peer ICE stays STUN-only. Selected-edge Host ingress still needs one forced-relay canary before a media-success claim.ADR-0004 still needs the 30-minute `1/3/5/8` matrix; a separate 20-viewer gate must pass before changing the default eight.@@ -66,6 +67,5 @@ host/provider UDP 7882 and bounded services are deployed, but real externalmedia/device evidence is absent. Shared IP remains smoke-only and cannot approveclean-port migration; the full gate still needs an isolated VM/IP.-- Whole-system versus selected-game audio for the first release.- Initial deployment region and network cohort.- Project license and distribution model.diff --git "a/docs/\346\226\271\346\241\210\350\256\276\350\256\241.md" "b/docs/\346\226\271\346\241\210\350\256\276\350\256\241.md"index e8696cb6..40fa2933 100644--- "a/docs/\346\226\271\346\241\210\350\256\276\350\256\241.md"+++ "b/docs/\346\226\271\346\241\210\350\256\276\350\256\241.md"@@ -160,7 +160,9 @@ PoC 不强制 codec,也不做 SDP munging。先通过 stats 记录实际协商`audio: true` 只是请求。若 `getAudioTracks()` 为空,分享者界面必须显示“当前来源没有可共享音频”。`systemAudio`、`windowAudio` 和 `audioSelection` 只影响选源器偏好,浏览器可以忽略且没有对应的标准音频来源类型读回,因此不能显示为系统、窗口或所选游戏音频保证。纯 Web 首版接受浏览器实际提供的标签页或系统音频,不承诺仅捕获某个游戏进程;系统音频可能把语音软件中观众自己的声音再次采入,这属于源内容泄漏而非 Host 页面回放,AEC 或 Web Audio 无法从已混合 track 恢复进程身份。Web 端必须保留 video-only 选择并明确风险,不能声称已隔离语音。-当前 Windows Native sender 以所选游戏 PID 与进程创建时间为本地边界,使用 WASAPI process loopback 的 include-target-process-tree 捕获游戏及其子进程。目标进程无渲染流、退出或捕获失败时显示无声/未知,不回退为整机音频,也不上传 PID、路径、窗口标题、设备标识或原始音频。该 source-only 路径当前只支持 Windows 11;Windows 10 game-only 音频保持明确不支持,打包与其他媒体 route 仍待完成。+当前 Windows Native sender 用本地 opaque ID 保存所选窗口的 HWND、PID 与进程创建时间;helper 在使用前重新验证三者绑定,并以 WASAPI process loopback 的 include-target-process-tree 捕获游戏及其子进程。目标进程无渲染流、退出或捕获失败时显示无声/未知,不回退为整机音频,也不上传 PID、路径、窗口标题、设备标识或原始音频。该路径只支持 Windows 11;Windows 10 game-only 音频保持明确不支持。++同一 helper 的显式 `native-window-h264` source 用 `CreateForWindow` 建立 WGC item,在所选 adapter 的 free-threaded D3D11 frame pool 接收 BGRA surface;D3D11 VideoProcessor 做固定 1280x720/30 aspect-fit NV12 转换,随后进入 adapter-LUID-bound、hardware-only asynchronous Media Foundation H.264 MFT。输出必须是 Annex-B `42c01f`,首次、周期和反馈恢复帧携带 SPS、PPS 与 IDR。PCM、H.264 access unit 和 local-only hardware status 通过一个有界 helper protocol 进入 generation-bound Go session;H.264 直接写现有 shared fanout,UI 只收一份单向预览副本,PCM 保持现有本地 WebCodecs Opus bridge。任一 Native capture/encode 失败都终止该 source,不静默改用软件 MFT、其他 codec、browser capture、monitor capture 或 system mix;browser VP8 保持默认,browser H.264 保持独立 opt-in。一名普通 Chrome Viewer 已 decoded/rendered 203 帧并收到 500 Opus packets;同一 helper PID 与 adapter LUID 的 `VideoEncode` 观测匹配。直播中切换来源仍由明确的按钮手势重新调用 `getDisplayMedia()`,不改变房间、host generation 或媒体拓扑。选源取消或失败时继续发送旧流;成功后为每个现有 sender 调用 `replaceTrack()`,健康连接不重新协商。初始 offer 预留 send-only audio transceiver,使音频可以在 `null -> track -> null` 间切换。若某个 peer 的替换或回滚失败,只重建该 peer;新流提交后才停止旧流,并以当前 stream 身份保护旧 track 的迟到 `ended` 事件。统计码率累加器在换源后重置,避免把不同 track 的计数器混算。diff --git "a/docs/\351\234\200\346\261\202\347\220\206\350\247\243.md" "b/docs/\351\234\200\346\261\202\347\220\206\350\247\243.md"index 6605ad53..18cae0dd 100644--- "a/docs/\351\234\200\346\261\202\347\220\206\350\247\243.md"+++ "b/docs/\351\234\200\346\261\202\347\220\206\350\247\243.md"@@ -53,7 +53,7 @@- [ ] 分包/分层多树实验:把编码包、对象或可独立解码层拆到多个 peer 路径以分散上传,只能在独立 spike 中验证。必须保持亚秒级,处理多父重组、排序、丢包恢复、关键帧、同步和节点退出;未通过门槛前不进入产品路由控制器。- [ ] SFU-root 中央兜底:产品部署目标必须提供经 canary 验收的单节点 SFU 资源;系统在 direct/peer UDP 不可用或明确拓扑预算无法接纳时自动把它作为主要中央兜底,不要求用户选模式。它只向总计一至两个必要 roots 出口;root 优先承担受限 peer descendants,没有可靠 relay root 时必要 viewer 只作为同一总上限内的零 descendants root。已关闭 PR #12 的显式整房 SFU 模式已被取代。room `1` 的历史 smoke 没有 retained SFU media;公网媒体、资源和恢复门禁仍适用于全站路由。- [ ] 分享端桌面封装:用 Electron 固定 Chromium 版本、改善选源/热键/系统音频体验,同时保留 Web 观看端。-- [ ] Windows 应用音频:Windows 11 原生候选默认仅捕获所选游戏及其子进程的音频,避免把语音聊天或通知声一并分享;“全部系统声音”只能由分享者显式选择。普通 Windows 10 不具备当前 process-loopback API 门槛,必须明确标为未解决/不支持而不是伪兼容。浏览器无法证明或隔离任意进程音频时必须如实提示,原生捕获失败、无声或进程退出不得静默扩大为整机捕获。+- [ ] Windows 应用音频:Windows 11 Native sender 使用本地 opaque window target 绑定 HWND、PID 与进程创建时间,只捕获所选游戏及其子进程的音频,避免把语音聊天或通知声一并分享;“全部系统声音”只能由分享者显式选择。普通 Windows 10 不具备当前 process-loopback API 门槛,必须明确标为未解决/不支持而不是伪兼容。浏览器无法证明或隔离任意进程音频时必须如实提示,原生捕获失败、无声或进程退出不得静默扩大为整机捕获。- [ ] 连接自检:在开始分享前验证信令、STUN 和 SFU ICE/UDP 可达性,并给出不暴露地址或凭据的诊断结果。启用 selected-edge TURN canary 时只报告脱敏的授权阶段与 selected candidate type,不输出 URL、username、credential、地址或原始 candidate;媒体 TCP 不在当前配置面。- [ ] 受限网络增强:短期鉴权 TURN 只能由完整、默认关闭的部署 tuple 与当前 room controller 启用,并且只能在 direct/peer UDP 与 SFU/UDP 均已按有界梯级失败后,向 controller 选中的当前异常 edge 一次性授权。普通 peer PC 从创建起保持 STUN-only;不得向整个 participant、房间或所有 peer connections 预发 TURN candidates。凭据必须绑定当前授权 session 与 route/connection generation、短期有效、内存态、一次性消费且不进入 URL、日志、浏览器持久化或 SQLite。媒体 TCP 仍需独立 canary;两者都不能被描述为更高清的传输方式。- [ ] 运行时编解码选择:先保留浏览器默认协商;只有端点能力相交、Media Capabilities 对目标分辨率/FPS/码率报告 supported/smooth/power-efficient,且实际 outbound codec、区间 encode time、游戏 FPS、CPU/GPU 与 sender 数量共同通过时,才调整 H.264、VP8、VP9 或 AV1 优先级。AV1 不得因压缩率推断而默认启用,任何候选都不得静默落入高负载软件编码。@@ -61,7 +61,7 @@### P2 - 可选功能-- [ ] 原生 Windows 捕获辅助程序,使用 WGC/DDA、WASAPI 和硬件编码器降低复制与编码开销。+- [x] 原生 Windows 捕获辅助程序:显式 `native-window-h264` 模式使用同一 bound window target 的 WGC/D3D11 画面与 WASAPI process-tree PCM,并用 adapter-bound、hardware-only Media Foundation H.264 `42c01f` 直接进入现有 Go fanout;浏览器 VP8/H.264 是显式替代选项,VP8 默认,不允许运行时静默改用软件编码、显示器画面或系统混音。一名普通 Chrome Viewer 已 decoded/rendered 203 帧并收到 500 Opus packets,同一 helper PID/adapter 的 `VideoEncode` 与 LUID 匹配。- [ ] iOS 原生分享端有界实验:当前不实现 ReplayKit/broadcast-extension 兼容栈。待 iOS 27 与对应 ScreenCaptureKit SDK 脱离 beta 并稳定后,只评估 ScreenCaptureKit system picker + fixed-`HIGH` + 一至两名 Web Viewer 的窄路径;VideoToolbox 必须要求并读回硬件编码,无法证明时 fail closed,不增加 ReplayKit fallback、`LOW`、AV1 或新拓扑。- [ ] Viewer 向电视输出:系统级屏幕镜像现在可直接使用;产品只复用平台 AirPlay/Cast,不自造投屏协议。首个 Web 候选只在运行时检测 `HTMLMediaElement.remote`,并在真实 AirPlay/Cast 目标上证明当前 live WebRTC `srcObject` 可播放后才显示入口;不支持时保留浏览器原生媒体控件/系统镜像。投屏是 Viewer 的本地播放输出,不能改变 room、TURN/SFU/peer route、Host fanout 或服务端媒体分发。Google Cast receiver 需要可由接收器获取的媒体 URL,当前 P2P `MediaStream` 不满足该输入,因此不加入 Cast SDK/receiver。- [ ] 多地区 SFU-root 调度,以及可选 TURN 兼容层的按地区、运营商可达率。@@ -158,6 +158,7 @@- Given 浏览器返回屏幕音轨,When 产品展示音频状态或未来连接详情,Then 只能把存在性、浏览器实际返回的 track settings、实际协商 codec/fmtp 和实际流量分别标为捕获、协商或观测事实;字段缺失必须显示为未知,且不得从 Opus 的 `48000/2` RTP 声明推断源采样率或实际 stereo。- Given Web 或后续 native sender 分享电影/视频音频,When 进行产品验收,Then 必须用独立 A/B/C 证据关联 capture audio track/settings、实际 negotiated codec/fmtp、outbound/inbound 实际码率、loss、jitter、concealment、jitter-buffer 与 A/V sync;native sender 必须通过该音质和同步矩阵,且不得把 audio `maxBitrate` ceiling 当作提质证明。- Given 分享者同时运行语音软件,When 选择仅分享游戏声音,Then supported Windows 11 native sender 只保留目标游戏及其子进程的渲染音频,观众、通知和其他应用声音不得进入分享;浏览器 system audio 无法满足时必须允许 video-only 或明确告知其可能包含语音/通知,不能用 AEC、降噪或来源标签伪称已隔离。+- Given 分享者显式选择 Native window H.264 与一个本地窗口,When 分享开始,Then helper 必须在再次验证 HWND/PID/creation-time 后以 WGC/D3D11 和 hardware-only MF H.264 产生一份 `42c01f` Annex-B 输出并直接复用现有 Go/Pion fanout;Viewer 仍使用普通浏览器。硬件 MFT、WGC、音频或目标身份任一失败都明确停止该模式,不得静默改用软件编码、浏览器画面、显示器画面或系统混音。### 仓库可移植性diff --git a/native/fixtures/mf-h264-hardware/README.md b/native/fixtures/mf-h264-hardware/README.mdindex d82505dd..c138b3b4 100644--- a/native/fixtures/mf-h264-hardware/README.md+++ b/native/fixtures/mf-h264-hardware/README.md@@ -1,8 +1,10 @@# Media Foundation H.264 Hardware Fixture-This is an offline Windows-only decision fixture. It is not linked into the-Screener sender and does not contact a room, signaling service, Viewer, or-production endpoint.+This is an offline Windows-only decision fixture. Its build compiles the same+Media Foundation encoder source used by `native/window-capture-helper`, under a+fixture-only macro for synthetic input and telemetry. The generated fixture is+not linked into the sender and does not contact a room, signaling service,+Viewer, or production endpoint.The fixture selects one DXGI adapter and one adapter-bound Media Foundationtransform. It never enumerates a software MFT and has no codec or software@@ -36,10 +38,9 @@ The fixture prints bounded key/value evidence to stdout and never writes theencoded bitstream. A nonzero exit is a measured no-go at the reported stage;it must not be worked around by adding a software retry.-The Pion/two-binding and browser decode boundary remains separate. The current-machine has no Go toolchain in `PATH`, and the existing product bridge is still-no-go before first Go RTP. Coupling either boundary into this encoder fixture-would hide which layer failed.+Pion transport and browser decode remain separate from this offline fixture;+the product path reuses its exact `42c01f`, Annex-B, adapter-bound hardware+contract without adding those network concerns to this executable.Primary API references, accessed 2026-08-20:diff --git a/native/fixtures/mf-h264-hardware/build.ps1 b/native/fixtures/mf-h264-hardware/build.ps1index 46883b3b..9bb08641 100644--- a/native/fixtures/mf-h264-hardware/build.ps1+++ b/native/fixtures/mf-h264-hardware/build.ps1@@ -28,7 +28,7 @@ if ([string]::IsNullOrWhiteSpace($installationPath)) {}$developerCommand = Join-Path $installationPath 'Common7\Tools\VsDevCmd.bat'-$sourcePath = Join-Path $fixtureDirectory 'main.cpp'+$sourcePath = Join-Path $fixtureDirectory '..\..\window-capture-helper\main.cpp'$executablePath = Join-Path $outputPath 'screener-mf-h264-fixture.exe'$objectPath = Join-Path $outputPath 'main.obj'@@ -36,7 +36,7 @@ New-Item -ItemType Directory -Path $outputPath -Force | Out-Null$compile = @('call "{0}" -arch=x64 -host_arch=x64 >nul' -f $developerCommand- 'cl.exe /nologo /std:c++20 /EHsc /W4 /WX /DUNICODE /D_UNICODE /DWIN32_LEAN_AND_MEAN /D_WIN32_WINNT=0x0A00 "{0}" /Fo:"{1}" /Fe:"{2}" /link mfplat.lib mf.lib mfuuid.lib d3d11.lib dxgi.lib dxguid.lib evr.lib ole32.lib oleaut32.lib pdh.lib' -f $sourcePath, $objectPath, $executablePath+ 'cl.exe /nologo /std:c++20 /EHsc /W4 /WX /DSCREENER_H264_FIXTURE /DUNICODE /D_UNICODE /DWIN32_LEAN_AND_MEAN /D_WIN32_WINNT=0x0A00 "{0}" /Fo:"{1}" /Fe:"{2}" /link mfplat.lib mf.lib mfuuid.lib d3d11.lib dxgi.lib dxguid.lib evr.lib ole32.lib oleaut32.lib pdh.lib windowsapp.lib' -f $sourcePath, $objectPath, $executablePath) -join ' && '& cmd.exe /d /s /c $compilediff --git a/native/process-audio-helper/README.md b/native/process-audio-helper/README.mddeleted file mode 100644index 844c7936..00000000--- a/native/process-audio-helper/README.md+++ /dev/null@@ -1,22 +0,0 @@-# Windows Process-Audio Helper--This Windows 11 helper is the local input for Native sender process-tree audio.-It enumerates visible top-level windows locally and captures only the explicitly-selected process plus its children through WASAPI application loopback. It has-no whole-system fallback and no network code.--The standard evaluation ZIP builds this helper beside the sender through-`scripts/package-native-sender.ps1`. For a helper-only development build, use-an output directory outside the repository (or set-`SCREENER_PROCESS_AUDIO_HELPER` for a development run):--```powershell-$out = Join-Path ([IO.Path]::GetTempPath()) 'screener-native-audio'-./native/process-audio-helper/build.ps1 -OutputDirectory $out-```--The implementation follows Microsoft's MIT-licensed Application Loopback-sample contract without copying its WIL/Media Foundation fixture machinery:--- <https://github.com/microsoft/Windows-classic-samples/tree/main/Samples/ApplicationLoopback>-- <https://learn.microsoft.com/en-us/windows/win32/api/audioclientactivationparams/ns-audioclientactivationparams-audioclient_activation_params>diff --git a/native/sender/README.md b/native/sender/README.mdindex 92d8a16c..ef59d8d8 100644--- a/native/sender/README.md+++ b/native/sender/README.md@@ -1,25 +1,27 @@# Screener Native Sender for Windows x64-This evaluation package contains the current Native sender and the Windows 11-process-audio helper. It has no installer, updater, code signature, or bundled+This evaluation package contains the current Native sender and its Windows 11+window-capture helper. It has no installer, updater, code signature, or bundledserver.## Run-1. Keep `screener-sender.exe` and `screener-process-audio.exe` in the same+1. Keep `screener-sender.exe` and `screener-window-capture.exe` in the samedirectory.2. Run `screener-sender.exe`. It opens a loopback-only UI in the defaultbrowser; current Chrome or Edge is required.-3. Enter the Screener server and site access password, choose VP8 (default)- or experimental H.264, then choose the screen or window to share.-4. Process audio is off by default. To enable it, select the exact target in- the audio list and select the same target in the browser picker.+3. Enter the Screener server and site access password. Browser VP8 remains the+ default; browser H.264 and Native window H.264 are explicit opt-ins.+4. For a browser video source, choose the browser screen/window and optionally+ select the same window target for process-tree audio. Native window H.264+ requires one target and captures its video and process-tree audio together.5. Close the console window to stop the sender.-Process audio requires Windows 11 build 22000 or newer. It captures only the-selected process tree and never falls back to the system mix. H.264 requests-browser hardware preference but does not prove which physical encoder Chrome-or Edge selected. Viewers continue using the normal Web client.+Native capture requires Windows 11 build 22000 or newer. It binds the selected+window to its process identity, never falls back to the system mix, and never+falls back from the hardware-only Media Foundation H.264 encoder to software.+Browser H.264 still requests browser hardware preference without proving which+physical encoder Chrome or Edge selected. Viewers use the normal Web client.Verify the downloaded ZIP against its adjacent `.sha256` file, then verify thefiles inside it with `SHA256SUMS.txt`. The package is unsigned, so Windows maydiff --git a/native/sender/internal/app/server.go b/native/sender/internal/app/server.goindex 63cde53c..f179ff0c 100644--- a/native/sender/internal/app/server.go+++ b/native/sender/internal/app/server.go@@ -16,11 +16,18 @@ import ("github.com/TNTcraftHIM/Screener/native/sender/internal/media""github.com/TNTcraftHIM/Screener/native/sender/internal/remote"- "github.com/TNTcraftHIM/Screener/native/sender/internal/windowaudio"+ "github.com/TNTcraftHIM/Screener/native/sender/internal/windowcapture""github.com/coder/websocket""github.com/coder/websocket/wsjson")+type videoSource string++const (+ videoSourceBrowser videoSource = "browser"+ videoSourceNativeWindow videoSource = "native-window-h264"+)+const (processTokenHeader = "X-Screener-Process-Token"maxLocalJSONBytes = 8 << 10@@ -38,43 +45,47 @@ type App struct {token stringattachTimeout time.Duration- mu sync.Mutex- writeMu sync.Mutex- server *http.Server- listener net.Listener- expectedHost string- origin string- starting bool- generation uint64- startCancel context.CancelFunc- startFailure string- session *remote.Session- sessionCancel context.CancelFunc- room remote.Room- codec media.Codec- mediaConn *websocket.Conn- mediaClaimed bool- mediaClaimID uint64- nextClaimID uint64- attachTimer *time.Timer- configTimeout time.Duration- activeViewers int- waitingViewers int- audioProvider windowaudio.Provider- audioTargets map[string]windowaudio.Target- audioTarget *windowaudio.Target+ mu sync.Mutex+ writeMu sync.Mutex+ server *http.Server+ listener net.Listener+ expectedHost string+ origin string+ starting bool+ generation uint64+ startCancel context.CancelFunc+ startFailure string+ session *remote.Session+ sessionCancel context.CancelFunc+ room remote.Room+ codec media.Codec+ mediaConn *websocket.Conn+ mediaClaimed bool+ mediaClaimID uint64+ nextClaimID uint64+ attachTimer *time.Timer+ configTimeout time.Duration+ activeViewers int+ waitingViewers int+ captureProvider windowcapture.Provider+ windowTargets map[string]windowcapture.Target+ windowTarget *windowcapture.Target+ videoSource videoSource+ keyFrameRequests chan struct{}}type startRequest struct {- ServerURL string `json:"serverUrl"`- Password string `json:"password"`- Codec string `json:"codec,omitempty"`- AudioTargetID string `json:"audioTargetId,omitempty"`+ ServerURL string `json:"serverUrl"`+ Password string `json:"password"`+ Codec string `json:"codec,omitempty"`+ VideoSource string `json:"videoSource,omitempty"`+ WindowTargetID string `json:"windowTargetId,omitempty"`}type encoderConfig struct {Kind string `json:"kind"`Codec string `json:"codec"`+ VideoSource string `json:"videoSource"`Width int `json:"width"`Height int `json:"height"`FPS int `json:"fps"`@@ -83,7 +94,7 @@ type encoderConfig struct {Audio bool `json:"audio"`}-type localAudioTarget struct {+type localWindowTarget struct {ID string `json:"id"`Title string `json:"title"`}@@ -96,11 +107,12 @@ func New() (*App, error) {ctx, cancel := context.WithCancel(context.Background())return &App{ctx: ctx, cancel: cancel, token: token,- attachTimeout: mediaAttachTimeout,- configTimeout: mediaConfigTimeout,- codec: media.CodecVP8,- audioProvider: windowaudio.NewProvider(),- audioTargets: make(map[string]windowaudio.Target),+ attachTimeout: mediaAttachTimeout,+ configTimeout: mediaConfigTimeout,+ codec: media.CodecVP8,+ videoSource: videoSourceBrowser,+ captureProvider: windowcapture.NewProvider(),+ windowTargets: make(map[string]windowcapture.Target),}, nil}@@ -157,8 +169,8 @@ func (app *App) ServeHTTP(response http.ResponseWriter, request *http.Request) {app.handleStart(response, request)case "/api/stop":app.handleStop(response, request)- case "/api/audio-targets":- app.handleAudioTargets(response, request)+ case "/api/window-targets":+ app.handleWindowTargets(response, request)case "/media":app.handleMedia(response, request)default:@@ -197,21 +209,35 @@ func (app *App) handleStart(response http.ResponseWriter, request *http.Request)writeJSON(response, http.StatusBadRequest, map[string]string{"error": "unsupported native codec"})return}+ source, validSource := parseVideoSource(strings.TrimSpace(input.VideoSource))+ if !validSource {+ writeJSON(response, http.StatusBadRequest, map[string]string{"error": "unsupported video source"})+ return+ }+ if source == videoSourceNativeWindow && codec != media.CodecH264 {+ writeJSON(response, http.StatusBadRequest, map[string]string{"error": "native window capture requires H.264"})+ return+ }baseURL, err := normalizeRemoteBase(input.ServerURL)if err != nil {writeJSON(response, http.StatusBadRequest, map[string]string{"error": err.Error()})return}app.mu.Lock()- var audioTarget *windowaudio.Target- if input.AudioTargetID != "" {- selected, ok := app.audioTargets[input.AudioTargetID]+ var windowTarget *windowcapture.Target+ if input.WindowTargetID != "" {+ selected, ok := app.windowTargets[input.WindowTargetID]if !ok {app.mu.Unlock()- writeJSON(response, http.StatusBadRequest, map[string]string{"error": "selected audio target is no longer available"})+ writeJSON(response, http.StatusBadRequest, map[string]string{"error": "selected window target is no longer available"})return}- audioTarget = &selected+ windowTarget = &selected+ }+ if source == videoSourceNativeWindow && windowTarget == nil {+ app.mu.Unlock()+ writeJSON(response, http.StatusBadRequest, map[string]string{"error": "native window capture requires a selected window"})+ return}if app.starting || app.session != nil {app.mu.Unlock()@@ -227,11 +253,11 @@ func (app *App) handleStart(response http.ResponseWriter, request *http.Request)app.mu.Unlock()session, room, err := remote.Start(startContext, remote.StartOptions{- BaseURL: baseURL,+ BaseURL: baseURL,SiteAccessPassword: strings.TrimSpace(input.Password),- Codec: codec,- EnableAudio: audioTarget != nil,- OnEvent: func(event remote.Event) { app.emit(generation, event) },+ Codec: codec,+ EnableAudio: windowTarget != nil,+ OnEvent: func(event remote.Event) { app.emit(generation, event) },})app.mu.Lock()app.starting = false@@ -245,7 +271,8 @@ func (app *App) handleStart(response http.ResponseWriter, request *http.Request)app.sessionCancel = startCancelapp.room = roomapp.codec = codec- app.audioTarget = audioTarget+ app.videoSource = source+ app.windowTarget = windowTargetapp.attachTimer = time.AfterFunc(app.attachTimeout, func() {app.mu.Lock()expired := app.generation == generation && app.session == session && app.mediaConn == nil@@ -281,31 +308,31 @@ func (app *App) handleStop(response http.ResponseWriter, request *http.Request)response.WriteHeader(http.StatusNoContent)}-func (app *App) handleAudioTargets(response http.ResponseWriter, request *http.Request) {+func (app *App) handleWindowTargets(response http.ResponseWriter, request *http.Request) {if !app.authorizeAPI(response, request, http.MethodPost) {return}- targets, err := app.audioProvider.List(request.Context())+ targets, err := app.captureProvider.List(request.Context())if err != nil {writeJSON(response, http.StatusOK, map[string]any{- "supported": false, "targets": []localAudioTarget{},- "reason": "Windows 11 process audio helper is unavailable",+ "supported": false, "targets": []localWindowTarget{},+ "reason": "Windows 11 window capture helper is unavailable",})return}- localTargets := make([]localAudioTarget, 0, len(targets))- resolved := make(map[string]windowaudio.Target, len(targets))+ localTargets := make([]localWindowTarget, 0, len(targets))+ resolved := make(map[string]windowcapture.Target, len(targets))for _, target := range targets {id, tokenErr := newOpaqueToken()if tokenErr != nil {- writeJSON(response, http.StatusInternalServerError, map[string]string{"error": "create local audio target identity failed"})+ writeJSON(response, http.StatusInternalServerError, map[string]string{"error": "create local window target identity failed"})return}resolved[id] = target- localTargets = append(localTargets, localAudioTarget{ID: id, Title: target.Title})+ localTargets = append(localTargets, localWindowTarget{ID: id, Title: target.Title})}app.mu.Lock()- app.audioTargets = resolved+ app.windowTargets = resolvedapp.mu.Unlock()writeJSON(response, http.StatusOK, map[string]any{"supported": true, "targets": localTargets})}@@ -388,8 +415,12 @@ func (app *App) handleMedia(response http.ResponseWriter, request *http.Request)return}var config encoderConfig+ app.mu.Lock()+ source := app.videoSource+ target := app.windowTarget+ app.mu.Unlock()if err = decodeLocalPayload(payload, &config); err != nil ||- !validEncoderConfigForCodec(config, session.Codec()) || config.Audio != session.AudioEnabled() {+ !validEncoderConfigForSource(config, session.Codec(), source) || config.Audio != session.AudioEnabled() {session.Fail(errors.New("local encoder configuration is invalid"))_ = connection.Close(websocket.StatusPolicyViolation, "invalid encoder config")return@@ -398,38 +429,80 @@ func (app *App) handleMedia(response http.ResponseWriter, request *http.Request)session.Fail(fmt.Errorf("acknowledge local encoder configuration: %w", err))return}- app.mu.Lock()- target := app.audioTarget- app.mu.Unlock()captureContext, cancelCapture := context.WithCancel(app.ctx)- audioDone := make(chan struct{})+ captureDone := make(chan struct{})+ keyFrameRequests := make(chan struct{}, 1)+ if source == videoSourceNativeWindow {+ app.mu.Lock()+ if app.generation == generation && app.session == session {+ app.keyFrameRequests = keyFrameRequests+ }+ app.mu.Unlock()+ }+ writePCM := func(chunk windowcapture.PCMChunk) error {+ envelope, encodeErr := media.EncodePacket(media.Packet{+ Kind: media.KindPCM, Timestamp100ns: chunk.Timestamp100ns,+ Duration100ns: chunk.Duration100ns, Data: chunk.Data,+ })+ if encodeErr != nil {+ return encodeErr+ }+ return app.writeLocalBinary(connection, envelope)+ }if target != nil {_ = app.writeLocal(connection, map[string]string{"kind": "audio-state", "state": "starting"})- go func(selected windowaudio.Target) {- defer close(audioDone)- captureErr := app.audioProvider.Capture(captureContext, selected, func(chunk windowaudio.PCMChunk) error {- envelope, encodeErr := media.EncodePacket(media.Packet{- Kind: media.KindPCM, Timestamp100ns: chunk.Timestamp100ns,- Duration100ns: chunk.Duration100ns, Data: chunk.Data,- })- if encodeErr != nil {- return encodeErr- }- return app.writeLocalBinary(connection, envelope)- })+ go func(selected windowcapture.Target) {+ defer close(captureDone)+ var captureErr error+ if source == videoSourceNativeWindow {+ captureErr = app.captureProvider.CaptureWindow(+ captureContext, selected, keyFrameRequests, writePCM,+ func(accessUnit windowcapture.H264AccessUnit) error {+ packet := media.Packet{+ Kind: media.KindVideo, KeyFrame: accessUnit.KeyFrame,+ Timestamp100ns: accessUnit.Timestamp100ns,+ Duration100ns: accessUnit.Duration100ns, Data: accessUnit.Data,+ }+ if writeErr := session.WriteMedia(packet); writeErr != nil {+ return writeErr+ }+ envelope, encodeErr := media.EncodePacket(packet)+ if encodeErr != nil {+ return encodeErr+ }+ return app.writeLocalBinary(connection, envelope)+ },+ func(status windowcapture.Status) error {+ return app.writeLocal(connection, map[string]any{+ "kind": "native-video-state", "status": status,+ })+ },+ )+ } else {+ captureErr = app.captureProvider.CaptureAudio(captureContext, selected, writePCM)+ }if captureErr != nil && captureContext.Err() == nil {- _ = app.writeLocal(connection, map[string]string{- "kind": "audio-state", "state": "unavailable",- "message": "Target audio stopped; system audio was not substituted",- })+ if source == videoSourceNativeWindow {+ session.Fail(errors.New("native window capture stopped"))+ } else {+ _ = app.writeLocal(connection, map[string]string{+ "kind": "audio-state", "state": "unavailable",+ "message": "Target audio stopped; system audio was not substituted",+ })+ }}}(*target)} else {- close(audioDone)+ close(captureDone)}defer func() {+ app.mu.Lock()+ if app.generation == generation && app.keyFrameRequests == keyFrameRequests {+ app.keyFrameRequests = nil+ }+ app.mu.Unlock()cancelCapture()- <-audioDone+ <-captureDone}()for {messageType, payload, err = connection.Read(app.ctx)@@ -450,7 +523,9 @@ func (app *App) handleMedia(response http.ResponseWriter, request *http.Request)_ = connection.Close(websocket.StatusPolicyViolation, "invalid encoded media")return}- if packet.Kind != media.KindVideo && packet.Kind != media.KindOpus {+ validKind := packet.Kind == media.KindOpus ||+ (source == videoSourceBrowser && packet.Kind == media.KindVideo)+ if !validKind {session.Fail(errors.New("local media bridge received an invalid encoded media kind"))_ = connection.Close(websocket.StatusPolicyViolation, "encoded media required")return@@ -502,7 +577,14 @@ func (app *App) emit(generation uint64, event remote.Event) {}connection := app.mediaConnsession := app.session+ keyFrameRequests := app.keyFrameRequestsapp.mu.Unlock()+ if event.Kind == "request-keyframe" && keyFrameRequests != nil {+ select {+ case keyFrameRequests <- struct{}{}:+ default:+ }+ }if connection != nil {if err := app.writeLocal(connection, event); err != nil && event.Kind != "fatal" && session != nil {go session.Fail(fmt.Errorf("write local sender event: %w", err))@@ -568,7 +650,9 @@ func (app *App) stopActiveGeneration(expectedGeneration uint64) {}app.room = remote.Room{}app.codec = media.CodecVP8- app.audioTarget = nil+ app.videoSource = videoSourceBrowser+ app.windowTarget = nil+ app.keyFrameRequests = nilapp.activeViewers = 0app.waitingViewers = 0app.mu.Unlock()@@ -587,15 +671,30 @@ func (app *App) stopActiveGeneration(expectedGeneration uint64) {}func validEncoderConfig(config encoderConfig) bool {- return validEncoderConfigForCodec(config, media.CodecVP8)+ return validEncoderConfigForSource(config, media.CodecVP8, videoSourceBrowser)}func validEncoderConfigForCodec(config encoderConfig, expected media.Codec) bool {+ return validEncoderConfigForSource(config, expected, videoSourceBrowser)+}++func validEncoderConfigForSource(config encoderConfig, expected media.Codec, source videoSource) bool {return config.Kind == "config" && config.Codec == string(expected) &&+ config.VideoSource == string(source) &&config.Width == 1280 && config.Height == 720 && config.FPS == 30 &&config.Bitrate == 3_000_000 && config.EncoderInstances == 1}+func parseVideoSource(value string) (videoSource, bool) {+ if value == "" || value == string(videoSourceBrowser) {+ return videoSourceBrowser, true+ }+ if value == string(videoSourceNativeWindow) {+ return videoSourceNativeWindow, true+ }+ return videoSourceBrowser, false+}+func decodeLocalJSON(request *http.Request, target any) error {payload, err := io.ReadAll(io.LimitReader(request.Body, maxLocalJSONBytes+1))if err != nil || len(payload) > maxLocalJSONBytes {diff --git a/native/sender/internal/app/server_test.go b/native/sender/internal/app/server_test.goindex 830625e8..c464769f 100644--- a/native/sender/internal/app/server_test.go+++ b/native/sender/internal/app/server_test.go@@ -14,13 +14,13 @@ import ("github.com/TNTcraftHIM/Screener/native/sender/internal/media""github.com/TNTcraftHIM/Screener/native/sender/internal/remote"- "github.com/TNTcraftHIM/Screener/native/sender/internal/windowaudio"+ "github.com/TNTcraftHIM/Screener/native/sender/internal/windowcapture""github.com/coder/websocket")func TestEncoderConfigUsesTheFixedHighContract(t *testing.T) {config := encoderConfig{- Kind: "config", Codec: "vp8", Width: 1280, Height: 720,+ Kind: "config", Codec: "vp8", VideoSource: "browser", Width: 1280, Height: 720,FPS: 30, Bitrate: 3_000_000, EncoderInstances: 1,}if !validEncoderConfig(config) {@@ -34,7 +34,7 @@ func TestEncoderConfigUsesTheFixedHighContract(t *testing.T) {func TestEncoderConfigAcceptsExplicitH264OnlyForAnH264Session(t *testing.T) {config := encoderConfig{- Kind: "config", Codec: "h264", Width: 1280, Height: 720,+ Kind: "config", Codec: "h264", VideoSource: "browser", Width: 1280, Height: 720,FPS: 30, Bitrate: 3_000_000, EncoderInstances: 1,}if !validEncoderConfigForCodec(config, media.CodecH264) {@@ -45,14 +45,28 @@ func TestEncoderConfigAcceptsExplicitH264OnlyForAnH264Session(t *testing.T) {}}+func TestNativeWindowConfigRequiresExactH264Source(t *testing.T) {+ config := encoderConfig{+ Kind: "config", Codec: "h264", VideoSource: "native-window-h264",+ Width: 1280, Height: 720, FPS: 30, Bitrate: 3_000_000, EncoderInstances: 1,+ }+ if !validEncoderConfigForSource(config, media.CodecH264, videoSourceNativeWindow) {+ t.Fatal("native window H.264 config was rejected")+ }+ if validEncoderConfigForSource(config, media.CodecH264, videoSourceBrowser) ||+ validEncoderConfigForSource(config, media.CodecVP8, videoSourceNativeWindow) {+ t.Fatal("native window config crossed a source or codec boundary")+ }+}+func TestDecodeLocalPayloadIsStrictAndSingleValued(t *testing.T) {- valid := []byte(`{"kind":"config","codec":"vp8","width":1280,"height":720,"fps":30,"bitrate":3000000,"encoderInstances":1}`)+ valid := []byte(`{"kind":"config","codec":"vp8","videoSource":"browser","width":1280,"height":720,"fps":30,"bitrate":3000000,"encoderInstances":1}`)var config encoderConfigif err := decodeLocalPayload(valid, &config); err != nil {t.Fatal(err)}invalid := [][]byte{- []byte(`{"kind":"config","codec":"vp8","width":1280,"height":720,"fps":30,"bitrate":3000000,"encoderInstances":1,"extra":true}`),+ []byte(`{"kind":"config","codec":"vp8","videoSource":"browser","width":1280,"height":720,"fps":30,"bitrate":3000000,"encoderInstances":1,"extra":true}`),append(append([]byte(nil), valid...), valid...),}for _, payload := range invalid {@@ -62,20 +76,20 @@ func TestDecodeLocalPayloadIsStrictAndSingleValued(t *testing.T) {}}-func TestAudioTargetListingKeepsProcessIdentityBehindAnOpaqueLocalID(t *testing.T) {+func TestWindowTargetListingKeepsBoundIdentityBehindAnOpaqueLocalID(t *testing.T) {application, err := New()if err != nil {t.Fatal(err)}- application.audioProvider = fakeAudioProvider{targets: []windowaudio.Target{{- PID: 424_242, CreationTime: 987_654_321, Title: "Game Window",+ application.captureProvider = fakeCaptureProvider{targets: []windowcapture.Target{{+ WindowHandle: 123_456, PID: 424_242, CreationTime: 987_654_321, Title: "Game Window",}}}if _, err = application.Start(); err != nil {t.Fatal(err)}defer application.Close()- request, err := http.NewRequest(http.MethodPost, application.origin+"/api/audio-targets", nil)+ request, err := http.NewRequest(http.MethodPost, application.origin+"/api/window-targets", nil)if err != nil {t.Fatal(err)}@@ -87,21 +101,21 @@ func TestAudioTargetListingKeepsProcessIdentityBehindAnOpaqueLocalID(t *testing.}defer response.Body.Close()var result struct {- Supported bool `json:"supported"`- Targets []localAudioTarget `json:"targets"`+ Supported bool `json:"supported"`+ Targets []localWindowTarget `json:"targets"`}if json.NewDecoder(response.Body).Decode(&result) != nil || !result.Supported || len(result.Targets) != 1 {- t.Fatalf("audio target response = %+v", result)+ t.Fatalf("window target response = %+v", result)}if result.Targets[0].ID == "" || result.Targets[0].Title != "Game Window" ||strings.Contains(result.Targets[0].ID, "424242") || strings.Contains(result.Targets[0].ID, "987654321") {- t.Fatalf("audio target was not locally anonymized: %+v", result.Targets[0])+ t.Fatalf("window target was not locally anonymized: %+v", result.Targets[0])}application.mu.Lock()- resolved := application.audioTargets[result.Targets[0].ID]+ resolved := application.windowTargets[result.Targets[0].ID]application.mu.Unlock()- if resolved.PID != 424_242 || resolved.CreationTime != 987_654_321 {- t.Fatal("opaque target did not resolve in process memory")+ if resolved.WindowHandle != 123_456 || resolved.PID != 424_242 || resolved.CreationTime != 987_654_321 {+ t.Fatal("opaque target did not resolve to its bound identity in process memory")}}@@ -472,7 +486,7 @@ func openConfiguredMedia(t *testing.T, application *App) *websocket.Conn {}t.Cleanup(func() { _ = connection.CloseNow() })readLocalKind(t, connection, "ready")- config := []byte(`{"kind":"config","codec":"vp8","width":1280,"height":720,"fps":30,"bitrate":3000000,"encoderInstances":1}`)+ config := []byte(`{"kind":"config","codec":"vp8","videoSource":"browser","width":1280,"height":720,"fps":30,"bitrate":3000000,"encoderInstances":1}`)writeContext, cancelWrite := context.WithTimeout(context.Background(), time.Second)err = connection.Write(writeContext, websocket.MessageText, config)cancelWrite()@@ -689,12 +703,23 @@ func appRoomResponse(origin string) map[string]any {}}-type fakeAudioProvider struct{ targets []windowaudio.Target }+type fakeCaptureProvider struct{ targets []windowcapture.Target }-func (provider fakeAudioProvider) List(context.Context) ([]windowaudio.Target, error) {+func (provider fakeCaptureProvider) List(context.Context) ([]windowcapture.Target, error) {return provider.targets, nil}-func (fakeAudioProvider) Capture(context.Context, windowaudio.Target, func(windowaudio.PCMChunk) error) error {+func (fakeCaptureProvider) CaptureAudio(context.Context, windowcapture.Target, func(windowcapture.PCMChunk) error) error {+ return nil+}++func (fakeCaptureProvider) CaptureWindow(+ context.Context,+ windowcapture.Target,+ <-chan struct{},+ func(windowcapture.PCMChunk) error,+ func(windowcapture.H264AccessUnit) error,+ func(windowcapture.Status) error,+) error {return nil}diff --git a/native/sender/internal/app/ui/app.js b/native/sender/internal/app/ui/app.jsindex fa71672a..9a3206ab 100644--- a/native/sender/internal/app/ui/app.js+++ b/native/sender/internal/app/ui/app.js@@ -25,11 +25,12 @@ const elements = {serverURL: document.querySelector("#server-url"),password: document.querySelector("#password"),codec: document.querySelector("#codec"),- audioTarget: document.querySelector("#audio-target"),- refreshAudio: document.querySelector("#refresh-audio"),+ windowTarget: document.querySelector("#window-target"),+ refreshWindows: document.querySelector("#refresh-windows"),start: document.querySelector("#start"),stop: document.querySelector("#stop"),preview: document.querySelector("#preview"),+ nativePreview: document.querySelector("#native-preview"),empty: document.querySelector("#empty-state"),invite: document.querySelector("#invite"),inviteLink: document.querySelector("#invite-link"),@@ -48,6 +49,7 @@ const elements = {let stream = null;let processorReader = null;let encoder = null;+let videoDecoder = null;let audioEncoder = null;let silentAudioChunks = 0;let mediaSocket = null;@@ -59,10 +61,12 @@ let forceKeyFrame = true;let frameNumber = 0;let captureSettings = null;let nativeDiagnostics = null;+let nativeVideoStatus = null;let metrics = emptyMetrics();let startGeneration = 0;let diagnosticsTimer = null;let pendingStartGeneration = 0;+let activeVideoSource = "browser";if (!processToken) {showError("本地发送端令牌已失效,请重新启动 sender.exe");@@ -80,47 +84,71 @@ elements.copy.addEventListener("click", async () => {showError("无法复制链接,请直接打开邀请链接");}});-elements.refreshAudio.addEventListener("click", () => void loadAudioTargets());-if (processToken) void loadAudioTargets();+elements.refreshWindows.addEventListener("click", () => void loadWindowTargets());+if (processToken) void loadWindowTargets();async function startSharing() {if (sharing || stopping || startInFlight) return;showError("");- if (!("VideoEncoder" in window) || !("MediaStreamTrackProcessor" in window)) {+ const videoSource = selectedVideoSource();+ const nativeWindow = videoSource === "native-window-h264";+ if (!nativeWindow && (!("VideoEncoder" in window) || !("MediaStreamTrackProcessor" in window))) {showError("此浏览器缺少 WebCodecs 屏幕编码支持,请使用当前版 Chrome 或 Edge");return;}+ if (nativeWindow && !("VideoDecoder" in window)) {+ showError("此浏览器缺少原生 H.264 本地预览支持,请使用当前版 Chrome 或 Edge");+ return;+ }startInFlight = true;elements.start.disabled = true;const generation = ++startGeneration;const codec = selectedCodec();- const audioTargetId = elements.audioTarget.value;- if (audioTargetId && (!("AudioEncoder" in window) || !("AudioData" in window))) {+ metrics.codec = codec;+ const windowTargetId = elements.windowTarget.value;+ activeVideoSource = videoSource;+ if (nativeWindow && !windowTargetId) {+ showError("原生 H.264 模式需要选择一个目标窗口");+ startInFlight = false;+ elements.start.disabled = false;+ return;+ }+ if (windowTargetId && (!("AudioEncoder" in window) || !("AudioData" in window))) {showError("当前浏览器缺少 WebCodecs Opus 编码支持,请关闭目标进程音频或使用当前版 Chrome / Edge");startInFlight = false;elements.start.disabled = false;return;}- setStatus("选择屏幕");try {- const capturedStream = await navigator.mediaDevices.getDisplayMedia({- video: {- width: { ideal: WIDTH, max: WIDTH },- height: { ideal: HEIGHT, max: HEIGHT },- frameRate: { ideal: FPS, max: FPS },- },- audio: false,- });- assertCurrentStart(generation, mediaSocket, capturedStream);- stream = capturedStream;- const videoTrack = stream.getVideoTracks()[0];- if (!videoTrack) throw new Error("没有可用的视频轨道");- await videoTrack.applyConstraints({ width: WIDTH, height: HEIGHT, frameRate: FPS });- assertCurrentStart(generation);- captureSettings = videoTrack.getSettings();+ let videoTrack = null;+ if (nativeWindow) {+ await startNativeDecoder(generation);+ captureSettings = { width: WIDTH, height: HEIGHT, frameRate: FPS };+ elements.preview.hidden = true;+ elements.nativePreview.hidden = false;+ } else {+ setStatus("选择屏幕");+ const capturedStream = await navigator.mediaDevices.getDisplayMedia({+ video: {+ width: { ideal: WIDTH, max: WIDTH },+ height: { ideal: HEIGHT, max: HEIGHT },+ frameRate: { ideal: FPS, max: FPS },+ },+ audio: false,+ });+ assertCurrentStart(generation, mediaSocket, capturedStream);+ stream = capturedStream;+ videoTrack = stream.getVideoTracks()[0];+ if (!videoTrack) throw new Error("没有可用的视频轨道");+ await videoTrack.applyConstraints({ width: WIDTH, height: HEIGHT, frameRate: FPS });+ assertCurrentStart(generation);+ captureSettings = videoTrack.getSettings();+ videoTrack.addEventListener("ended", () => void stopSharing(generation), { once: true });+ elements.preview.srcObject = stream;+ elements.preview.hidden = false;+ elements.nativePreview.hidden = true;+ }updateDiagnostics();- videoTrack.addEventListener("ended", () => void stopSharing(generation), { once: true });- elements.preview.srcObject = stream;elements.empty.hidden = true;setStatus("创建房间");@@ -129,7 +157,8 @@ async function startSharing() {serverUrl: elements.serverURL.value,password: elements.password.value,codec,- audioTargetId,+ videoSource,+ windowTargetId,});assertCurrentStart(generation);if (pendingStartGeneration === generation) pendingStartGeneration = 0;@@ -139,15 +168,18 @@ async function startSharing() {elements.inviteLink.textContent = room.inviteUrl;elements.invite.hidden = false;- if (audioTargetId) await startAudioEncoder(generation);- await connectMediaSocket(generation, codec, Boolean(audioTargetId));- assertCurrentStart(generation);- const activeEncoder = await startEncoder(generation, codec);+ if (windowTargetId) await startAudioEncoder(generation);+ await connectMediaSocket(generation, codec, videoSource, Boolean(windowTargetId));assertCurrentStart(generation);+ let activeEncoder = null;+ if (!nativeWindow) {+ activeEncoder = await startEncoder(generation, codec);+ assertCurrentStart(generation);+ }sharing = true;elements.stop.hidden = false;setStatus("正在分享");- void readFrames(videoTrack, generation, activeEncoder, mediaSocket);+ if (!nativeWindow) void readFrames(videoTrack, generation, activeEncoder, mediaSocket);} catch (error) {if (generation === startGeneration && !(error instanceof StaleStartError)) {showError(readableError(error, "无法开始分享"));@@ -159,7 +191,7 @@ async function startSharing() {}}-async function connectMediaSocket(generation, codec, audio) {+async function connectMediaSocket(generation, codec, videoSource, audio) {const url = new URL("/media", location.href);url.protocol = "ws:";const socket = new WebSocket(url, [`screener.token.${processToken}`]);@@ -181,6 +213,7 @@ async function connectMediaSocket(generation, codec, audio) {socket.send(JSON.stringify({kind: "config",codec,+ videoSource,width: WIDTH,height: HEIGHT,fps: FPS,@@ -233,6 +266,34 @@ async function startEncoder(generation, codec) {return activeEncoder;}+async function startNativeDecoder(generation) {+ const config = {+ codec: "avc1.42c01f",+ codedWidth: WIDTH,+ codedHeight: HEIGHT,+ optimizeForLatency: true,+ };+ const support = await VideoDecoder.isConfigSupported(config);+ assertCurrentStart(generation);+ if (!support.supported) throw new Error("此浏览器不支持 H.264 42c01f 本地预览");+ let activeDecoder;+ activeDecoder = new VideoDecoder({+ output: (frame) => {+ try {+ if (generation === startGeneration && videoDecoder === activeDecoder) {+ const context = elements.nativePreview.getContext("2d");+ context?.drawImage(frame, 0, 0, WIDTH, HEIGHT);+ }+ } finally {+ frame.close();+ }+ },+ error: (error) => void failSharing(generation, `原生 H.264 本地预览失败:${error.message}`),+ });+ videoDecoder = activeDecoder;+ activeDecoder.configure(support.config);+}+async function readFrames(videoTrack, generation, activeEncoder, activeSocket) {const processor = new MediaStreamTrackProcessor({ track: videoTrack });const reader = processor.readable.getReader();@@ -319,7 +380,14 @@ function sendEncodedChunk(generation, activeEncoder, chunk) {function handleLocalEvent(generation, socket, event) {if (generation !== startGeneration || socket !== mediaSocket) return;if (event.data instanceof ArrayBuffer) {- handlePCM(generation, event.data);+ const kind = event.data.byteLength > 1 ? new DataView(event.data).getUint8(1) : 0;+ if (kind === MEDIA_KIND_PCM) {+ handlePCM(generation, event.data);+ } else if (kind === MEDIA_KIND_VIDEO && activeVideoSource === "native-window-h264") {+ handleNativeVideo(generation, event.data);+ } else {+ void failSharing(generation, "本地媒体桥接返回了意外的二进制数据");+ }return;}if (typeof event.data !== "string") return;@@ -339,11 +407,40 @@ function handleLocalEvent(generation, socket, event) {? "不可用(未回退系统声音)": "正在连接目标进程";}+ if (message.kind === "native-video-state") {+ nativeVideoStatus = { ...nativeVideoStatus, ...message.status };+ updateDiagnostics();+ setStatus(message.status?.state === "active" ? "正在分享(原生 H.264)" : "正在启动原生硬件采集");+ }if (message.kind === "fatal") {void failSharing(generation, message.message || "分享会话已失败并停止");}}+function handleNativeVideo(generation, message) {+ if (generation !== startGeneration || !videoDecoder || videoDecoder.state !== "configured") return;+ try {+ const view = new DataView(message);+ const flags = view.getUint8(2);+ if (message.byteLength <= ENVELOPE_HEADER_BYTES || view.getUint8(0) !== 1 ||+ view.getUint8(1) !== MEDIA_KIND_VIDEO || flags > 1 || view.getUint8(3) !== 0) {+ throw new Error("本地 H.264 envelope 无效");+ }+ const data = new Uint8Array(message, ENVELOPE_HEADER_BYTES);+ if (data.byteLength < 1 || data.byteLength > MAX_FRAME_BYTES) {+ throw new Error("本地 H.264 access unit 长度无效");+ }+ videoDecoder.decode(new EncodedVideoChunk({+ type: flags === 1 ? "key" : "delta",+ timestamp: Number(view.getBigUint64(4, false) / 10n),+ duration: Number(view.getBigUint64(12, false) / 10n),+ data,+ }));+ } catch (error) {+ void failSharing(generation, readableError(error, "处理本地 H.264 预览失败"));+ }+}+async function failSharing(generation, message) {if (generation !== startGeneration || stopping) return;showError(message);@@ -381,6 +478,15 @@ async function stopSharing(expectedGeneration = startGeneration) {rememberError(error);}encoder = null;+ try {+ if (videoDecoder && videoDecoder.state !== "closed") videoDecoder.close();+ } catch (error) {+ rememberError(error);+ }+ videoDecoder = null;+ elements.nativePreview.getContext("2d")?.clearRect(0, 0, WIDTH, HEIGHT);+ elements.nativePreview.hidden = true;+ elements.preview.hidden = false;try {if (audioEncoder && audioEncoder.state !== "closed") audioEncoder.close();} catch (error) {@@ -408,6 +514,8 @@ async function stopSharing(expectedGeneration = startGeneration) {silentAudioChunks = 0;captureSettings = null;nativeDiagnostics = null;+ nativeVideoStatus = null;+ activeVideoSource = "browser";metrics = emptyMetrics();if (diagnosticsTimer !== null) {clearTimeout(diagnosticsTimer);@@ -488,25 +596,25 @@ async function localRequest(path, body) {return result;}-async function loadAudioTargets() {- elements.refreshAudio.disabled = true;- const previous = elements.audioTarget.value;+async function loadWindowTargets() {+ elements.refreshWindows.disabled = true;+ const previous = elements.windowTarget.value;try {- const result = await localRequest("/api/audio-targets");- elements.audioTarget.replaceChildren(new Option("关闭", ""));+ const result = await localRequest("/api/window-targets");+ elements.windowTarget.replaceChildren(new Option("关闭窗口音频", ""));for (const target of Array.isArray(result.targets) ? result.targets : []) {- if (target?.id && target?.title) elements.audioTarget.add(new Option(target.title, target.id));+ if (target?.id && target?.title) elements.windowTarget.add(new Option(target.title, target.id));}- if ([...elements.audioTarget.options].some((option) => option.value === previous)) {- elements.audioTarget.value = previous;+ if ([...elements.windowTarget.options].some((option) => option.value === previous)) {+ elements.windowTarget.value = previous;}- elements.audioTarget.disabled = !result.supported;+ elements.windowTarget.disabled = !result.supported;elements.audioStatus.textContent = result.supported ? "关闭" : "Windows 11 helper 不可用";} catch (error) {- elements.audioTarget.disabled = true;+ elements.windowTarget.disabled = true;elements.audioStatus.textContent = readableError(error, "窗口列表不可用");} finally {- elements.refreshAudio.disabled = false;+ elements.refreshWindows.disabled = false;}}@@ -595,7 +703,16 @@ function updateDiagnostics() {const height = captureSettings?.height ?? "未知";const rate = captureSettings?.frameRate ?? "未知";elements.captureDiagnostic.textContent = `${width} × ${height} · ${rate} FPS`;- elements.encoderDiagnostic.textContent = `${codecLabel(metrics.codec)} · 3 Mbps 上限 · 单对象 · 硬件偏好 ${hardwarePreferenceLabel(metrics.hardwarePreference)} · 硬件证据 ${hardwareEvidenceLabel(metrics.hardwareEvidence)}`;+ if (activeVideoSource === "native-window-h264") {+ const mftState = nativeVideoStatus?.state === "active"+ ? "hardware-only MFT 已绑定并输出"+ : nativeVideoStatus?.state === "starting"+ ? "hardware-only MFT 已绑定,等待首帧"+ : "hardware-only MFT 等待状态";+ elements.encoderDiagnostic.textContent = `H.264 · 3 Mbps 上限 · 单 MF 编码器 · ${mftState}`;+ } else {+ elements.encoderDiagnostic.textContent = `${codecLabel(metrics.codec)} · 3 Mbps 上限 · 单对象 · 硬件偏好 ${hardwarePreferenceLabel(metrics.hardwarePreference)} · 硬件证据 ${hardwareEvidenceLabel(metrics.hardwareEvidence)}`;+ }elements.bridgeDiagnostic.textContent = `读取 ${metrics.framesRead} · 提交 ${metrics.framesSubmitted} · 输出 ${metrics.encoderOutputs} · 编码队列丢弃 ${metrics.encoderQueueDrops} · 编码队列峰值 ${metrics.encoderQueuePeak} 帧 · socket 峰值 ${metrics.socketBufferedPeak} B`;const helper = nativeDiagnostics?.media;const queue = helper?.queue;@@ -645,7 +762,11 @@ function boundedCount(value) {}function selectedCodec() {- return elements.codec?.value === "h264" ? "h264" : "vp8";+ return elements.codec?.value === "h264" || elements.codec?.value === "native-h264" ? "h264" : "vp8";+}++function selectedVideoSource() {+ return elements.codec?.value === "native-h264" ? "native-window-h264" : "browser";}function emptyMetrics() {diff --git a/native/sender/internal/app/ui/index.html b/native/sender/internal/app/ui/index.htmlindex 2a75ffaa..51919583 100644--- a/native/sender/internal/app/ui/index.html+++ b/native/sender/internal/app/ui/index.html@@ -22,6 +22,7 @@</div><div class="preview-shell"><video id="preview" autoplay muted playsinline></video>+ <canvas id="native-preview" width="1280" height="720" hidden></canvas><div id="empty-state" class="empty-state">选择屏幕后,邀请链接会显示在这里</div></div><div id="invite" class="invite" hidden>@@ -43,16 +44,17 @@<select id="codec"><option value="vp8">VP8 (default)</option><option value="h264">H.264 (hardware preference)</option>+ <option value="native-h264">H.264 (native window hardware)</option></select></label><label>- <span>目标进程音频</span>+ <span>目标窗口</span><span class="select-row">- <select id="audio-target"><option value="">关闭</option></select>- <button id="refresh-audio" class="icon-button" type="button" title="刷新窗口列表" aria-label="刷新窗口列表">↻</button>+ <select id="window-target"><option value="">关闭窗口音频</option></select>+ <button id="refresh-windows" class="icon-button" type="button" title="刷新窗口列表" aria-label="刷新窗口列表">↻</button></span></label>- <p class="audio-note">请在浏览器画面选择器中选择同一目标。音频只包含所选进程树,失败时不会改用系统混音。</p>+ <p class="audio-note">浏览器画面模式下请在系统选择器中选择同一窗口。原生模式直接采集这里选择的窗口;音频只包含其进程树,失败时不会改用系统混音。</p><button id="start" class="button primary" type="button">开始分享</button><p id="error" class="error" role="alert"></p><details>diff --git a/native/sender/internal/app/ui/styles.css b/native/sender/internal/app/ui/styles.cssindex 7d6da525..769d9480 100644--- a/native/sender/internal/app/ui/styles.css+++ b/native/sender/internal/app/ui/styles.css@@ -37,7 +37,7 @@ main { display: grid; grid-template-columns: minmax(0, 1fr) 340px; width: min(14h1 { margin: 0; font-size: 24px; }.stage-heading p { margin: 7px 0 0; color: var(--muted); font-size: 13px; }.preview-shell { position: relative; display: grid; width: 100%; margin-top: 18px; overflow: hidden; aspect-ratio: 16 / 9; place-items: center; border: 1px solid var(--border); border-radius: 6px; background: #080a08; }-video { display: block; width: 100%; height: 100%; object-fit: contain; }+video, canvas { display: block; width: 100%; height: 100%; object-fit: contain; }.empty-state { position: absolute; color: var(--muted); font-size: 14px; }.invite { display: grid; grid-template-columns: minmax(0, 1fr) 42px; gap: 8px; margin-top: 14px; }.invite a { min-width: 0; overflow: hidden; padding: 11px 12px; border: 1px solid var(--border); border-radius: 6px; color: #c8f7d9; text-overflow: ellipsis; white-space: nowrap; }diff --git a/native/sender/internal/windowaudio/provider.go b/native/sender/internal/windowaudio/provider.godeleted file mode 100644index 0704e735..00000000--- a/native/sender/internal/windowaudio/provider.go+++ /dev/null@@ -1,25 +0,0 @@-package windowaudio--import (- "context"- "errors"-)--var ErrUnavailable = errors.New("Windows 11 process audio is unavailable")--type Target struct {- PID uint32- CreationTime uint64- Title string-}--type PCMChunk struct {- Timestamp100ns uint64- Duration100ns uint64- Data []byte-}--type Provider interface {- List(context.Context) ([]Target, error)- Capture(context.Context, Target, func(PCMChunk) error) error-}diff --git a/native/sender/internal/windowaudio/provider_other.go b/native/sender/internal/windowaudio/provider_other.godeleted file mode 100644index 7fb74cf6..00000000--- a/native/sender/internal/windowaudio/provider_other.go+++ /dev/null@@ -1,15 +0,0 @@-//go:build !windows--package windowaudio--import "context"--type unsupportedProvider struct{}--func NewProvider() Provider { return unsupportedProvider{} }--func (unsupportedProvider) List(context.Context) ([]Target, error) { return nil, ErrUnavailable }--func (unsupportedProvider) Capture(context.Context, Target, func(PCMChunk) error) error {- return ErrUnavailable-}diff --git a/native/sender/internal/windowaudio/provider_windows.go b/native/sender/internal/windowaudio/provider_windows.godeleted file mode 100644index 3fa6433f..00000000--- a/native/sender/internal/windowaudio/provider_windows.go+++ /dev/null@@ -1,145 +0,0 @@-//go:build windows--package windowaudio--import (- "context"- "encoding/binary"- "encoding/json"- "errors"- "io"- "os"- "os/exec"- "path/filepath"- "strconv"- "strings"-- "golang.org/x/sys/windows"-)--const (- maxListBytes = 64 << 10- maxTargets = 100- helperHeaderSize = 28-)--type windowsProvider struct{ helper string }--type helperTarget struct {- PID uint32 `json:"pid"`- CreationTime uint64 `json:"creationTime"`- Title string `json:"title"`-}--func NewProvider() Provider {- helper := strings.TrimSpace(os.Getenv("SCREENER_PROCESS_AUDIO_HELPER"))- if helper == "" {- if executable, err := os.Executable(); err == nil {- helper = filepath.Join(filepath.Dir(executable), "screener-process-audio.exe")- }- }- return &windowsProvider{helper: helper}-}--func (provider *windowsProvider) available() bool {- version := windows.RtlGetVersion()- if version.MajorVersion < 10 || version.BuildNumber < 22_000 || provider.helper == "" {- return false- }- info, err := os.Stat(provider.helper)- return err == nil && !info.IsDir()-}--func (provider *windowsProvider) List(ctx context.Context) ([]Target, error) {- if !provider.available() {- return nil, ErrUnavailable- }- command := exec.CommandContext(ctx, provider.helper, "--list")- stdout, err := command.StdoutPipe()- if err != nil {- return nil, ErrUnavailable- }- command.Stderr = io.Discard- if err = command.Start(); err != nil {- return nil, ErrUnavailable- }- payload, readErr := io.ReadAll(io.LimitReader(stdout, maxListBytes+1))- if (readErr != nil || len(payload) > maxListBytes) && ctx.Err() == nil {- _ = command.Process.Kill()- }- waitErr := command.Wait()- if readErr != nil || waitErr != nil || len(payload) > maxListBytes {- return nil, ErrUnavailable- }- var listed []helperTarget- if json.Unmarshal(payload, &listed) != nil || len(listed) > maxTargets {- return nil, ErrUnavailable- }- targets := make([]Target, 0, len(listed))- for _, candidate := range listed {- title := strings.TrimSpace(candidate.Title)- if candidate.PID == 0 || candidate.CreationTime == 0 || title == "" || len(title) > 512 {- continue- }- targets = append(targets, Target{- PID: candidate.PID, CreationTime: candidate.CreationTime, Title: title,- })- }- return targets, nil-}--func (provider *windowsProvider) Capture(ctx context.Context, target Target, onChunk func(PCMChunk) error) error {- if !provider.available() || target.PID == 0 || target.CreationTime == 0 || onChunk == nil {- return ErrUnavailable- }- command := exec.CommandContext(ctx, provider.helper, "--capture",- strconv.FormatUint(uint64(target.PID), 10), strconv.FormatUint(target.CreationTime, 10))- stdout, err := command.StdoutPipe()- if err != nil {- return ErrUnavailable- }- command.Stderr = io.Discard- if err = command.Start(); err != nil {- return ErrUnavailable- }- started := false- for {- header := make([]byte, helperHeaderSize)- _, readErr := io.ReadFull(stdout, header)- if readErr != nil {- err = readErr- break- }- length := binary.BigEndian.Uint32(header[24:28])- if string(header[:4]) != "SPCM" || header[4] != 1 || header[5] != 0 ||- header[6] != 0 || header[7] != 0 || length == 0 || length > 48_000*2*2/10 || length%4 != 0 {- err = errors.New("process audio helper returned an invalid PCM header")- break- }- data := make([]byte, length)- if _, readErr = io.ReadFull(stdout, data); readErr != nil {- err = readErr- break- }- started = true- if err = onChunk(PCMChunk{- Timestamp100ns: binary.BigEndian.Uint64(header[8:16]),- Duration100ns: binary.BigEndian.Uint64(header[16:24]),- Data: data,- }); err != nil {- _ = command.Process.Kill()- break- }- }- if err != nil && ctx.Err() == nil {- _ = command.Process.Kill()- }- _ = command.Wait()- if ctx.Err() != nil {- return nil- }- if err != nil || !started {- return errors.New("target-process audio capture stopped")- }- return errors.New("target process exited")-}diff --git a/native/sender/internal/windowaudio/provider_windows_test.go b/native/sender/internal/windowaudio/provider_windows_test.godeleted file mode 100644index 7c7a6360..00000000--- a/native/sender/internal/windowaudio/provider_windows_test.go+++ /dev/null@@ -1,100 +0,0 @@-//go:build windows--package windowaudio--import (- "context"- "encoding/binary"- "flag"- "os"- "testing"- "time"-)--const testCreationTime = uint64(123456)--var (- captureHelperPID = flag.String("capture", "", "run as the process-audio capture test helper")- listHelper = flag.Bool("list", false, "run as the process-audio list test helper")-)--func TestMain(main *testing.M) {- flag.Parse()- if *listHelper {- _, _ = os.Stdout.Write(make([]byte, maxListBytes+1))- select {}- }- if *captureHelperPID != "" {- if len(flag.Args()) != 1 || flag.Args()[0] != "123456" {- os.Exit(3)- }- header := make([]byte, helperHeaderSize)- copy(header, "SPCM")- header[4] = 1- binary.BigEndian.PutUint64(header[16:24], 200_000)- binary.BigEndian.PutUint32(header[24:28], 4)- _, _ = os.Stdout.Write(header)- _, _ = os.Stdout.Write(make([]byte, 4))- _, _ = os.Stdout.Write(make([]byte, helperHeaderSize))- select {}- }- os.Exit(main.Run())-}--func TestCapturePassesIdentityAndKillsHelperBeforeWaitingOnInvalidHeader(t *testing.T) {- provider := &windowsProvider{helper: os.Args[0]}- if !provider.available() {- t.Skip("requires Windows 11")- }-- ctx, cancel := context.WithCancel(context.Background())- result := make(chan error, 1)- chunks := 0- go func() {- result <- provider.Capture(ctx, Target{PID: 1, CreationTime: testCreationTime}, func(PCMChunk) error {- chunks++- return nil- })- }()-- select {- case err := <-result:- if err == nil {- t.Fatal("Capture returned nil after an invalid helper header")- }- if chunks != 1 {- t.Fatalf("Capture delivered %d chunks; process identity was not passed to the helper", chunks)- }- case <-time.After(2 * time.Second):- cancel()- <-result- t.Fatal("Capture waited on a running helper after an invalid header")- }- cancel()-}--func TestListKillsHelperBeforeWaitingOnOversizeOutput(t *testing.T) {- provider := &windowsProvider{helper: os.Args[0]}- if !provider.available() {- t.Skip("requires Windows 11")- }-- ctx, cancel := context.WithCancel(context.Background())- result := make(chan error, 1)- go func() {- _, err := provider.List(ctx)- result <- err- }()-- select {- case err := <-result:- if err == nil {- t.Fatal("List accepted oversized helper output")- }- case <-time.After(2 * time.Second):- cancel()- <-result- t.Fatal("List waited on a running helper after oversized output")- }- cancel()-}diff --git a/native/sender/internal/windowcapture/provider.go b/native/sender/internal/windowcapture/provider.gonew file mode 100644index 00000000..5c8a0cbf--- /dev/null+++ b/native/sender/internal/windowcapture/provider.go@@ -0,0 +1,56 @@+package windowcapture++import (+ "context"+ "errors"+)++var ErrUnavailable = errors.New("Windows 11 window capture is unavailable")++type Target struct {+ WindowHandle uint64+ PID uint32+ CreationTime uint64+ Title string+}++type PCMChunk struct {+ Timestamp100ns uint64+ Duration100ns uint64+ Data []byte+}++type H264AccessUnit struct {+ KeyFrame bool+ Timestamp100ns uint64+ Duration100ns uint64+ Data []byte+}++type Status struct {+ State string `json:"state"`+ HardwareOnly bool `json:"hardwareOnly"`+ AdapterIndex *uint `json:"adapterIndex,omitempty"`+ AdapterName string `json:"adapterName,omitempty"`+ AdapterLUID string `json:"adapterLuid,omitempty"`+ MFTIndex *uint `json:"mftIndex,omitempty"`+ MFTName string `json:"mftName,omitempty"`+ MFTCLSID string `json:"mftClsid,omitempty"`+ ProfileLevelID string `json:"profileLevelId,omitempty"`+ Width int `json:"width,omitempty"`+ Height int `json:"height,omitempty"`+ FPS int `json:"fps,omitempty"`+}++type Provider interface {+ List(context.Context) ([]Target, error)+ CaptureAudio(context.Context, Target, func(PCMChunk) error) error+ CaptureWindow(+ context.Context,+ Target,+ <-chan struct{},+ func(PCMChunk) error,+ func(H264AccessUnit) error,+ func(Status) error,+ ) error+}diff --git a/native/sender/internal/windowcapture/provider_other.go b/native/sender/internal/windowcapture/provider_other.gonew file mode 100644index 00000000..7fad201d--- /dev/null+++ b/native/sender/internal/windowcapture/provider_other.go@@ -0,0 +1,26 @@+//go:build !windows++package windowcapture++import "context"++type unsupportedProvider struct{}++func NewProvider() Provider { return unsupportedProvider{} }++func (unsupportedProvider) List(context.Context) ([]Target, error) { return nil, ErrUnavailable }++func (unsupportedProvider) CaptureAudio(context.Context, Target, func(PCMChunk) error) error {+ return ErrUnavailable+}++func (unsupportedProvider) CaptureWindow(+ context.Context,+ Target,+ <-chan struct{},+ func(PCMChunk) error,+ func(H264AccessUnit) error,+ func(Status) error,+) error {+ return ErrUnavailable+}diff --git a/native/sender/internal/windowcapture/provider_windows.go b/native/sender/internal/windowcapture/provider_windows.gonew file mode 100644index 00000000..dc0b3527--- /dev/null+++ b/native/sender/internal/windowcapture/provider_windows.go@@ -0,0 +1,307 @@+//go:build windows++package windowcapture++import (+ "bytes"+ "context"+ "encoding/binary"+ "encoding/json"+ "errors"+ "io"+ "os"+ "os/exec"+ "path/filepath"+ "strconv"+ "strings"++ "golang.org/x/sys/windows"+)++const (+ maxListBytes = 64 << 10+ maxTargets = 100+ helperHeaderSize = 28+ maxStatusBytes = 4 << 10+ maxMediaBytes = 1 << 20+ maxPCMBytes = 48_000 * 2 * 2 / 10+ helperKindPCM = 1+ helperKindH264 = 2+ helperKindStatus = 3+ helperProtocolMagic = "SMED"+)++type windowsProvider struct {+ helper string+ adapterIndex uint+ mftIndex uint+}++type helperTarget struct {+ WindowHandle uint64 `json:"windowHandle"`+ PID uint32 `json:"pid"`+ CreationTime uint64 `json:"creationTime"`+ Title string `json:"title"`+}++func NewProvider() Provider {+ helper := strings.TrimSpace(os.Getenv("SCREENER_WINDOW_CAPTURE_HELPER"))+ if helper == "" {+ if executable, err := os.Executable(); err == nil {+ helper = filepath.Join(filepath.Dir(executable), "screener-window-capture.exe")+ }+ }+ return &windowsProvider{+ helper: helper,+ adapterIndex: parseOptionalIndex("SCREENER_WINDOW_CAPTURE_ADAPTER_INDEX"),+ mftIndex: parseOptionalIndex("SCREENER_WINDOW_CAPTURE_MFT_INDEX"),+ }+}++func parseOptionalIndex(name string) uint {+ value, err := strconv.ParseUint(strings.TrimSpace(os.Getenv(name)), 10, 32)+ if err != nil {+ return 0+ }+ return uint(value)+}++func (provider *windowsProvider) available() bool {+ version := windows.RtlGetVersion()+ if version.MajorVersion < 10 || version.BuildNumber < 22_000 || provider.helper == "" {+ return false+ }+ info, err := os.Stat(provider.helper)+ return err == nil && !info.IsDir()+}++func (provider *windowsProvider) List(ctx context.Context) ([]Target, error) {+ if !provider.available() {+ return nil, ErrUnavailable+ }+ command := exec.CommandContext(ctx, provider.helper, "--list")+ stdout, err := command.StdoutPipe()+ if err != nil {+ return nil, ErrUnavailable+ }+ command.Stderr = io.Discard+ if err = command.Start(); err != nil {+ return nil, ErrUnavailable+ }+ payload, readErr := io.ReadAll(io.LimitReader(stdout, maxListBytes+1))+ if (readErr != nil || len(payload) > maxListBytes) && ctx.Err() == nil {+ _ = command.Process.Kill()+ }+ waitErr := command.Wait()+ if readErr != nil || waitErr != nil || len(payload) > maxListBytes {+ return nil, ErrUnavailable+ }+ var listed []helperTarget+ decoder := json.NewDecoder(bytes.NewReader(payload))+ decoder.DisallowUnknownFields()+ if decoder.Decode(&listed) != nil || decoder.Decode(&struct{}{}) != io.EOF || len(listed) > maxTargets {+ return nil, ErrUnavailable+ }+ targets := make([]Target, 0, len(listed))+ for _, candidate := range listed {+ title := strings.TrimSpace(candidate.Title)+ if candidate.WindowHandle == 0 || candidate.PID == 0 || candidate.CreationTime == 0 ||+ title == "" || len(title) > 512 {+ continue+ }+ targets = append(targets, Target{+ WindowHandle: candidate.WindowHandle,+ PID: candidate.PID,+ CreationTime: candidate.CreationTime,+ Title: title,+ })+ }+ return targets, nil+}++func (provider *windowsProvider) CaptureAudio(+ ctx context.Context,+ target Target,+ onPCM func(PCMChunk) error,+) error {+ return provider.capture(ctx, target,+ []string{"--capture-audio", targetPID(target), targetCreationTime(target)},+ nil, onPCM, nil, nil)+}++func (provider *windowsProvider) CaptureWindow(+ ctx context.Context,+ target Target,+ keyFrames <-chan struct{},+ onPCM func(PCMChunk) error,+ onH264 func(H264AccessUnit) error,+ onStatus func(Status) error,+) error {+ if keyFrames == nil || onPCM == nil || onH264 == nil || onStatus == nil {+ return ErrUnavailable+ }+ arguments := []string{+ "--capture-window", targetPID(target), targetCreationTime(target),+ strconv.FormatUint(target.WindowHandle, 10),+ "--adapter-index", strconv.FormatUint(uint64(provider.adapterIndex), 10),+ "--mft-index", strconv.FormatUint(uint64(provider.mftIndex), 10),+ "--protocol-v1",+ }+ return provider.capture(ctx, target, arguments, keyFrames, onPCM, onH264, onStatus)+}++func targetPID(target Target) string { return strconv.FormatUint(uint64(target.PID), 10) }++func targetCreationTime(target Target) string { return strconv.FormatUint(target.CreationTime, 10) }++func (provider *windowsProvider) capture(+ ctx context.Context,+ target Target,+ arguments []string,+ keyFrames <-chan struct{},+ onPCM func(PCMChunk) error,+ onH264 func(H264AccessUnit) error,+ onStatus func(Status) error,+) error {+ windowMode := onH264 != nil && onStatus != nil+ if !provider.available() || target.PID == 0 || target.CreationTime == 0 || onPCM == nil ||+ (windowMode && target.WindowHandle == 0) {+ return ErrUnavailable+ }+ command := exec.CommandContext(ctx, provider.helper, arguments...)+ stdout, err := command.StdoutPipe()+ if err != nil {+ return ErrUnavailable+ }+ var stdin io.WriteCloser+ if windowMode {+ stdin, err = command.StdinPipe()+ if err != nil {+ return ErrUnavailable+ }+ }+ command.Stderr = io.Discard+ if err = command.Start(); err != nil {+ return ErrUnavailable+ }+ keyFrameDone := make(chan struct{})+ keyFrameStop := make(chan struct{})+ if windowMode {+ go func() {+ defer close(keyFrameDone)+ defer stdin.Close()+ for {+ select {+ case <-ctx.Done():+ return+ case <-keyFrameStop:+ return+ case _, ok := <-keyFrames:+ if !ok {+ return+ }+ if _, writeErr := stdin.Write([]byte{'K'}); writeErr != nil {+ return+ }+ }+ }+ }()+ } else {+ close(keyFrameDone)+ }++ startedPCM := false+ startedH264 := false+ for {+ header := make([]byte, helperHeaderSize)+ if _, err = io.ReadFull(stdout, header); err != nil {+ break+ }+ kind := header[5]+ flags := header[6]+ timestamp := binary.BigEndian.Uint64(header[8:16])+ duration := binary.BigEndian.Uint64(header[16:24])+ length := binary.BigEndian.Uint32(header[24:28])+ if string(header[:4]) != helperProtocolMagic || header[4] != 1 || header[7] != 0 ||+ length == 0 || !validHelperHeader(kind, flags, timestamp, duration, length, windowMode) {+ err = errors.New("window capture helper returned an invalid media header")+ break+ }+ data := make([]byte, length)+ if _, err = io.ReadFull(stdout, data); err != nil {+ break+ }+ switch kind {+ case helperKindPCM:+ startedPCM = true+ err = onPCM(PCMChunk{Timestamp100ns: timestamp, Duration100ns: duration, Data: data})+ case helperKindH264:+ startedH264 = true+ err = onH264(H264AccessUnit{+ KeyFrame: flags == 1, Timestamp100ns: timestamp, Duration100ns: duration, Data: data,+ })+ case helperKindStatus:+ var status Status+ err = decodeStatus(data, &status)+ if err == nil {+ err = onStatus(status)+ }+ }+ if err != nil {+ break+ }+ }+ if err != nil && ctx.Err() == nil {+ _ = command.Process.Kill()+ }+ _ = command.Wait()+ if windowMode {+ close(keyFrameStop)+ _ = stdin.Close()+ }+ <-keyFrameDone+ if ctx.Err() != nil {+ return nil+ }+ if err != nil || (windowMode && !startedH264) || (!windowMode && !startedPCM) {+ return errors.New("selected window capture stopped")+ }+ return errors.New("selected target process exited")+}++func validHelperHeader(kind, flags byte, timestamp, duration uint64, length uint32, windowMode bool) bool {+ switch kind {+ case helperKindPCM:+ return flags == 0 && duration > 0 && duration <= 10_000_000 &&+ length <= maxPCMBytes && length%4 == 0+ case helperKindH264:+ return windowMode && flags <= 1 && timestamp > 0 && duration >= 10 &&+ duration <= 10_000_000 && duration%10 == 0 && length <= maxMediaBytes+ case helperKindStatus:+ return windowMode && flags == 0 && timestamp == 0 && duration == 0 && length <= maxStatusBytes+ default:+ return false+ }+}++func decodeStatus(payload []byte, status *Status) error {+ decoder := json.NewDecoder(bytes.NewReader(payload))+ decoder.DisallowUnknownFields()+ if decoder.Decode(status) != nil || decoder.Decode(&struct{}{}) != io.EOF || !status.HardwareOnly {+ return errors.New("window capture helper returned invalid status")+ }+ switch status.State {+ case "starting":+ if status.AdapterIndex == nil || status.MFTIndex == nil || status.AdapterName == "" ||+ status.AdapterLUID == "" || status.MFTName == "" || status.MFTCLSID == "" {+ return errors.New("window capture helper returned incomplete hardware status")+ }+ case "active":+ if status.ProfileLevelID != "42c01f" || status.Width != 1280 || status.Height != 720 || status.FPS != 30 {+ return errors.New("window capture helper returned unexpected video status")+ }+ default:+ return errors.New("window capture helper returned unknown status")+ }+ return nil+}diff --git a/native/sender/internal/windowcapture/provider_windows_test.go b/native/sender/internal/windowcapture/provider_windows_test.gonew file mode 100644index 00000000..e54c21c6--- /dev/null+++ b/native/sender/internal/windowcapture/provider_windows_test.go@@ -0,0 +1,149 @@+//go:build windows++package windowcapture++import (+ "context"+ "encoding/binary"+ "io"+ "os"+ "reflect"+ "testing"+ "time"+)++const testCreationTime = uint64(123456)+const testWindowHandle = uint64(987654)++func TestMain(main *testing.M) {+ if len(os.Args) > 1 {+ switch os.Args[1] {+ case "--list":+ _, _ = os.Stdout.Write(make([]byte, maxListBytes+1))+ select {}+ case "--capture-audio":+ if !reflect.DeepEqual(os.Args[2:], []string{"1", "123456"}) {+ os.Exit(3)+ }+ writeTestEnvelope(helperKindPCM, 0, 1, 200_000, make([]byte, 4))+ _, _ = os.Stdout.Write(make([]byte, helperHeaderSize))+ select {}+ case "--capture-window":+ expected := []string{+ "1", "123456", "987654", "--adapter-index", "2",+ "--mft-index", "3", "--protocol-v1",+ }+ if !reflect.DeepEqual(os.Args[2:], expected) {+ os.Exit(4)+ }+ status := []byte(`{"state":"starting","hardwareOnly":true,"adapterIndex":2,"adapterName":"GPU","adapterLuid":"0:1","mftIndex":3,"mftName":"MFT","mftClsid":"clsid"}`)+ writeTestEnvelope(helperKindStatus, 0, 0, 0, status)+ writeTestEnvelope(helperKindH264, 1, 100, 333_330, []byte{0, 0, 0, 1, 0x65})+ request := make([]byte, 1)+ if _, err := io.ReadFull(os.Stdin, request); err != nil || request[0] != 'K' {+ os.Exit(5)+ }+ _, _ = os.Stdout.Write(make([]byte, helperHeaderSize))+ select {}+ }+ }+ os.Exit(main.Run())+}++func writeTestEnvelope(kind, flags byte, timestamp, duration uint64, payload []byte) {+ header := make([]byte, helperHeaderSize)+ copy(header, helperProtocolMagic)+ header[4] = 1+ header[5] = kind+ header[6] = flags+ binary.BigEndian.PutUint64(header[8:16], timestamp)+ binary.BigEndian.PutUint64(header[16:24], duration)+ binary.BigEndian.PutUint32(header[24:28], uint32(len(payload)))+ _, _ = os.Stdout.Write(header)+ _, _ = os.Stdout.Write(payload)+}++func TestCaptureAudioPassesIdentityAndKillsHelperBeforeWaitOnInvalidHeader(t *testing.T) {+ provider := &windowsProvider{helper: os.Args[0]}+ if !provider.available() {+ t.Skip("requires Windows 11")+ }++ result := make(chan error, 1)+ chunks := 0+ go func() {+ result <- provider.CaptureAudio(context.Background(), Target{+ WindowHandle: testWindowHandle, PID: 1, CreationTime: testCreationTime,+ }, func(PCMChunk) error {+ chunks+++ return nil+ })+ }()++ select {+ case err := <-result:+ if err == nil || chunks != 1 {+ t.Fatalf("CaptureAudio error = %v, chunks = %d", err, chunks)+ }+ case <-time.After(2 * time.Second):+ t.Fatal("CaptureAudio waited on a running helper after an invalid header")+ }+}++func TestCaptureWindowPassesBoundIdentityAndForwardsKeyFrameRequest(t *testing.T) {+ provider := &windowsProvider{helper: os.Args[0], adapterIndex: 2, mftIndex: 3}+ if !provider.available() {+ t.Skip("requires Windows 11")+ }++ keyFrames := make(chan struct{}, 1)+ result := make(chan error, 1)+ statuses, accessUnits := 0, 0+ go func() {+ result <- provider.CaptureWindow(context.Background(), Target{+ WindowHandle: testWindowHandle, PID: 1, CreationTime: testCreationTime,+ }, keyFrames, func(PCMChunk) error {+ return nil+ }, func(unit H264AccessUnit) error {+ accessUnits+++ if !unit.KeyFrame {+ t.Error("first access unit was not a key frame")+ }+ keyFrames <- struct{}{}+ return nil+ }, func(Status) error {+ statuses+++ return nil+ })+ }()++ select {+ case err := <-result:+ if err == nil || statuses != 1 || accessUnits != 1 {+ t.Fatalf("CaptureWindow error = %v, statuses = %d, access units = %d", err, statuses, accessUnits)+ }+ case <-time.After(2 * time.Second):+ t.Fatal("CaptureWindow did not forward the key-frame request or kill the invalid helper")+ }+}++func TestListKillsHelperBeforeWaitingOnOversizeOutput(t *testing.T) {+ provider := &windowsProvider{helper: os.Args[0]}+ if !provider.available() {+ t.Skip("requires Windows 11")+ }++ result := make(chan error, 1)+ go func() {+ _, err := provider.List(context.Background())+ result <- err+ }()+ select {+ case err := <-result:+ if err == nil {+ t.Fatal("List accepted oversized helper output")+ }+ case <-time.After(2 * time.Second):+ t.Fatal("List waited on a running helper after oversized output")+ }+}diff --git a/native/window-capture-helper/README.md b/native/window-capture-helper/README.mdnew file mode 100644index 00000000..6b1c5ea4--- /dev/null+++ b/native/window-capture-helper/README.md@@ -0,0 +1,30 @@+# Windows Window-Capture Helper++This Windows 11 helper is the local input for Native sender window capture. It+enumerates visible top-level windows locally, binds the selected HWND to its PID+and process creation time, and has two explicit modes:++- process-tree PCM for the existing browser video sources; or+- WGC/D3D11 window video plus process-tree PCM, with adapter-bound,+ hardware-only Media Foundation H.264 output.++It has no whole-system audio, software encoder, alternate codec, or network+fallback. Build it outside the repository, then place it beside the sender (or+set `SCREENER_WINDOW_CAPTURE_HELPER` for a development run):++```powershell+$out = Join-Path ([IO.Path]::GetTempPath()) 'screener-native-capture'+./native/window-capture-helper/build.ps1 -OutputDirectory $out+```++The implementation follows Microsoft's MIT-licensed reference samples and+official API contracts without copying their WIL framework. The retained MF+fixture compiles the same encoder source with `SCREENER_H264_FIXTURE`; there is+not a second product MFT implementation.++- <https://github.com/microsoft/Windows-classic-samples/tree/main/Samples/ApplicationLoopback>+- <https://github.com/microsoft/Windows.UI.Composition-Win32-Samples/tree/master/cpp/ScreenCaptureforHWND>+- <https://learn.microsoft.com/en-us/windows/win32/api/audioclientactivationparams/ns-audioclientactivationparams-audioclient_activation_params>+- <https://learn.microsoft.com/en-us/windows/uwp/audio-video-camera/screen-capture>+- <https://learn.microsoft.com/en-us/windows/win32/api/windows.graphics.capture.interop/nf-windows-graphics-capture-interop-igraphicscaptureiteminterop-createforwindow>+- <https://learn.microsoft.com/en-us/uwp/api/windows.graphics.capture.direct3d11captureframepool.createfreethreaded>diff --git a/native/process-audio-helper/build.ps1 b/native/window-capture-helper/build.ps1similarity index 57%rename from native/process-audio-helper/build.ps1rename to native/window-capture-helper/build.ps1index 59e14cac..bb3f978c 100644--- a/native/process-audio-helper/build.ps1+++ b/native/window-capture-helper/build.ps1@@ -23,18 +23,22 @@ if ([string]::IsNullOrWhiteSpace($installationPath)) {}$developerCommand = Join-Path $installationPath 'Common7\Tools\VsDevCmd.bat'-$sourcePath = Join-Path $helperDirectory 'main.cpp'-$executablePath = Join-Path $outputPath 'screener-process-audio.exe'-$objectPath = Join-Path $outputPath 'main.obj'+$mainSourcePath = Join-Path $helperDirectory 'main.cpp'+$audioSourcePath = Join-Path $helperDirectory 'process_audio.cpp'+$executablePath = Join-Path $outputPath 'screener-window-capture.exe'+$mainObjectPath = Join-Path $outputPath 'window-capture.obj'+$audioObjectPath = Join-Path $outputPath 'process-audio.obj'New-Item -ItemType Directory -Path $outputPath -Force | Out-Null$compile = @('call "{0}" -arch=x64 -host_arch=x64 >nul' -f $developerCommand- 'cl.exe /nologo /std:c++20 /EHsc /W4 /WX /DUNICODE /D_UNICODE /DWIN32_LEAN_AND_MEAN /D_WIN32_WINNT=0x0A00 /DNTDDI_VERSION=0x0A00000A "{0}" /Fo:"{1}" /Fe:"{2}" /link ole32.lib mmdevapi.lib runtimeobject.lib user32.lib' -f $sourcePath, $objectPath, $executablePath+ 'cl.exe /nologo /c /std:c++20 /EHsc /W4 /WX /DUNICODE /D_UNICODE /DWIN32_LEAN_AND_MEAN /D_WIN32_WINNT=0x0A00 /DNTDDI_VERSION=0x0A00000A "{0}" /Fo:"{1}"' -f $mainSourcePath, $mainObjectPath+ 'cl.exe /nologo /c /std:c++20 /EHsc /W4 /WX /DUNICODE /D_UNICODE /DWIN32_LEAN_AND_MEAN /D_WIN32_WINNT=0x0A00 /DNTDDI_VERSION=0x0A00000A "{0}" /Fo:"{1}"' -f $audioSourcePath, $audioObjectPath+ 'link.exe /nologo /out:"{0}" "{1}" "{2}" ole32.lib mmdevapi.lib runtimeobject.lib user32.lib mfplat.lib mf.lib mfuuid.lib d3d11.lib dxgi.lib dxguid.lib evr.lib oleaut32.lib windowsapp.lib' -f $executablePath, $mainObjectPath, $audioObjectPath) -join ' && '& cmd.exe /d /s /c $compileif ($LASTEXITCODE -ne 0) {- throw "Process-audio helper compilation failed with exit code $LASTEXITCODE."+ throw "Window-capture helper compilation failed with exit code $LASTEXITCODE."}Write-Output $executablePathdiff --git a/native/fixtures/mf-h264-hardware/main.cpp b/native/window-capture-helper/main.cppsimilarity index 60%rename from native/fixtures/mf-h264-hardware/main.cpprename to native/window-capture-helper/main.cppindex 53e5821b..e10768ed 100644--- a/native/fixtures/mf-h264-hardware/main.cpp+++ b/native/window-capture-helper/main.cpp@@ -6,22 +6,38 @@#include <d3d11.h>#include <dxgi1_2.h>#include <evr.h>+#include <fcntl.h>+#include <io.h>#include <mfapi.h>#include <mferror.h>#include <mfidl.h>#include <mftransform.h>+#ifdef SCREENER_H264_FIXTURE#include <pdh.h>#include <pdhmsg.h>+#endif#include <propvarutil.h>+#include <windows.graphics.capture.interop.h>+#include <windows.graphics.directx.direct3d11.interop.h>#include <wrl/client.h>+#include <winrt/Windows.Foundation.h>+#include <winrt/Windows.Graphics.Capture.h>+#include <winrt/Windows.Graphics.DirectX.Direct3D11.h>+#include <winrt/Windows.Graphics.DirectX.h>+#include <winrt/base.h>++#include "process_audio.h"+#include <algorithm>+#include <array>#include <atomic>#include <chrono>#include <cmath>#include <cstdint>#include <cwctype>#include <exception>+#include <functional>#include <iomanip>#include <iostream>#include <limits>@@ -44,13 +60,16 @@ constexpr UINT32 kHeight = 720;constexpr UINT32 kFrameRate = 30;constexpr UINT32 kBitRate = 3'000'000;constexpr UINT32 kVbvBytes = kBitRate / kFrameRate / 8;-constexpr UINT32 kFrameCount = 360;constexpr UINT32 kGopFrames = 60;-constexpr UINT32 kMaxInFlight = 8;constexpr UINT32 kMaxEventsPerPump = 64;constexpr DWORD kMaxEncodedSampleBytes = 4 * 1024 * 1024;+#ifdef SCREENER_H264_FIXTURE+constexpr UINT32 kFrameCount = 360;+constexpr UINT32 kMaxInFlight = 8;constexpr DWORD kMaxPdhArrayBytes = 4 * 1024 * 1024;constexpr DWORD kMaxPdhArrayItems = 16 * 1024;+#endif+constexpr LONGLONG kFrameDuration100ns = 10'000'000 / kFrameRate;class GateFailure final : public std::runtime_error {@@ -552,6 +571,7 @@ void ValidateCodecReadback(ICodecAPI* codec) {}}+#ifdef SCREENER_H264_FIXTUREstd::vector<UINT8> SyntheticNv12(UINT32 frame_index) {std::vector<UINT8> pixels(kWidth * kHeight * 3 / 2);const UINT32 bar_start = (frame_index * 11) % kWidth;@@ -607,6 +627,7 @@ ComPtr<IMFSample> CreateInputSample(ID3D11Device* device, UINT32 frame_index) {"input-sample-duration");return sample;}+#endifstruct NalSummary final {bool annex_b = false;@@ -694,6 +715,7 @@ std::vector<UINT8> ReadSample(IMFSample* sample) {return bytes;}+#ifdef SCREENER_H264_FIXTUREULONGLONG ProcessCpu100ns() {FILETIME created = {};FILETIME exited = {};@@ -851,6 +873,7 @@ struct RunEvidence final {ULONGLONG cpu_100ns = 0;double wall_seconds = 0.0;};+#endifvoid ForceKeyFrame(ICodecAPI* codec) {VARIANT value;@@ -928,6 +951,7 @@ ComPtr<IMFSample> PullOutput(IMFTransform* transform,return produced;}+#ifdef SCREENER_H264_FIXTUREbool IsRecoveryFrame(UINT32 frame) {return frame % kGopFrames == 0;}@@ -1164,6 +1188,7 @@ RunEvidence RunEncoder(const Adapter& adapter, const DeviceContext& device,}return evidence;}+#endifUINT ParseIndex(const wchar_t* value, const std::string& stage) {try {@@ -1178,6 +1203,719 @@ UINT ParseIndex(const wchar_t* value, const std::string& stage) {}}+#ifndef SCREENER_H264_FIXTURE+constexpr DWORD kMaxProductAccessUnitBytes = 1 * 1024 * 1024;+constexpr DWORD kMaxStatusBytes = 4 * 1024;+constexpr UINT64 kEnvelopeFrameDuration100ns =+ (static_cast<UINT64>(kFrameDuration100ns) / 10) * 10;++enum class OutputKind : UINT8 {+ pcm = 1,+ h264 = 2,+ status = 3,+};++class UniqueHandle final {+ public:+ UniqueHandle() = default;+ explicit UniqueHandle(HANDLE value) : value_(value) {}+ UniqueHandle(const UniqueHandle&) = delete;+ UniqueHandle& operator=(const UniqueHandle&) = delete;+ UniqueHandle(UniqueHandle&& other) noexcept+ : value_(std::exchange(other.value_, nullptr)) {}+ ~UniqueHandle() {+ if (value_ != nullptr && value_ != INVALID_HANDLE_VALUE) CloseHandle(value_);+ }+ HANDLE get() const { return value_; }++ private:+ HANDLE value_ = nullptr;+};++void PutUint32BE(BYTE* output, UINT32 value) {+ for (int index = 3; index >= 0; --index) {+ output[index] = static_cast<BYTE>(value & 0xff);+ value >>= 8;+ }+}++void PutUint64BE(BYTE* output, UINT64 value) {+ for (int index = 7; index >= 0; --index) {+ output[index] = static_cast<BYTE>(value & 0xff);+ value >>= 8;+ }+}++class ProtocolWriter final {+ public:+ HRESULT Write(OutputKind kind, UINT8 flags, UINT64 timestamp100ns,+ UINT64 duration100ns, const BYTE* data, DWORD size) {+ DWORD maximum = kind == OutputKind::status ? kMaxStatusBytes+ : kMaxProductAccessUnitBytes;+ if (data == nullptr || size == 0 || size > maximum ||+ (kind != OutputKind::status && duration100ns == 0)) {+ return HRESULT_FROM_WIN32(ERROR_INVALID_DATA);+ }+ std::array<BYTE, 28> header{};+ header[0] = 'S';+ header[1] = 'M';+ header[2] = 'E';+ header[3] = 'D';+ header[4] = 1;+ header[5] = static_cast<BYTE>(kind);+ header[6] = flags;+ PutUint64BE(header.data() + 8, timestamp100ns);+ PutUint64BE(header.data() + 16, duration100ns);+ PutUint32BE(header.data() + 24, size);++ std::lock_guard<std::mutex> lock(mutex_);+ HRESULT result = WriteAll(header.data(), static_cast<DWORD>(header.size()));+ return SUCCEEDED(result) ? WriteAll(data, size) : result;+ }++ HRESULT WriteStatus(const std::string& value) {+ return Write(OutputKind::status, 0, 0, 0,+ reinterpret_cast<const BYTE*>(value.data()),+ static_cast<DWORD>(value.size()));+ }++ private:+ HRESULT WriteAll(const BYTE* data, DWORD size) {+ HANDLE output = GetStdHandle(STD_OUTPUT_HANDLE);+ while (size > 0) {+ DWORD written = 0;+ if (!WriteFile(output, data, size, &written, nullptr)) {+ return HRESULT_FROM_WIN32(GetLastError());+ }+ if (written == 0) return HRESULT_FROM_WIN32(ERROR_WRITE_FAULT);+ data += written;+ size -= written;+ }+ return S_OK;+ }++ std::mutex mutex_;+};++ComPtr<IMFSample> CreateSurfaceSample(ID3D11Texture2D* texture,+ LONGLONG timestamp100ns,+ LONGLONG duration100ns) {+ ComPtr<IMFMediaBuffer> buffer;+ Check(MFCreateDXGISurfaceBuffer(IID_ID3D11Texture2D, texture, 0, FALSE,+ &buffer),+ "input-dxgi-buffer");+ ComPtr<IMFSample> sample;+ Check(MFCreateVideoSampleFromSurface(nullptr, &sample),+ "input-video-sample");+ Check(sample->AddBuffer(buffer.Get()), "input-sample-buffer");+ Check(sample->SetSampleTime(timestamp100ns), "input-sample-time");+ Check(sample->SetSampleDuration(duration100ns), "input-sample-duration");+ return sample;+}++struct EncodedAccessUnit final {+ UINT64 timestamp100ns = 0;+ bool key_frame = false;+ std::vector<UINT8> bytes;+};++class LiveEncoder final {+ public:+ explicit LiveEncoder(SelectedTransform selected)+ : selected_(std::move(selected)) {+ ConfigureCodec(selected_.codec.Get());+ ComPtr<IMFMediaType> output_type = CreateOutputType();+ Check(selected_.transform->SetOutputType(0, output_type.Get(), 0),+ "mft-set-output-type");+ ComPtr<IMFMediaType> input_type = CreateInputType();+ Check(selected_.transform->SetInputType(0, input_type.Get(), 0),+ "mft-set-input-type");+ ValidateMediaTypes(selected_.transform.Get());+ ValidateCodecReadback(selected_.codec.Get());+ Check(selected_.transform->GetOutputStreamInfo(0, &output_info_),+ "mft-output-stream-info");+ Check(selected_.transform->ProcessMessage(+ MFT_MESSAGE_NOTIFY_BEGIN_STREAMING, 0),+ "mft-begin-streaming");+ Check(selected_.transform->ProcessMessage(+ MFT_MESSAGE_NOTIFY_START_OF_STREAM, 0),+ "mft-start-stream");+ started_ = true;+ }++ LiveEncoder(const LiveEncoder&) = delete;+ LiveEncoder& operator=(const LiveEncoder&) = delete;++ ~LiveEncoder() {+ if (started_) {+ selected_.transform->ProcessMessage(MFT_MESSAGE_COMMAND_FLUSH, 0);+ selected_.transform->ProcessMessage(MFT_MESSAGE_NOTIFY_END_STREAMING, 0);+ }+ }++ const SelectedTransform& selected() const { return selected_; }++ EncodedAccessUnit Encode(ID3D11Texture2D* texture, UINT64 timestamp100ns,+ bool force_key_frame) {+ WaitForInput();+ if (force_key_frame) ForceKeyFrame(selected_.codec.Get());+ ComPtr<IMFSample> sample = CreateSurfaceSample(+ texture, static_cast<LONGLONG>(timestamp100ns), kFrameDuration100ns);+ Check(selected_.transform->ProcessInput(0, sample.Get(), 0),+ "mft-process-input");+ --input_requests_;++ const auto deadline = std::chrono::steady_clock::now() ++ std::chrono::seconds(2);+ while (std::chrono::steady_clock::now() < deadline) {+ MediaEventType type = NextEvent(deadline);+ if (type == METransformNeedInput) {+ ++input_requests_;+ continue;+ }+ if (type != METransformHaveOutput) continue;++ ComPtr<IMFSample> output = PullOutput(selected_.transform.Get(),+ output_info_);+ LONGLONG output_time = -1;+ Check(output->GetSampleTime(&output_time), "output-sample-time");+ if (output_time != static_cast<LONGLONG>(timestamp100ns)) {+ Fail("output-order", "hardware MFT changed or reordered a live timestamp");+ }+ std::vector<UINT8> bytes = ReadSample(output.Get());+ NalSummary nal = InspectAnnexB(bytes);+ if (!nal.annex_b) {+ Fail("bitstream-annexb", "live output is not Annex-B H264");+ }+ if (nal.profile_level_id) {+ if (*nal.profile_level_id != "42c01f") {+ Fail("bitstream-profile", "hardware MFT did not emit profile-level-id 42c01f");+ }+ if (profile_level_id_ && *profile_level_id_ != *nal.profile_level_id) {+ Fail("bitstream-profile-change", "hardware MFT changed SPS profile");+ }+ profile_level_id_ = nal.profile_level_id;+ }+ if (force_key_frame && !(nal.sps && nal.pps && nal.idr)) {+ Fail("bitstream-recovery-unit",+ "requested live recovery unit lacks SPS, PPS, or IDR");+ }+ if (!profile_level_id_) {+ Fail("bitstream-sps", "first live access unit did not carry an SPS");+ }+ return EncodedAccessUnit{timestamp100ns, nal.idr, std::move(bytes)};+ }+ Fail("mft-output-timeout", "hardware MFT did not produce live output in time");+ }++ private:+ MediaEventType NextEvent(std::chrono::steady_clock::time_point deadline) {+ while (std::chrono::steady_clock::now() < deadline) {+ ComPtr<IMFMediaEvent> event;+ HRESULT result = selected_.events->GetEvent(MF_EVENT_FLAG_NO_WAIT,+ &event);+ if (result == MF_E_NO_EVENTS_AVAILABLE) {+ std::this_thread::sleep_for(std::chrono::milliseconds(1));+ continue;+ }+ Check(result, "mft-get-event");+ HRESULT status = S_OK;+ Check(event->GetStatus(&status), "mft-event-status-read");+ Check(status, "mft-event-status");+ MediaEventType type = MEUnknown;+ Check(event->GetType(&type), "mft-event-type");+ if (type == MEError) Fail("mft-error-event", "hardware MFT emitted MEError");+ return type;+ }+ Fail("mft-event-timeout", "hardware MFT did not request input in time");+ }++ void WaitForInput() {+ const auto deadline = std::chrono::steady_clock::now() ++ std::chrono::seconds(2);+ while (input_requests_ == 0) {+ MediaEventType type = NextEvent(deadline);+ if (type == METransformNeedInput) {+ ++input_requests_;+ } else if (type == METransformHaveOutput) {+ Fail("mft-unexpected-output", "hardware MFT produced output without live input");+ }+ }+ }++ SelectedTransform selected_;+ MFT_OUTPUT_STREAM_INFO output_info_ = {};+ UINT32 input_requests_ = 0;+ std::optional<std::string> profile_level_id_;+ bool started_ = false;+};++class FrameConverter final {+ public:+ explicit FrameConverter(ID3D11Device* device) : device_(device) {+ Check(device_->QueryInterface(IID_PPV_ARGS(&video_device_)),+ "video-processor-device");+ ComPtr<ID3D11DeviceContext> context;+ device_->GetImmediateContext(&context);+ Check(context.As(&video_context_), "video-processor-context");+ }++ ComPtr<ID3D11Texture2D> Convert(ID3D11Texture2D* source, UINT32 width,+ UINT32 height) {+ if (source == nullptr || width == 0 || height == 0 ||+ width > 16'384 || height > 16'384) {+ Fail("capture-size", "captured window dimensions are invalid");+ }+ if (!enumerator_ || width != input_width_ || height != input_height_) {+ Configure(width, height);+ }++ D3D11_TEXTURE2D_DESC output_description = {};+ output_description.Width = kWidth;+ output_description.Height = kHeight;+ output_description.MipLevels = 1;+ output_description.ArraySize = 1;+ output_description.Format = DXGI_FORMAT_NV12;+ output_description.SampleDesc.Count = 1;+ output_description.Usage = D3D11_USAGE_DEFAULT;+ output_description.BindFlags = D3D11_BIND_RENDER_TARGET;+ ComPtr<ID3D11Texture2D> output;+ Check(device_->CreateTexture2D(&output_description, nullptr, &output),+ "video-processor-output-texture");++ D3D11_VIDEO_PROCESSOR_INPUT_VIEW_DESC input_description = {};+ input_description.ViewDimension = D3D11_VPIV_DIMENSION_TEXTURE2D;+ input_description.Texture2D.ArraySlice = 0;+ input_description.Texture2D.MipSlice = 0;+ ComPtr<ID3D11VideoProcessorInputView> input_view;+ Check(video_device_->CreateVideoProcessorInputView(+ source, enumerator_.Get(), &input_description, &input_view),+ "video-processor-input-view");++ D3D11_VIDEO_PROCESSOR_OUTPUT_VIEW_DESC output_view_description = {};+ output_view_description.ViewDimension = D3D11_VPOV_DIMENSION_TEXTURE2D;+ output_view_description.Texture2D.MipSlice = 0;+ ComPtr<ID3D11VideoProcessorOutputView> output_view;+ Check(video_device_->CreateVideoProcessorOutputView(+ output.Get(), enumerator_.Get(), &output_view_description,+ &output_view),+ "video-processor-output-view");++ RECT source_rect = {0, 0, static_cast<LONG>(width),+ static_cast<LONG>(height)};+ double scale = std::min(static_cast<double>(kWidth) / width,+ static_cast<double>(kHeight) / height);+ LONG target_width = std::max<LONG>(+ 2, static_cast<LONG>(std::llround(width * scale)) & ~1L);+ LONG target_height = std::max<LONG>(+ 2, static_cast<LONG>(std::llround(height * scale)) & ~1L);+ target_width = std::min<LONG>(target_width, kWidth);+ target_height = std::min<LONG>(target_height, kHeight);+ LONG left = (static_cast<LONG>(kWidth) - target_width) / 2;+ LONG top = (static_cast<LONG>(kHeight) - target_height) / 2;+ RECT target_rect = {left, top, left + target_width, top + target_height};+ RECT output_rect = {0, 0, static_cast<LONG>(kWidth),+ static_cast<LONG>(kHeight)};++ D3D11_VIDEO_COLOR background = {};+ background.RGBA.A = 1.0f;+ video_context_->VideoProcessorSetOutputBackgroundColor(processor_.Get(),+ FALSE, &background);+ video_context_->VideoProcessorSetOutputTargetRect(processor_.Get(), TRUE,+ &output_rect);+ video_context_->VideoProcessorSetStreamFrameFormat(+ processor_.Get(), 0, D3D11_VIDEO_FRAME_FORMAT_PROGRESSIVE);+ video_context_->VideoProcessorSetStreamSourceRect(processor_.Get(), 0,+ TRUE, &source_rect);+ video_context_->VideoProcessorSetStreamDestRect(processor_.Get(), 0, TRUE,+ &target_rect);+ video_context_->VideoProcessorSetStreamAutoProcessingMode(processor_.Get(),+ 0, FALSE);+ D3D11_VIDEO_PROCESSOR_STREAM stream = {};+ stream.Enable = TRUE;+ stream.pInputSurface = input_view.Get();+ Check(video_context_->VideoProcessorBlt(processor_.Get(), output_view.Get(),+ 0, 1, &stream),+ "video-processor-blit");+ return output;+ }++ private:+ void Configure(UINT32 width, UINT32 height) {+ D3D11_VIDEO_PROCESSOR_CONTENT_DESC description = {};+ description.InputFrameFormat = D3D11_VIDEO_FRAME_FORMAT_PROGRESSIVE;+ description.InputFrameRate = {kFrameRate, 1};+ description.InputWidth = width;+ description.InputHeight = height;+ description.OutputFrameRate = {kFrameRate, 1};+ description.OutputWidth = kWidth;+ description.OutputHeight = kHeight;+ description.Usage = D3D11_VIDEO_USAGE_PLAYBACK_NORMAL;+ ComPtr<ID3D11VideoProcessorEnumerator> enumerator;+ Check(video_device_->CreateVideoProcessorEnumerator(&description,+ &enumerator),+ "video-processor-enumerator");+ ComPtr<ID3D11VideoProcessor> processor;+ Check(video_device_->CreateVideoProcessor(enumerator.Get(), 0, &processor),+ "video-processor-create");+ enumerator_ = std::move(enumerator);+ processor_ = std::move(processor);+ input_width_ = width;+ input_height_ = height;+ }++ ComPtr<ID3D11Device> device_;+ ComPtr<ID3D11VideoDevice> video_device_;+ ComPtr<ID3D11VideoContext> video_context_;+ ComPtr<ID3D11VideoProcessorEnumerator> enumerator_;+ ComPtr<ID3D11VideoProcessor> processor_;+ UINT32 input_width_ = 0;+ UINT32 input_height_ = 0;+};++winrt::Windows::Graphics::DirectX::Direct3D11::IDirect3DDevice+CreateCaptureDevice(ID3D11Device* device) {+ ComPtr<IDXGIDevice> dxgi_device;+ Check(device->QueryInterface(IID_PPV_ARGS(&dxgi_device)),+ "capture-dxgi-device");+ winrt::com_ptr<IInspectable> inspectable;+ Check(CreateDirect3D11DeviceFromDXGIDevice(dxgi_device.Get(),+ inspectable.put()),+ "capture-winrt-device");+ return inspectable.as<+ winrt::Windows::Graphics::DirectX::Direct3D11::IDirect3DDevice>();+}++winrt::Windows::Graphics::Capture::GraphicsCaptureItem CreateCaptureItem(+ HWND window) {+ using winrt::Windows::Graphics::Capture::GraphicsCaptureItem;+ auto interop = winrt::get_activation_factory<GraphicsCaptureItem,+ IGraphicsCaptureItemInterop>();+ GraphicsCaptureItem item{nullptr};+ Check(interop->CreateForWindow(window, winrt::guid_of<GraphicsCaptureItem>(),+ winrt::put_abi(item)),+ "capture-item-window");+ return item;+}++ComPtr<ID3D11Texture2D> CaptureTexture(+ const winrt::Windows::Graphics::Capture::Direct3D11CaptureFrame& frame) {+ auto access = frame.Surface().as<+ ::Windows::Graphics::DirectX::Direct3D11::IDirect3DDxgiInterfaceAccess>();+ ComPtr<ID3D11Texture2D> texture;+ Check(access->GetInterface(IID_PPV_ARGS(&texture)),+ "capture-frame-texture");+ return texture;+}++std::string JSONString(const std::string& value) {+ std::string output = "\"";+ for (unsigned char character : value) {+ if (character == '\\' || character == '"') {+ output.push_back('\\');+ output.push_back(static_cast<char>(character));+ } else if (character >= 0x20 && character <= 0x7e) {+ output.push_back(static_cast<char>(character));+ } else {+ output.push_back('?');+ }+ }+ output.push_back('"');+ return output;+}++bool ConsumeKeyFrameRequest() {+ HANDLE input = GetStdHandle(STD_INPUT_HANDLE);+ DWORD available = 0;+ if (input == INVALID_HANDLE_VALUE || input == nullptr ||+ !PeekNamedPipe(input, nullptr, 0, nullptr, &available, nullptr) ||+ available == 0) {+ return false;+ }+ std::array<BYTE, 64> bytes{};+ DWORD read = 0;+ if (!ReadFile(input, bytes.data(),+ std::min<DWORD>(available, static_cast<DWORD>(bytes.size())),+ &read, nullptr)) {+ return false;+ }+ return std::find(bytes.begin(), bytes.begin() + read,+ static_cast<BYTE>('K')) != bytes.begin() + read;+}++UINT64 ParseUint64(const wchar_t* value, const std::string& stage) {+ try {+ size_t consumed = 0;+ unsigned long long parsed = std::stoull(value, &consumed, 10);+ if (value[consumed] != L'\0' || parsed == 0) {+ Fail(stage, "value is not a positive bounded integer");+ }+ return static_cast<UINT64>(parsed);+ } catch (const std::exception&) {+ Fail(stage, "value is not a positive bounded integer");+ }+}++struct ProductArguments final {+ enum class Mode { list, audio, window } mode = Mode::list;+ DWORD pid = 0;+ UINT64 creation_time = 0;+ UINT64 window_handle = 0;+ UINT adapter_index = 0;+ UINT mft_index = 0;+};++ProductArguments ParseProductArguments(int count, wchar_t** values) {+ ProductArguments arguments;+ if (count == 2 && std::wstring(values[1]) == L"--list") return arguments;+ if (count == 4 && std::wstring(values[1]) == L"--capture-audio") {+ arguments.mode = ProductArguments::Mode::audio;+ } else if (count == 10 && std::wstring(values[1]) == L"--capture-window" &&+ std::wstring(values[5]) == L"--adapter-index" &&+ std::wstring(values[7]) == L"--mft-index" &&+ std::wstring(values[9]) == L"--protocol-v1") {+ arguments.mode = ProductArguments::Mode::window;+ arguments.window_handle = ParseUint64(values[4], "argument-window");+ arguments.adapter_index = ParseIndex(values[6], "argument-adapter");+ arguments.mft_index = ParseIndex(values[8], "argument-mft");+ } else {+ Fail("arguments", "unsupported or incomplete command-line argument");+ }+ UINT64 pid = ParseUint64(values[2], "argument-pid");+ if (pid > std::numeric_limits<DWORD>::max()) {+ Fail("argument-pid", "PID is outside the Windows process range");+ }+ arguments.pid = static_cast<DWORD>(pid);+ arguments.creation_time = ParseUint64(values[3], "argument-creation-time");+ return arguments;+}++HRESULT RunAudioCapture(const ProductArguments& arguments) {+ UniqueHandle stop(CreateEventW(nullptr, TRUE, FALSE, nullptr));+ if (stop.get() == nullptr) return HRESULT_FROM_WIN32(GetLastError());+ ProtocolWriter writer;+ return screener::capture::CaptureProcessAudio(+ arguments.pid, arguments.creation_time, stop.get(),+ [&writer](UINT64 timestamp100ns, const BYTE* data, DWORD size) {+ return writer.Write(OutputKind::pcm, 0, timestamp100ns,+ screener::capture::kAudioChunkDuration100ns, data,+ size);+ });+}++void RunWindowCapture(const ProductArguments& arguments) {+ HRESULT identity = screener::capture::ValidateWindowTarget(+ arguments.window_handle, arguments.pid, arguments.creation_time);+ Check(identity, "target-identity");++ std::vector<Adapter> adapters = EnumerateAdapters();+ const Adapter& adapter = SelectAdapter(adapters, arguments.adapter_index);+ DeviceContext device = CreateDevice(adapter);+ ActivationList activations = EnumerateHardwareEncoders(adapter);+ SelectedTransform selected = ActivateTransform(+ activations, arguments.mft_index, device.manager.Get());+ LiveEncoder encoder(std::move(selected));+ FrameConverter converter(device.device.Get());+ ProtocolWriter writer;++ using namespace winrt::Windows::Graphics::Capture;+ using winrt::Windows::Graphics::DirectX::DirectXPixelFormat;+ if (!GraphicsCaptureSession::IsSupported()) {+ Fail("capture-support", "Windows Graphics Capture is unavailable");+ }+ UniqueHandle process(OpenProcess(SYNCHRONIZE, FALSE, arguments.pid));+ if (process.get() == nullptr) {+ Check(HRESULT_FROM_WIN32(GetLastError()), "target-process-handle");+ }+ Check(screener::capture::ValidateWindowTarget(+ arguments.window_handle, arguments.pid, arguments.creation_time),+ "target-identity-before-capture");+ HWND window = reinterpret_cast<HWND>(+ static_cast<UINT_PTR>(arguments.window_handle));+ auto capture_device = CreateCaptureDevice(device.device.Get());+ GraphicsCaptureItem item = CreateCaptureItem(window);+ Check(screener::capture::ValidateWindowTarget(+ arguments.window_handle, arguments.pid, arguments.creation_time),+ "target-identity-after-item");+ auto initial_size = item.Size();+ if (initial_size.Width <= 0 || initial_size.Height <= 0) {+ Fail("capture-size", "selected window has no capturable content");+ }+ Direct3D11CaptureFramePool pool = Direct3D11CaptureFramePool::CreateFreeThreaded(+ capture_device, DirectXPixelFormat::B8G8R8A8UIntNormalized, 2,+ initial_size);+ GraphicsCaptureSession capture_session = pool.CreateCaptureSession(item);++ UniqueHandle shutdown(CreateEventW(nullptr, TRUE, FALSE, nullptr));+ UniqueHandle frame_ready(CreateEventW(nullptr, FALSE, FALSE, nullptr));+ if (shutdown.get() == nullptr || frame_ready.get() == nullptr) {+ Check(HRESULT_FROM_WIN32(GetLastError()), "capture-events");+ }++ std::atomic<LONG> audio_result{S_OK};+ std::atomic<bool> audio_finished{false};+ std::atomic<bool> item_closed{false};+ auto frame_token = pool.FrameArrived(+ [event = frame_ready.get()](const Direct3D11CaptureFramePool&,+ const winrt::Windows::Foundation::IInspectable&) {+ SetEvent(event);+ });+ auto closed_token = item.Closed(+ [event = shutdown.get(), &item_closed](const GraphicsCaptureItem&,+ const winrt::Windows::Foundation::IInspectable&) {+ item_closed.store(true);+ SetEvent(event);+ });+ std::thread audio([&]() {+ HRESULT result = screener::capture::CaptureProcessAudio(+ arguments.pid, arguments.creation_time, shutdown.get(),+ [&writer](UINT64 timestamp100ns, const BYTE* data, DWORD size) {+ return writer.Write(OutputKind::pcm, 0, timestamp100ns,+ screener::capture::kAudioChunkDuration100ns,+ data, size);+ });+ audio_result.store(result);+ audio_finished.store(true);+ SetEvent(shutdown.get());+ });++ auto cleanup = [&]() noexcept {+ SetEvent(shutdown.get());+ try {+ pool.FrameArrived(frame_token);+ } catch (...) {+ }+ try {+ item.Closed(closed_token);+ } catch (...) {+ }+ try {+ capture_session.Close();+ } catch (...) {+ }+ try {+ pool.Close();+ } catch (...) {+ }+ if (audio.joinable()) audio.join();+ };++ try {+ std::ostringstream starting;+ starting << "{\"state\":\"starting\",\"hardwareOnly\":true,"+ << "\"adapterIndex\":" << adapter.index+ << ",\"adapterName\":"+ << JSONString(NarrowAscii(adapter.description.Description))+ << ",\"adapterLuid\":"+ << JSONString(LuidString(adapter.description.AdapterLuid))+ << ",\"mftIndex\":" << arguments.mft_index+ << ",\"mftName\":"+ << JSONString(encoder.selected().name)+ << ",\"mftClsid\":"+ << JSONString(encoder.selected().clsid) << '}';+ Check(writer.WriteStatus(starting.str()), "capture-status-starting");+ capture_session.StartCapture();++ UINT64 previous_timestamp = 0;+ UINT64 encoded_frames = 0;+ bool active_status_written = false;+ auto pool_size = initial_size;+ const HANDLE waits[] = {process.get(), shutdown.get(), frame_ready.get()};+ for (;;) {+ DWORD wait = WaitForMultipleObjects(3, waits, FALSE, 5'000);+ if (wait == WAIT_OBJECT_0) {+ Fail("target-exited", "selected target process exited");+ }+ if (wait == WAIT_OBJECT_0 + 1) {+ if (item_closed.load()) {+ Fail("capture-closed", "selected window stopped capture");+ }+ HRESULT result = static_cast<HRESULT>(audio_result.load());+ if (audio_finished.load() && result != HRESULT_FROM_WIN32(ERROR_CANCELLED)) {+ throw GateFailure("process-audio", "target process audio stopped",+ result);+ }+ Fail("capture-stopped", "window capture stopped");+ }+ if (wait == WAIT_TIMEOUT) {+ Fail("capture-frame-timeout", "selected window produced no frame in five seconds");+ }+ if (wait != WAIT_OBJECT_0 + 2) {+ Check(HRESULT_FROM_WIN32(GetLastError()), "capture-wait");+ }++ Direct3D11CaptureFrame latest{nullptr};+ for (;;) {+ Direct3D11CaptureFrame next = pool.TryGetNextFrame();+ if (!next) break;+ latest = std::move(next);+ }+ if (!latest) continue;+ INT64 signed_timestamp = latest.SystemRelativeTime().count();+ if (signed_timestamp <= 0) {+ Fail("capture-timestamp", "captured frame has no QPC timestamp");+ }+ UINT64 timestamp = static_cast<UINT64>(signed_timestamp);+ if (previous_timestamp != 0 && timestamp <= previous_timestamp) continue;+ if (previous_timestamp != 0 &&+ timestamp - previous_timestamp <+ static_cast<UINT64>(kFrameDuration100ns) * 19 / 20) {+ continue;+ }+ auto content_size = latest.ContentSize();+ if (content_size.Width <= 0 || content_size.Height <= 0) continue;+ if (content_size.Width != pool_size.Width ||+ content_size.Height != pool_size.Height) {+ latest.Close();+ latest = nullptr;+ pool.Recreate(capture_device,+ DirectXPixelFormat::B8G8R8A8UIntNormalized, 2,+ content_size);+ pool_size = content_size;+ continue;+ }+ ComPtr<ID3D11Texture2D> source = CaptureTexture(latest);+ D3D11_TEXTURE2D_DESC source_description = {};+ source->GetDesc(&source_description);+ UINT32 content_width = std::min<UINT32>(+ source_description.Width, static_cast<UINT32>(content_size.Width));+ UINT32 content_height = std::min<UINT32>(+ source_description.Height, static_cast<UINT32>(content_size.Height));+ ComPtr<ID3D11Texture2D> nv12 = converter.Convert(+ source.Get(), content_width, content_height);+ bool requested_key_frame = ConsumeKeyFrameRequest();+ bool key_frame = encoded_frames == 0 || encoded_frames % kGopFrames == 0 ||+ requested_key_frame;+ EncodedAccessUnit access_unit = encoder.Encode(nv12.Get(), timestamp,+ key_frame);+ if (access_unit.bytes.size() > kMaxProductAccessUnitBytes) {+ Fail("output-buffer-bounds", "live H264 access unit exceeded one MiB");+ }+ UINT64 wire_timestamp = (access_unit.timestamp100ns / 10) * 10;+ Check(writer.Write(OutputKind::h264, access_unit.key_frame ? 1 : 0,+ wire_timestamp, kEnvelopeFrameDuration100ns,+ access_unit.bytes.data(),+ static_cast<DWORD>(access_unit.bytes.size())),+ "capture-video-output");+ previous_timestamp = timestamp;+ ++encoded_frames;+ if (!active_status_written) {+ std::string active =+ "{\"state\":\"active\",\"hardwareOnly\":true,"+ "\"profileLevelId\":\"42c01f\",\"width\":1280,"+ "\"height\":720,\"fps\":30}";+ Check(writer.WriteStatus(active), "capture-status-active");+ active_status_written = true;+ }+ }+ } catch (...) {+ cleanup();+ throw;+ }+}+#endif++#ifdef SCREENER_H264_FIXTUREstruct Arguments final {bool list = false;std::optional<UINT> adapter_index;@@ -1236,9 +1974,11 @@ void PrintEvidence(const Adapter& adapter, UINT mft_index,<< "videoencode_mean_percent=" << mean_video_encode << '\n'<< "videoencode_max_percent=" << max_video_encode << '\n';}+#endif} // namespace+#ifdef SCREENER_H264_FIXTUREint wmain(int argc, wchar_t** argv) {try {Arguments arguments = ParseArguments(argc, argv);@@ -1288,3 +2028,39 @@ int wmain(int argc, wchar_t** argv) {return 3;}}+#endif++#ifndef SCREENER_H264_FIXTURE+int wmain(int argc, wchar_t** argv) {+ try {+ ProductArguments arguments = ParseProductArguments(argc, argv);+ if (arguments.mode == ProductArguments::Mode::list) {+ return screener::capture::WriteWindowList();+ }+ if (_setmode(_fileno(stdout), _O_BINARY) == -1) {+ Fail("stdout-binary", "could not switch stdout to binary mode");+ }+ if (arguments.mode == ProductArguments::Mode::audio) {+ Check(RunAudioCapture(arguments), "process-audio-capture");+ return 0;+ }+ Runtime runtime;+ RunWindowCapture(arguments);+ return 0;+ } catch (const GateFailure& error) {+ std::cerr << "result=window-capture-failed\n"+ << "stage=" << error.stage() << '\n'+ << "detail=" << error.what() << '\n';+ if (FAILED(error.result())) {+ std::cerr << "hresult=0x" << std::hex << std::setfill('0')+ << std::setw(8) << static_cast<UINT32>(error.result()) << '\n';+ }+ return 2;+ } catch (const std::exception& error) {+ std::cerr << "result=window-capture-failed\n"+ << "stage=unclassified\n"+ << "detail=" << error.what() << '\n';+ return 3;+ }+}+#endifdiff --git a/native/process-audio-helper/main.cpp b/native/window-capture-helper/process_audio.cppsimilarity index 73%rename from native/process-audio-helper/main.cpprename to native/window-capture-helper/process_audio.cppindex 95c5b6e7..85af9c90 100644--- a/native/process-audio-helper/main.cpp+++ b/native/window-capture-helper/process_audio.cpp@@ -9,10 +9,11 @@#include <wrl.h>#include <wrl/implements.h>-#include <array>+#include "process_audio.h"+#include <cstdint>-#include <cstdlib>#include <iostream>+#include <limits>#include <string>#include <vector>@@ -23,13 +24,8 @@ using Microsoft::WRL::RuntimeClass;using Microsoft::WRL::RuntimeClassFlags;using Microsoft::WRL::ClassicCom;+namespace screener::capture {namespace {-constexpr UINT32 kSampleRate = 48'000;-constexpr UINT16 kChannels = 2;-constexpr UINT16 kBytesPerSample = 2;-constexpr UINT32 kFramesPerChunk = 960;-constexpr DWORD kBytesPerChunk = kFramesPerChunk * kChannels * kBytesPerSample;-constexpr UINT64 kChunkDuration100ns = 200'000;constexpr size_t kMaxWindows = 100;class ActivationHandler final : public RuntimeClass<@@ -57,6 +53,7 @@ class ActivationHandler final : public RuntimeClass<};struct WindowTarget {+ UINT64 windowHandle;DWORD pid;UINT64 creationTime;std::string title;@@ -116,7 +113,7 @@ std::string JsonString(const std::string& value) {BOOL CALLBACK CollectWindow(HWND window, LPARAM parameter) {auto* targets = reinterpret_cast<std::vector<WindowTarget>*>(parameter);- if (targets->size() >= kMaxWindows) return FALSE;+ if (targets->size() >= kMaxWindows) return TRUE;if (!IsWindowVisible(window) || GetWindow(window, GW_OWNER) != nullptr) return TRUE;const int length = GetWindowTextLengthW(window);if (length <= 0 || length > 512) return TRUE;@@ -134,17 +131,23 @@ BOOL CALLBACK CollectWindow(HWND window, LPARAM parameter) {UINT64 creationTime = 0;const HRESULT result = ReadProcessCreationTime(process, &creationTime);CloseHandle(process);- if (SUCCEEDED(result)) targets->push_back({pid, creationTime, std::move(utf8)});+ if (SUCCEEDED(result)) {+ targets->push_back({static_cast<UINT64>(reinterpret_cast<UINT_PTR>(window)),+ pid, creationTime, std::move(utf8)});+ }return TRUE;}-int ListWindows() {+} // namespace++int WriteWindowList() {std::vector<WindowTarget> targets;if (!EnumWindows(CollectWindow, reinterpret_cast<LPARAM>(&targets))) return 2;std::cout << '[';for (size_t index = 0; index < targets.size(); ++index) {if (index != 0) std::cout << ',';- std::cout << "{\"pid\":" << targets[index].pid << ",\"creationTime\":"+ std::cout << "{\"windowHandle\":" << targets[index].windowHandle+ << ",\"pid\":" << targets[index].pid << ",\"creationTime\":"<< targets[index].creationTime << ",\"title\":"<< JsonString(targets[index].title) << '}';}@@ -152,43 +155,6 @@ int ListWindows() {return std::cout.good() ? 0 : 2;}-void PutUint32BE(BYTE* output, UINT32 value) {- for (int index = 3; index >= 0; --index) {- output[index] = static_cast<BYTE>(value & 0xff);- value >>= 8;- }-}--void PutUint64BE(BYTE* output, UINT64 value) {- for (int index = 7; index >= 0; --index) {- output[index] = static_cast<BYTE>(value & 0xff);- value >>= 8;- }-}--HRESULT WriteAll(HANDLE output, const BYTE* data, DWORD size) {- while (size > 0) {- DWORD written = 0;- if (!WriteFile(output, data, size, &written, nullptr) || written == 0) {- return HRESULT_FROM_WIN32(GetLastError());- }- data += written;- size -= written;- }- return S_OK;-}--HRESULT WritePCM(HANDLE output, UINT64 timestamp100ns, const BYTE* data) {- std::array<BYTE, 28> header{};- header[0] = 'S'; header[1] = 'P'; header[2] = 'C'; header[3] = 'M';- header[4] = 1;- PutUint64BE(header.data() + 8, timestamp100ns);- PutUint64BE(header.data() + 16, kChunkDuration100ns);- PutUint32BE(header.data() + 24, kBytesPerChunk);- HRESULT result = WriteAll(output, header.data(), static_cast<DWORD>(header.size()));- return SUCCEEDED(result) ? WriteAll(output, data, kBytesPerChunk) : result;-}-HRESULT ActivateProcessLoopback(DWORD pid, HANDLE completed, ComPtr<IAudioClient>* client) {AUDIOCLIENT_ACTIVATION_PARAMS parameters{};parameters.ActivationType = AUDIOCLIENT_ACTIVATION_TYPE_PROCESS_LOOPBACK;@@ -213,9 +179,42 @@ HRESULT ActivateProcessLoopback(DWORD pid, HANDLE completed, ComPtr<IAudioClientreturn result;}-HRESULT Capture(DWORD pid, UINT64 expectedCreationTime) {- HANDLE process = OpenProcess(SYNCHRONIZE | PROCESS_QUERY_LIMITED_INFORMATION, FALSE, pid);+HRESULT ValidateWindowTarget(UINT64 window_handle, DWORD pid,+ UINT64 expected_creation_time) {+ if (window_handle == 0 || pid == 0 || expected_creation_time == 0 ||+ window_handle > static_cast<UINT64>(std::numeric_limits<UINT_PTR>::max())) {+ return HRESULT_FROM_WIN32(ERROR_INVALID_PARAMETER);+ }+ HWND window = reinterpret_cast<HWND>(static_cast<UINT_PTR>(window_handle));+ DWORD window_pid = 0;+ if (!IsWindow(window) || GetWindowThreadProcessId(window, &window_pid) == 0 ||+ window_pid != pid) {+ return HRESULT_FROM_WIN32(ERROR_INVALID_WINDOW_HANDLE);+ }+ HANDLE process = OpenProcess(PROCESS_QUERY_LIMITED_INFORMATION, FALSE, pid);if (process == nullptr) return HRESULT_FROM_WIN32(GetLastError());+ UINT64 actual_creation_time = 0;+ HRESULT result = ReadProcessCreationTime(process, &actual_creation_time);+ CloseHandle(process);+ if (SUCCEEDED(result) && actual_creation_time != expected_creation_time) {+ result = HRESULT_FROM_WIN32(ERROR_INVALID_DATA);+ }+ return result;+}++HRESULT CaptureProcessAudio(DWORD pid, UINT64 expectedCreationTime,+ HANDLE stop_event, const PCMWriter& writer) {+ if (stop_event == nullptr || !writer) {+ return HRESULT_FROM_WIN32(ERROR_INVALID_PARAMETER);+ }+ HRESULT com_result = CoInitializeEx(nullptr, COINIT_MULTITHREADED);+ if (FAILED(com_result)) return com_result;+ HANDLE process = OpenProcess(SYNCHRONIZE | PROCESS_QUERY_LIMITED_INFORMATION, FALSE, pid);+ if (process == nullptr) {+ HRESULT result = HRESULT_FROM_WIN32(GetLastError());+ CoUninitialize();+ return result;+ }HANDLE completed = CreateEventW(nullptr, FALSE, FALSE, nullptr);HANDLE sampleReady = CreateEventW(nullptr, FALSE, FALSE, nullptr);if (completed == nullptr || sampleReady == nullptr) {@@ -223,6 +222,7 @@ HRESULT Capture(DWORD pid, UINT64 expectedCreationTime) {if (completed != nullptr) CloseHandle(completed);if (sampleReady != nullptr) CloseHandle(sampleReady);CloseHandle(process);+ CoUninitialize();return result;}UINT64 creationTime = 0;@@ -235,11 +235,11 @@ HRESULT Capture(DWORD pid, UINT64 expectedCreationTime) {ComPtr<IAudioCaptureClient> capture;WAVEFORMATEX format{};format.wFormatTag = WAVE_FORMAT_PCM;- format.nChannels = kChannels;- format.nSamplesPerSec = kSampleRate;- format.wBitsPerSample = kBytesPerSample * 8;- format.nBlockAlign = kChannels * kBytesPerSample;- format.nAvgBytesPerSec = kSampleRate * format.nBlockAlign;+ format.nChannels = kAudioChannels;+ format.nSamplesPerSec = kAudioSampleRate;+ format.wBitsPerSample = kAudioBytesPerSample * 8;+ format.nBlockAlign = kAudioChannels * kAudioBytesPerSample;+ format.nAvgBytesPerSec = kAudioSampleRate * format.nBlockAlign;if (SUCCEEDED(result)) {result = client->Initialize(AUDCLNT_SHAREMODE_SHARED,AUDCLNT_STREAMFLAGS_LOOPBACK | AUDCLNT_STREAMFLAGS_EVENTCALLBACK |@@ -253,15 +253,18 @@ HRESULT Capture(DWORD pid, UINT64 expectedCreationTime) {std::vector<BYTE> pending;size_t consumed = 0;UINT64 nextTimestamp = 0;- HANDLE output = GetStdHandle(STD_OUTPUT_HANDLE);- const HANDLE waits[] = {process, sampleReady};+ const HANDLE waits[] = {process, sampleReady, stop_event};while (SUCCEEDED(result)) {- const DWORD wait = WaitForMultipleObjects(2, waits, FALSE, 1'000);+ const DWORD wait = WaitForMultipleObjects(3, waits, FALSE, 1'000);if (wait == WAIT_OBJECT_0) {result = HRESULT_FROM_WIN32(ERROR_PROCESS_ABORTED);break;}if (wait == WAIT_TIMEOUT) continue;+ if (wait == WAIT_OBJECT_0 + 2) {+ result = HRESULT_FROM_WIN32(ERROR_CANCELLED);+ break;+ }if (wait != WAIT_OBJECT_0 + 1) {result = HRESULT_FROM_WIN32(GetLastError());break;@@ -294,14 +297,15 @@ HRESULT Capture(DWORD pid, UINT64 expectedCreationTime) {pending.insert(pending.end(), data, data + bytes);}capture->ReleaseBuffer(frames);- while (pending.size() - consumed >= kBytesPerChunk) {- result = WritePCM(output, nextTimestamp, pending.data() + consumed);+ while (pending.size() - consumed >= kAudioBytesPerChunk) {+ result = writer(nextTimestamp, pending.data() + consumed,+ kAudioBytesPerChunk);if (FAILED(result)) break;- consumed += kBytesPerChunk;- nextTimestamp += kChunkDuration100ns;+ consumed += kAudioBytesPerChunk;+ nextTimestamp += kAudioChunkDuration100ns;}if (FAILED(result)) break;- if (consumed > kBytesPerChunk * 4) {+ if (consumed > kAudioBytesPerChunk * 4) {pending.erase(pending.begin(), pending.begin() + static_cast<ptrdiff_t>(consumed));consumed = 0;}@@ -311,24 +315,8 @@ HRESULT Capture(DWORD pid, UINT64 expectedCreationTime) {CloseHandle(sampleReady);CloseHandle(completed);CloseHandle(process);+ CoUninitialize();return result;}-} // namespace-int wmain(int argc, wchar_t** argv) {- if (argc == 2 && std::wstring(argv[1]) == L"--list") return ListWindows();- if (argc != 4 || std::wstring(argv[1]) != L"--capture") return 2;- wchar_t* pidEnd = nullptr;- const unsigned long parsedPID = wcstoul(argv[2], &pidEnd, 10);- wchar_t* creationEnd = nullptr;- const unsigned long long parsedCreationTime = wcstoull(argv[3], &creationEnd, 10);- if (parsedPID == 0 || pidEnd == argv[2] || *pidEnd != L'\0' ||- parsedCreationTime == 0 || creationEnd == argv[3] || *creationEnd != L'\0') return 2;- if (_setmode(_fileno(stdout), _O_BINARY) == -1) return 2;- HRESULT result = CoInitializeEx(nullptr, COINIT_MULTITHREADED);- if (FAILED(result)) return 2;- result = Capture(static_cast<DWORD>(parsedPID), static_cast<UINT64>(parsedCreationTime));- CoUninitialize();- if (FAILED(result)) std::cerr << "process_audio_capture_failed hresult=0x" << std::hex << result << '\n';- return SUCCEEDED(result) ? 0 : 2;-}+} // namespace screener::capturediff --git a/native/window-capture-helper/process_audio.h b/native/window-capture-helper/process_audio.hnew file mode 100644index 00000000..d2a95d4b--- /dev/null+++ b/native/window-capture-helper/process_audio.h@@ -0,0 +1,27 @@+#pragma once++#include <windows.h>++#include <cstdint>+#include <functional>++namespace screener::capture {++constexpr UINT32 kAudioSampleRate = 48'000;+constexpr UINT16 kAudioChannels = 2;+constexpr UINT16 kAudioBytesPerSample = 2;+constexpr UINT32 kAudioFramesPerChunk = 960;+constexpr DWORD kAudioBytesPerChunk =+ kAudioFramesPerChunk * kAudioChannels * kAudioBytesPerSample;+constexpr UINT64 kAudioChunkDuration100ns = 200'000;++using PCMWriter =+ std::function<HRESULT(UINT64 timestamp100ns, const BYTE* data, DWORD size)>;++int WriteWindowList();+HRESULT ValidateWindowTarget(UINT64 window_handle, DWORD pid,+ UINT64 expected_creation_time);+HRESULT CaptureProcessAudio(DWORD pid, UINT64 expected_creation_time,+ HANDLE stop_event, const PCMWriter& writer);++} // namespace screener::capturediff --git a/scripts/native-one-viewer-gate.ts b/scripts/native-one-viewer-gate.tsindex ba249cee..a931ea63 100644--- a/scripts/native-one-viewer-gate.ts+++ b/scripts/native-one-viewer-gate.ts@@ -60,6 +60,9 @@ interface GateReport {interface SenderSnapshot {getDisplayMediaRequested: boolean;getDisplayMediaResolved: boolean;+ nativeHardwareStarting: boolean;+ nativeHardwareActive: boolean;+ nativeAdapterLuid: string | null;ready: number;configAccepted: number;encoderInstances: number;@@ -77,6 +80,7 @@ interface SenderSnapshot {maxPeers: number;config: {codec: string | null;+ videoSource: string | null;width: number | null;height: number | null;fps: number | null;@@ -99,6 +103,13 @@ interface SenderSnapshot {}>;}+type GateMode = "vp8" | "h264" | "native-h264";++export function parseGateMode(value: string | undefined): GateMode {+ const normalized = value?.trim();+ return normalized === "h264" || normalized === "native-h264" ? normalized : "vp8";+}+interface ViewerSnapshot {signaling: {authenticateSent: number;@@ -229,16 +240,24 @@ function observeNodeStages(server: ScreenerServer, ledger: SenderStartLedger): v});}-function senderStartObservation(snapshot: SenderSnapshot, expectedCodec = "vp8"): Partial<SenderStartObservation> {+function senderStartObservation(+ snapshot: SenderSnapshot,+ expectedCodec = "vp8",+ expectedVideoSource = "browser",+): Partial<SenderStartObservation> {+ const nativeWindow = expectedVideoSource === "native-window-h264";return {getDisplayMediaRequested: snapshot.getDisplayMediaRequested,getDisplayMediaResolved: snapshot.getDisplayMediaResolved,bridgeConnected: snapshot.ready > 0,fixedHighConfigured: snapshot.configAccepted > 0 &&- snapshot.config?.codec === expectedCodec && snapshot.config.width === 1280 &&+ snapshot.config?.codec === expectedCodec && snapshot.config.videoSource === expectedVideoSource &&+ snapshot.config.width === 1280 &&snapshot.config.height === 720 && snapshot.config.fps === 30 &&snapshot.config.bitrate === 3_000_000 && snapshot.config.encoderInstances === 1,- firstEncodedChunk: snapshot.encoderOutputs > 0,+ firstEncodedChunk: nativeWindow+ ? snapshot.nativeHardwareActive && (snapshot.media?.framesWritten ?? 0) > 0+ : snapshot.encoderOutputs > 0,bridgeGeneration: snapshot.bridgeGeneration,binarySendAttempts: snapshot.binarySendAttempts,binarySendSucceeded: snapshot.binarySendSucceeded,@@ -257,6 +276,28 @@ function senderStartObservation(snapshot: SenderSnapshot, expectedCodec = "vp8")};}+function retainsExpectedBridge(snapshot: SenderSnapshot, nativeWindow: boolean): boolean {+ return nativeWindow+ ? snapshot.bridgeGeneration === 1 && snapshot.binarySendFailed === 0+ : retainsFirstBridgeSend(snapshot);+}++function nativeSenderStartComplete(+ snapshot: SenderSnapshot,+ ledger: SenderStartObservation,+): boolean {+ return !snapshot.getDisplayMediaRequested && !snapshot.getDisplayMediaResolved &&+ snapshot.nativeHardwareStarting && snapshot.nativeHardwareActive &&+ typeof snapshot.nativeAdapterLuid === "string" && snapshot.nativeAdapterLuid.length > 0 &&+ snapshot.encoderInstances === 0 && snapshot.encoderErrors === 0 && snapshot.fatalEvents === 0 &&+ snapshot.config?.videoSource === "native-window-h264" &&+ ledger.siteAccessAccepted && ledger.roomCreated && ledger.hostWssAuthenticated &&+ ledger.bridgeConnected && ledger.fixedHighConfigured && ledger.firstEncodedChunk &&+ ledger.bridgeGeneration === 1 && ledger.binarySendFailed === 0 &&+ ledger.diagnosticsObserved && ledger.framesWritten > 0 &&+ ledger.sourceRtpPacketsWritten > 0 && ledger.sourceRtpBytesWritten > 0;+}+function requiredEnvironmentPath(name: string): string {const value = process.env[name]?.trim();if (!value) throw new Error(`${name} is required`);@@ -276,8 +317,15 @@ function appendLedgerRecord(path: string, record: object): void {async function main(): Promise<void> {const chromePath = requiredEnvironmentPath("SCREENER_NATIVE_GATE_CHROME");const goPath = process.env.SCREENER_NATIVE_GATE_GO?.trim() || "go";- const gateCodec = process.env.SCREENER_NATIVE_GATE_CODEC?.trim() === "h264" ? "h264" : "vp8";- const audioTargetTitle = process.env.SCREENER_NATIVE_GATE_AUDIO_TARGET_TITLE?.trim() || "";+ const gateMode = parseGateMode(process.env.SCREENER_NATIVE_GATE_CODEC);+ const nativeWindow = gateMode === "native-h264";+ const gateCodec = gateMode === "vp8" ? "vp8" : "h264";+ const gateVideoSource = nativeWindow ? "native-window-h264" : "browser";+ const nativeHelperPath = nativeWindow+ ? requiredEnvironmentPath("SCREENER_WINDOW_CAPTURE_HELPER")+ : null;+ const windowTargetTitle = process.env.SCREENER_NATIVE_GATE_WINDOW_TARGET_TITLE?.trim() ||+ (nativeWindow ? "Native Gate Source" : "");const accessKey = randomBytes(9).toString("base64url");const report: GateReport = {schemaVersion: 1,@@ -285,9 +333,13 @@ async function main(): Promise<void> {failedStage: "preflight",stages: {},limitations: [- "The source is a real animated Chrome tab captured through getDisplayMedia, not a game workload.",- "This run proves one WebCodecs object, not one physical or hardware encoder.",- audioTargetTitle+ nativeWindow+ ? "The source is a real animated top-level Chrome window captured through WGC, not a game workload."+ : "The source is a real animated Chrome tab captured through getDisplayMedia, not a game workload.",+ nativeWindow+ ? "Hardware status is the helper's hardware-only MF contract; physical VideoEncode use requires correlated OS telemetry."+ : "This run proves one WebCodecs object, not one physical or hardware encoder.",+ windowTargetTitle? "The loopback run does not prove public STUN, restrictive networks, TURN, packaging, or endurance.": "The loopback run does not prove public STUN, restrictive networks, TURN, audio, packaging, or endurance.",],@@ -323,7 +375,7 @@ async function main(): Promise<void> {ALLOWED_ORIGINS: baseUrl,SITE_ACCESS_PASSWORD: accessKey,ROOM_DATABASE_PATH: "",- PEER_ASSISTED_MEDIA: "false",+ PEER_ASSISTED_MEDIA: "true",STUN_URLS: "",});const roomStore = new ObservedRoomStore({@@ -363,45 +415,66 @@ async function main(): Promise<void> {cdp = await CdpConnection.connect(version.webSocketDebuggerUrl, Date.now() + 5_000);report.stages.preflight = {serverInMemory: config.roomDatabasePath === undefined,- peerAssistedDisabled: !config.peerAssistedMedia,+ peerAssistedEnabled: config.peerAssistedMedia,oneViewerOnly: true,+ videoSource: gateVideoSource,+ nativeHelperConfigured: !nativeWindow || nativeHelperPath !== null,chromeMajor: Number(version.Browser.match(/Chrome\/(\d+)/)?.[1] ?? 0),};report.failedStage = "sender-start";- await createPage(cdp, animatedSourceUrl());- const activeSenderPage = await createPage(cdp, launchUrl, senderProbe());+ await createPage(cdp, animatedSourceUrl(), undefined, nativeWindow);+ const activeSenderPage = await createPage(cdp, launchUrl, senderProbe(), nativeWindow);senderPage = activeSenderPage;- let audioTargetId = "";- if (audioTargetTitle) {+ let windowTargetId = "";+ if (windowTargetTitle) {await waitForSample((deadline) => evaluate<string>(cdp!, activeSenderPage,- `([...document.querySelector('#audio-target').options].find((option) => option.text.includes(${JSON.stringify(audioTargetTitle)}))?.value || '')`, deadline),+ `([...document.querySelector('#window-target').options].find((option) => option.text.includes(${JSON.stringify(windowTargetTitle)}))?.value || '')`, deadline),Boolean,5_000,);- audioTargetId = await evaluate<string>(cdp, activeSenderPage,- `([...document.querySelector('#audio-target').options].find((option) => option.text.includes(${JSON.stringify(audioTargetTitle)}))?.value || '')`, Date.now() + 2_000);+ windowTargetId = await evaluate<string>(cdp, activeSenderPage,+ `([...document.querySelector('#window-target').options].find((option) => option.text.includes(${JSON.stringify(windowTargetTitle)}))?.value || '')`, Date.now() + 2_000);}await evaluate<void>(cdp, activeSenderPage, `(() => {document.querySelector('#server-url').value = ${JSON.stringify(baseUrl)};document.querySelector('#password').value = ${JSON.stringify(accessKey)};- document.querySelector('#codec').value = ${JSON.stringify(gateCodec)};- document.querySelector('#audio-target').value = ${JSON.stringify(audioTargetId)};+ document.querySelector('#codec').value = ${JSON.stringify(gateMode)};+ document.querySelector('#window-target').value = ${JSON.stringify(windowTargetId)};document.querySelector('#start').click();})()`, Date.now() + 5_000);- await waitForSenderStart(- async (deadline) => senderStartObservation(await senderSnapshot(cdp!, activeSenderPage, deadline), gateCodec),- senderLedger,- { timeoutMs: 20_000 },- );+ if (nativeWindow) {+ await waitForSample(async (deadline) => {+ const snapshot = await senderSnapshot(cdp!, activeSenderPage, deadline);+ senderLedger.record(senderStartObservation(snapshot, gateCodec, gateVideoSource));+ return { snapshot, ledger: senderLedger.snapshot() };+ }, ({ snapshot, ledger }) => nativeSenderStartComplete(snapshot, ledger), 20_000);+ } else {+ await waitForSenderStart(+ async (deadline) => senderStartObservation(+ await senderSnapshot(cdp!, activeSenderPage, deadline), gateCodec, gateVideoSource,+ ),+ senderLedger,+ { timeoutMs: 20_000 },+ );+ }const senderReady = await senderSnapshot(cdp, activeSenderPage, Date.now() + 3_000);const senderStartChecks = {- oneEncoderObject: senderReady.encoderInstances === 1,- currentBridge: retainsFirstBridgeSend(senderReady),+ correctVideoSource: senderReady.config?.videoSource === gateVideoSource,+ currentBridge: retainsExpectedBridge(senderReady, nativeWindow),noCriticalErrors: senderReady.encoderErrors === 0 && senderReady.fatalEvents === 0,+ ...(nativeWindow ? {+ noBrowserCapture: !senderReady.getDisplayMediaRequested && !senderReady.getDisplayMediaResolved,+ nativeHardwareStarting: senderReady.nativeHardwareStarting,+ nativeHardwareActive: senderReady.nativeHardwareActive,+ noBrowserVideoEncoder: senderReady.encoderInstances === 0,+ } : { oneEncoderObject: senderReady.encoderInstances === 1 }),+ };+ report.stages["sender-start"] = {+ ...senderLedger.snapshot(), ...senderStartChecks,+ adapterLuid: nativeWindow ? senderReady.nativeAdapterLuid : null,};- report.stages["sender-start"] = { ...senderLedger.snapshot(), ...senderStartChecks };requireChecks(senderStartChecks);report.failedStage = "node-host";const inviteUrl = await evaluate<string>(cdp, activeSenderPage,@@ -435,7 +508,7 @@ async function main(): Promise<void> {pionConnected: sender.peers.length === 1 && sender.peers[0]?.connectionState === "connected" &&["connected", "completed"].includes(sender.peers[0]?.iceConnectionState ?? ""),sameGeneration: identity !== null,- sameBridge: retainsFirstBridgeSend(sender),+ sameBridge: retainsExpectedBridge(sender, nativeWindow),edgeBound: sender.maxPeers <= 1 && sender.peers.length === 1,};report.stages["viewer-signal"] = {@@ -476,13 +549,17 @@ async function main(): Promise<void> {videoReady: viewer.videoReadyState >= 2 && viewer.videoWidth > 0 && viewer.videoHeight > 0,playbackClock: viewer.videoCurrentTime > beforeViewer.videoCurrentTime,renderedFrames: viewer.renderedFrames > beforeViewer.renderedFrames,- pionOutbound: (sender.peers[0]?.packetsSent ?? 0) > (beforeSender.peers[0]?.packetsSent ?? 0) &&- (sender.peers[0]?.bytesSent ?? 0) > (beforeSender.peers[0]?.bytesSent ?? 0),+ pionOutbound: (sender.peers[0]?.packetsSent ?? 0) > 0 &&+ (sender.peers[0]?.bytesSent ?? 0) > 0,sameGeneration: retainsOneViewerIdentity(identity!, viewer.identity, senderIdentity(sender)),- sameBridge: retainsFirstBridgeSend(sender),- oneEncoderObject: sender.encoderInstances === 1,+ sameBridge: retainsExpectedBridge(sender, nativeWindow),criticalErrors: sender.fatalEvents === 0 && sender.encoderErrors === 0,- ...(audioTargetTitle ? {+ ...(nativeWindow ? {+ nativeHardwareActive: sender.nativeHardwareActive,+ noBrowserCapture: !sender.getDisplayMediaRequested && !sender.getDisplayMediaResolved,+ noBrowserVideoEncoder: sender.encoderInstances === 0,+ } : { oneEncoderObject: sender.encoderInstances === 1 }),+ ...(windowTargetTitle ? {audioTrack: viewer.audioTrackCount === 1,audioInbound: viewer.audioPacketsReceived > beforeViewer.audioPacketsReceived,} : {}),@@ -495,6 +572,7 @@ async function main(): Promise<void> {renderedDelta: viewer.renderedFrames - beforeViewer.renderedFrames,audioPacketDelta: viewer.audioPacketsReceived - beforeViewer.audioPacketsReceived,pionPacketDelta: (sender.peers[0]?.packetsSent ?? 0) - (beforeSender.peers[0]?.packetsSent ?? 0),+ pionPacketsSent: sender.peers[0]?.packetsSent ?? 0,pcOrdinal: viewer.identity.pcOrdinal,pionSlot: sender.peers[0]?.slot ?? -1,pionEdgeGeneration: sender.peers[0]?.edgeGeneration ?? 0,@@ -506,8 +584,13 @@ async function main(): Promise<void> {}, 10_000);const finalSender = await senderSnapshot(cdp, activeSenderPage, Date.now() + 3_000);const finalChecks = {- finalBridge: retainsFirstBridgeSend(finalSender),+ finalBridge: retainsExpectedBridge(finalSender, nativeWindow),finalCriticalErrors: hasNoCriticalSenderErrors(finalSender),+ finalVideoSource: finalSender.config?.videoSource === gateVideoSource,+ ...(nativeWindow ? {+ finalNativeHardwareActive: finalSender.nativeHardwareActive,+ finalNoBrowserVideoEncoder: finalSender.encoderInstances === 0,+ } : {}),};report.stages["viewer-media"] = { ...report.stages["viewer-media"], ...finalChecks };requireChecks(finalChecks);@@ -522,7 +605,7 @@ async function main(): Promise<void> {finalSenderEvidence = await verifyFinalSenderEvidence(() => senderSnapshot(finalCdp, finalSenderPage, Date.now() + 1_000),(snapshot) => {- senderLedger.record(senderStartObservation(snapshot, gateCodec));+ senderLedger.record(senderStartObservation(snapshot, gateCodec, gateVideoSource));return senderLedger.snapshot();},);@@ -544,6 +627,8 @@ async function main(): Promise<void> {export function senderProbe(): string {return `(() => {const state = { getDisplayMediaRequested: false, getDisplayMediaResolved: false,+ nativeHardwareStarting: false, nativeHardwareActive: false,+ nativeAdapterLuid: null,ready: 0, configAccepted: 0,encoderInstances: 0, encoderOutputs: 0, encoderErrors: 0, fatalEvents: 0,bridgeGeneration: 0, binarySendAttempts: 0, binarySendSucceeded: 0, binarySendFailed: 0,@@ -580,6 +665,22 @@ export function senderProbe(): string {if (message.kind === 'ready') state.ready += 1;if (message.kind === 'config-accepted') state.configAccepted += 1;if (message.kind === 'fatal') mark('fatalEvents');+ if (message.kind === 'native-video-state' && message.status?.hardwareOnly === true) {+ if (message.status.state === 'starting' && Number.isFinite(message.status.adapterIndex) &&+ typeof message.status.adapterName === 'string' && message.status.adapterName.length > 0 &&+ typeof message.status.adapterLuid === 'string' && message.status.adapterLuid.length > 0 &&+ Number.isFinite(message.status.mftIndex) &&+ typeof message.status.mftName === 'string' && message.status.mftName.length > 0 &&+ typeof message.status.mftClsid === 'string' && message.status.mftClsid.length > 0) {+ state.nativeHardwareStarting = true;+ state.nativeAdapterLuid = message.status.adapterLuid;+ }+ if (message.status.state === 'active' && state.nativeHardwareStarting &&+ message.status.profileLevelId === '42c01f' && message.status.width === 1280 &&+ message.status.height === 720 && message.status.fps === 30) {+ state.nativeHardwareActive = true;+ }+ }if (message.kind === 'ready' || message.kind === 'viewer-count') {state.activeViewers = finite(message.activeViewers);}@@ -627,6 +728,7 @@ export function senderProbe(): string {let message; try { message = JSON.parse(value); } catch { message = null; }if (message?.kind === 'config') state.config = {codec: typeof message.codec === 'string' ? message.codec : null,+ videoSource: typeof message.videoSource === 'string' ? message.videoSource : null,width: finite(message.width), height: finite(message.height), fps: finite(message.fps),bitrate: finite(message.bitrate), encoderInstances: finite(message.encoderInstances),};@@ -827,9 +929,16 @@ function viewerProbe(): string {})();`;}-async function createPage(cdp: CdpConnection, url: string, probe?: string): Promise<PageHandle> {+async function createPage(+ cdp: CdpConnection,+ url: string,+ probe?: string,+ newWindow = false,+): Promise<PageHandle> {const deadline = Date.now() + 15_000;- const created = await cdp.call<{ targetId: string }>("Target.createTarget", { url: "about:blank", background: false }, undefined, deadline);+ const created = await cdp.call<{ targetId: string }>(+ "Target.createTarget", { url: "about:blank", background: false, newWindow }, undefined, deadline,+ );const attached = await cdp.call<{ sessionId: string }>("Target.attachToTarget", { targetId: created.targetId, flatten: true }, undefined, deadline);const page = { sessionId: attached.sessionId };await Promise.all([@@ -1055,9 +1164,11 @@ async function removeGateProfile(profile: string): Promise<boolean> {10_000, { ...process.env, SCREENER_GATE_PROFILE_TO_REMOVE: profile });}-function animatedSourceUrl(): string {+export function animatedSourceUrl(): string {const html = `<!doctype html><title>Native Gate Source</title><canvas width="1280" height="720"></canvas>- <script>const c=document.querySelector('canvas'),x=c.getContext('2d');let f=0;+ <script>const a=new AudioContext(),o=a.createOscillator(),g=a.createGain();g.gain.value=.02;+ o.connect(g).connect(a.destination);o.start();+ const c=document.querySelector('canvas'),x=c.getContext('2d');let f=0;setInterval(()=>{f++;x.fillStyle='hsl('+f%360+' 80% 35%)';x.fillRect(0,0,1280,720);x.fillStyle='white';x.fillRect(f*13%1280,0,20,720);x.font='64px monospace';x.fillText(f,40,90)},33);<\/script>`;return `data:text/html;charset=utf-8,${encodeURIComponent(html)}`;diff --git a/scripts/package-native-sender.ps1 b/scripts/package-native-sender.ps1index 382e0597..551d5b21 100644--- a/scripts/package-native-sender.ps1+++ b/scripts/package-native-sender.ps1@@ -68,9 +68,11 @@ try {$env:CGO_ENABLED = $previousCGO}-& (Join-Path $repositoryRoot 'native\process-audio-helper\build.ps1') -OutputDirectory $packageDirectory | Out-Null-$helperObject = Join-Path $packageDirectory 'main.obj'-if (Test-Path -LiteralPath $helperObject) { Remove-Item -LiteralPath $helperObject -Force }+& (Join-Path $repositoryRoot 'native\window-capture-helper\build.ps1') -OutputDirectory $packageDirectory | Out-Null+@('window-capture.obj', 'process-audio.obj') | ForEach-Object {+ $helperObject = Join-Path $packageDirectory $_+ if (Test-Path -LiteralPath $helperObject) { Remove-Item -LiteralPath $helperObject -Force }+}$moduleManifest = @()foreach ($line in $moduleLines) {diff --git a/scripts/required-project-paths.txt b/scripts/required-project-paths.txtindex fcab6b50..fccf309b 100644--- a/scripts/required-project-paths.txt+++ b/scripts/required-project-paths.txt@@ -25,6 +25,12 @@ docs/deployment.mddocs/adr/0001-p2p-first-media-topology.mddocs/research/agent-context-governance.mddocs/research/webrtc-p2p-screen-sharing.md+native/sender/README.md+native/window-capture-helper/README.md+native/window-capture-helper/build.ps1+native/window-capture-helper/main.cpp+native/window-capture-helper/process_audio.cpp+native/window-capture-helper/process_audio.hscripts/required-project-paths.txtscripts/check-project-state.shscripts/check-project-state.ps1diff --git a/tests/native-one-viewer-gate-core.test.ts b/tests/native-one-viewer-gate-core.test.tsindex d7cb16fb..21d7a567 100644--- a/tests/native-one-viewer-gate-core.test.ts+++ b/tests/native-one-viewer-gate-core.test.ts@@ -17,7 +17,12 @@ import {type SenderIdentityEvidence,type ViewerIdentityEvidence,} from "../scripts/native-one-viewer-gate-core";-import { profileCleanupScript, senderProbe } from "../scripts/native-one-viewer-gate";+import {+ animatedSourceUrl,+ parseGateMode,+ profileCleanupScript,+ senderProbe,+} from "../scripts/native-one-viewer-gate";const viewer: ViewerIdentityEvidence = {socketCount: 1, socketOrdinal: 1, authGeneration: 1,@@ -33,6 +38,9 @@ const clean: CleanupResult = {};interface BridgeProbeSnapshot {+ nativeHardwareStarting: boolean;+ nativeHardwareActive: boolean;+ nativeAdapterLuid: string | null;bridgeGeneration: number;binarySendAttempts: number;binarySendSucceeded: number;@@ -41,6 +49,8 @@ interface BridgeProbeSnapshot {postSendDiagnosticsObserved: boolean;postSendDiagnosticsSequence: number;fatalEvents: number;+ encoderInstances: number;+ config: { codec: string; videoSource: string } | null;}class ProbeWebSocket {@@ -79,6 +89,40 @@ function bridgeProbeContext(): {}describe("native one-viewer gate invariants", () => {+ it("selects the explicit native H.264 source without changing its RTP codec", () => {+ expect(parseGateMode("native-h264")).toBe("native-h264");+ expect(parseGateMode("h264")).toBe("h264");+ expect(parseGateMode("unsupported")).toBe("vp8");+ });++ it("keeps the fixed native target title and an audible process-tree source", () => {+ const source = decodeURIComponent(animatedSourceUrl());+ expect(source).toContain("<title>Native Gate Source</title>");+ expect(source).toContain("createOscillator()");+ });++ it("retains the exact native hardware lifecycle without creating a browser encoder", () => {+ const probe = bridgeProbeContext();+ const socket = new probe.WebSocket("ws://127.0.0.1/media");+ socket.send(JSON.stringify({ kind: "config", codec: "h264", videoSource: "native-window-h264",+ width: 1280, height: 720, fps: 30, bitrate: 3_000_000, encoderInstances: 1 }));+ socket.emitMessage(JSON.stringify({ kind: "native-video-state", status: {+ state: "starting", hardwareOnly: true, adapterIndex: 0, adapterName: "GPU",+ adapterLuid: "0:1", mftIndex: 0, mftName: "Hardware H264", mftClsid: "clsid",+ } }));+ socket.emitMessage(JSON.stringify({ kind: "native-video-state", status: {+ state: "active", hardwareOnly: true, profileLevelId: "42c01f",+ width: 1280, height: 720, fps: 30,+ } }));+ expect(probe.snapshot()).toMatchObject({+ nativeHardwareStarting: true,+ nativeHardwareActive: true,+ nativeAdapterLuid: "0:1",+ encoderInstances: 0,+ config: { codec: "h264", videoSource: "native-window-h264" },+ });+ });+it("rejects cumulative evidence from a rebuilt Viewer PC", () => {const baseline = captureOneViewerIdentity(viewer, sender);expect(baseline).not.toBeNull();