Skip to content

armgit / Piik

Public Imported from GitHub
0

Free, open-source screen sharing for private live streams with friends. Watch together in a browser or self-host Piik. 免费开源的私密屏幕共享,支持游戏直播、一起看电影与自部署。

feat(native): add WGC hardware H264 source (#109)

AAAa560f54bcc3c
diff --git a/.github/workflows/native-sender-package.yml b/.github/workflows/native-sender-package.yml
index 69af246a..2dadd58c 100644
--- a/.github/workflows/native-sender-package.yml
+++ b/.github/workflows/native-sender-package.yml
@@ -5,7 +5,7 @@ on:
pull_request:
paths:
- "native/sender/**"
- - "native/process-audio-helper/**"
+ - "native/window-capture-helper/**"
- "scripts/package-native-sender.ps1"
- ".github/workflows/native-sender-package.yml"
push:
@@ -13,7 +13,7 @@ on:
- main
paths:
- "native/sender/**"
- - "native/process-audio-helper/**"
+ - "native/window-capture-helper/**"
- "scripts/package-native-sender.ps1"
- ".github/workflows/native-sender-package.yml"
diff --git a/docs/adr/0008-window-scoped-audio-capture.md b/docs/adr/0008-window-scoped-audio-capture.md
index e089d448..a344e67c 100644
--- a/docs/adr/0008-window-scoped-audio-capture.md
+++ b/docs/adr/0008-window-scoped-audio-capture.md
@@ -2,9 +2,10 @@
Date: 2026-08-21
-Status: Accepted. The browser hint, opt-in Native Windows P1 slice, and bounded
-Windows x64 evaluation-package path are source-complete; Native remains
-default-off, formally unreleased, and undeployed.
+Status: Accepted. The browser hint and opt-in Native Windows audio slice are
+source-complete, including the bounded Windows x64 evaluation-package path.
+The paired WGC/MF hardware-video source passed its one-Viewer acceptance smoke;
+all Native sources remain default-off, formally unreleased, and undeployed.
## Context
@@ -33,12 +34,12 @@ Windows 11 desktop (build 22000 or newer) and is not a Windows 10 fallback.
2. The default Web route remains unchanged in topology, codecs, and relay
behavior. No Web Audio mixer, SDP rewrite, or app-owned audio bitrate knob
is introduced by this ADR.
-3. The native sender's next audio slice is an explicit opt-in
- `window-process-audio` mode on Windows 11. The local capture session owns
- one selected target PID/process tree, one WGC/DXGI video source, and one
- WASAPI loopback audio source. Process IDs, creation tokens, titles, paths,
- device identity, and PCM stay local; they do not enter signaling, URLs,
- logs, or storage.
+3. The native sender has an explicit Windows 11 local window target. Browser
+ VP8/H.264 may use that target only for `window-process-audio`; the separate
+ `native-window-h264` source uses the same bound HWND/PID/creation time for
+ WGC/DXGI video and WASAPI process-tree audio. Process IDs, HWNDs, creation
+ tokens, titles, paths, device identity, PCM, and local hardware status stay
+ local; they do not enter signaling, URLs, logs, or storage.
4. If the target process exits, produces no render stream, or the loopback
activation is denied, the sender reports `audio unavailable` or `audio
silent` and stops/asks the user. It never widens to whole-system audio.
@@ -50,7 +51,7 @@ Windows 11 desktop (build 22000 or newer) and is not a Windows 10 fallback.
This is an interface boundary, not a new framework or a current wire change:
```text
-NativeCaptureTarget { pid: uint32, creationTime: uint64, includeProcessTree: true }
+NativeCaptureTarget { hwnd: uint64, pid: uint32, creationTime: uint64, includeProcessTree: true }
NativeCaptureSession.start(target)
-> VideoFrames(WGC/DXGI) + AudioPcm(WASAPI process loopback)
-> existing native sender timeline/encoder/fanout
@@ -80,9 +81,9 @@ timestamp, duration, bounded payload); it does not add a public signaling field.
The Go session owns one shared Opus `TrackLocalStaticRTP` and adds it before the
offer on every native PeerConnection. Audio packetization uses the 48 kHz RTP
clock and the source timestamp delta, so one encoded packet stream can feed the
-existing two-edge cap. The WGC/DXGI video adapter remains the next small slice;
-its `SystemRelativeTime` is retained in the fixture so it can share the same
-QPC origin when native video replaces the Web source.
+existing two-edge cap. In this historical audio-only phase, the WGC/DXGI video
+adapter was the next small slice; its retained `SystemRelativeTime` let the
+completed native-video path share the same QPC origin.
Expected implementation size is about **500--800 new LOC**, excluding the
browser/Windows SDK and existing video/MF fixture code: Windows loopback
@@ -112,11 +113,29 @@ visual marker, while an independent voice process and notification emit
different markers. Require an Opus inbound track and rendered video/audio,
monotonic QPC timestamps, and a bounded A/V offset; verify the unrelated
markers are absent and that target exit/silence stops or asks without widening
-capture. SFU/TURN, second Viewer, endurance, and the native WGC video swap are
-follow-up gates. On 2026-08-21, 100 20-ms chunks isolated a 440 Hz target from
+capture. SFU/TURN, second Viewer, and endurance remain follow-up gates; paired
+WGC video completion is recorded below. On 2026-08-21, 100 20-ms chunks isolated a 440 Hz target from
an independent 880 Hz process by 4017.8x. Chrome 151 then received one audio
track and 495 inbound Opus packets while video decoded/rendered 296 frames.
+## P1 WGC/MF Completion
+
+The paired native-video source reuses the proven hardware fixture rather than
+adding a second encoder framework. WGC creates a capture item directly from the
+bound HWND on the selected D3D11 device. The frame pool is free-threaded and is
+recreated on content-size changes. D3D11 VideoProcessor performs aspect-fit
+BGRA-to-NV12 conversion at fixed 1280x720/30, then the existing adapter-bound,
+hardware-only asynchronous MF H.264 path emits Annex-B `42c01f` access units.
+There is no software MFT, alternate codec, monitor, or browser-video fallback.
+
+The helper multiplexes process-tree PCM, H.264 access units, and bounded local
+status. Go writes H.264 directly to the existing shared fanout; the browser UI
+receives only a one-way local copy for `VideoDecoder` preview. PCM retains the
+existing WebCodecs Opus bridge. PLI/FIR recovery is generation-bound and
+coalesced into a helper key-frame command. Normal browser VP8/H.264 sources
+remain explicit alternatives, with VP8 as the default; neither is an implicit
+runtime downgrade.
+
## Consequences
- Current Web users get a useful Chromium hint and an honest status without a
@@ -131,8 +150,8 @@ track and 495 inbound Opus packets while video decoded/rendered 296 frames.
## Follow-Up TODO
+- Keep a second Viewer, SFU/TURN, endurance, and real-game A/V sync as separate
+ follow-ups; the one-Viewer H.264 and attributed `VideoEncode` gate passed.
- Download and run the short-lived Windows x64 evaluation artifact; keep
explicit target selection default-off. A formal release remains blocked on
the project license, signing, and its own acceptance boundary.
-- Keep native WGC video, a second Viewer, SFU/TURN, endurance, and real-game
- A/V sync as separate follow-ups.
diff --git a/docs/project-memory.md b/docs/project-memory.md
index 388f6458..3565c421 100644
--- a/docs/project-memory.md
+++ b/docs/project-memory.md
@@ -5,7 +5,7 @@ Last updated: 2026-08-21
## Confirmed Intent
- Build low-latency game sharing for one broadcaster and trusted friends; public or large broadcasts belong on OBS/Twitch-class services.
-- Viewers join from normal desktop/mobile browsers. A later native sender may share encoding and improve capture/audio without changing the Web viewer requirement.
+- Viewers join from normal desktop/mobile browsers. Native senders may improve capture/audio without changing the Web viewer requirement.
- Use a small central service for access, rooms, signaling, deterministic topology, STUN, observability, and bounded SFU-root capacity.
- Minimize server bandwidth. Ordinary peer ICE is STUN-only. Failure tries restart, rebuild, alternate peer and bounded SFU/UDP roots; only then may the controller authorize short-lived TURN for one selected exceptional edge before clear failure. TURN is transport, not topology.
- Non-server nodes have at most two downstream edges; browser relays stay at one until resource gates pass. SFU normally feeds one or two roots that retain peer descendants; separately capped server edges may serve exceptional viewers that cannot attach behind a healthy root.
@@ -23,7 +23,7 @@ Last updated: 2026-08-21
- Use direct host P2P for one or two viewers, then the bounded controller for every room when `PEER_ASSISTED_MEDIA=true`. Room `1` is historical smoke, not a runtime gate; resource, quality, recovery, SFU/UDP, bounded-failure, and browser/mobile evidence remain separate.
- Browser relays resend remote `MediaStreamTrack` values and re-encode at each hop; WebRTC does not guarantee a shared encoder across peer connections, so measure the cost.
- Keep experiments bounded: the standard representation sequence is below; native RTP relay, encoded-object striping, and FEC remain separate.
-- ADR-0006 remains no-go. VP8 is default; H.264 loopback rendered 298/299. Windows x64 evaluation packaging is source-complete; downloaded use, Pion timing, hardware, multi-viewer/endurance/public proof remain open.
+- ADR-0006's historical browser-bridge canary remains no-go; VP8 stays default. Browser H.264 rendered 298/299; Native WGC/MF rendered 203 with PID/LUID-correlated `VideoEncode`. Downloaded use, multi-viewer/endurance/public proof remain open.
- Mobile Web Host unsupported. Android 14+ sender is direct-child video-only; device, audio, rotation, and SFU remain open. iOS deferred; TV output is local-only P2.
- ADR-0005 accepts direct/peer UDP, bounded SFU roots, then optional selected-edge TURN. Production now enables the one-shot selected-edge tuple after SFU; participant-wide TURN remains rejected and ordinary peer ICE stays STUN-only. No real TURN/SFU media canary has yet been run.
- The controller is process-enabled, not room-allowlisted: `PEER_ASSISTED_MEDIA=true` gives every normal room the same direct/peer -> SFU -> selected-edge path with per-room state. Room `1` is historical smoke. Preserve sticky P2P, mobile leaves and break-before-make; recovery gets one attempt per layer (ICE restart, same-parent rebuild, alternate peer, then SFU), never three identical retries.
@@ -34,7 +34,6 @@ Last updated: 2026-08-21
- Viewer-local names and the opt-in Web participant roster remain control-plane-only. Web Host names are deployed in the current release and stay socket/localStorage-only; Native remains outside the capability boundary.
- Do not add scene detection, dynamic-FPS control, or forced AV1 without negotiation, encode, game, CPU/GPU, and sender evidence.
- ADR-0002 grant/public access, the v3 room-password migration, and Web Host display names are deployed. Names/presence remain session-only without account or roster tables.
-- Deferred architecture audit: `docs/maintenance.md`.
## Current Execution Principle
@@ -53,7 +52,7 @@ Last updated: 2026-08-21
- Chrome 151 synthetic topology/quality runs kept fanout 2/1 and decoding; one relay close recovered in 5.32 seconds. Control evidence only.
- Room `1` publishes `HIGH+LOW` with Dynacast off and subscriber `HIGH` ceilings, and now has bounded local quality reparenting. Both remain unverified on real media; test zero-child leaves and calibrated C+B loss before broad rollout.
- C+B uses three hard-bad pairs, one-use samples and guarded intent/cooldown. Deployed admission rescue moves the oldest childless zero-capacity Host leaf below an unassigned one-slot relay in one revision; no score/timer/global rebalance. Deployment passed, but no real room triggered it.
-- Web Host name/presence and window-audio hint are deployed. Native source accepts authenticated peer-assisted direct-child assignments, ignores presence as an edge authority, and reports unsupported SFU/selected ingress as bounded route failure; it does not implement LiveKit or TURN media. Native H.264 source and Windows x64 evaluation packaging are present, but no Native release/deployment exists; VP8 remains the Web/production default. See `docs/research/native-h264-opt-in-path.md`.
+- Web Host name/presence and window-audio hint are deployed. Native accepts bounded peer-assisted direct children and fails unsupported SFU/selected ingress; it has no LiveKit/TURN media. Its WGC/MF H.264 direct-child smoke rendered 203 frames plus 500 Opus packets and matched helper PID/LUID `VideoEncode`; packaging is source-only, unreleased and undeployed. VP8 remains the Web/production default. See `docs/research/native-h264-opt-in-path.md`.
## Provisional Quality Targets
@@ -69,7 +68,7 @@ These are measurement gates, not performance claims.
- Sustainable count by hardware, quality, network, and route: finish instrumented `1/3/5/8`, then pass a 20-viewer matrix before changing the accepted target default to 20.
- Whether ADR-0004 passes fanout, re-encoding, depth-four latency, reparenting, silent-partition, and mobile-leaf gates.
- Whether source-complete selected-edge passes forced relay, expiry/failure, mobile, 1-GiB resource and relay-bandwidth gates. Coturn's bearer is non-revocable until expiry; application guards and TTL only bound reuse. Performance comparison and the retained matrix follow functional landing; Media TCP is out of scope.
-- Exact mobile lifecycle behavior and whether Windows per-application audio is required for the first release.
+- Exact mobile lifecycle behavior.
- Project license and distribution model, which determines whether GPL/AGPL sources can move beyond study-only use.
- Initial deployment regions and expected mainland China, Hong Kong, and overseas network mix.
- Whether voice chat ever enters scope or Screener stays complementary to an existing voice application.
diff --git a/docs/research/native-h264-hardware-decision.md b/docs/research/native-h264-hardware-decision.md
index 6b81e4a3..85a3ece9 100644
--- a/docs/research/native-h264-hardware-decision.md
+++ b/docs/research/native-h264-hardware-decision.md
@@ -2,16 +2,21 @@
Date: 2026-08-20
-Status: `no-go-native-h264-hardware-pinned-fmtp`
+Status: Historical gate. Its `no-go-native-h264-hardware-pinned-fmtp`
+stop resolved when the explicit opt-in Pion path registered exact `42c01f`;
+the product WGC/MF path later passed its attributed one-Viewer smoke.
The later source-only functional slice in
[`native-h264-opt-in-path.md`](./native-h264-opt-in-path.md) registers the
-fixture profile explicitly and proves one H.264 Host/Viewer loopback. It does
-not change this spike's physical-hardware decision: WebCodecs hardware
-attribution remains unverified and production remains VP8/default.
-
-This spike does not amend ADR-0006 or authorize another product, room, or
-production run. It answers one narrow question: can a bounded Windows H.264
+fixture profile explicitly and proves one browser H.264 Host/Viewer loopback.
+The native-window source now compiles this fixture's MF encoder source into the
+product helper, retaining hardware-only enumeration and exact readback. That
+path rendered 203 frames for one Chrome Viewer while the exact helper PID and
+adapter LUID matched nonzero Windows `VideoEncode`; offline fixture telemetry
+remains separate evidence.
+
+This spike did not amend ADR-0006 or authorize another product, room, or
+production run. It answered one narrow question: can a bounded Windows H.264
path satisfy the project's physical hardware-encode and existing default Pion
fmtp contracts before product integration?
@@ -20,7 +25,8 @@ emitted SPS differed from the requested codec string. A subsequent
hardware-only Media Foundation run did prove the selected NVIDIA encoder and
GPU engine, but emitted exact `profile-level-id=42c01f`, which is absent from
the default Pion mode-1 fmtp set. It therefore stopped before Pion, a Viewer,
-rooms, or product wiring. Neither result authorizes an H.264 product switch.
+rooms, or product wiring. The later opt-in source resolved that exact
+registration mismatch without changing the browser VP8 default.
## Decision Summary
@@ -31,8 +37,8 @@ rooms, or product wiring. Neither result authorizes an H.264 product switch.
| Did the requested codec string describe the emitted SPS exactly? | No. The requested `42e01f` produced SPS bytes `42 04 1f`. |
| Did forced key frames carry in-band recovery data? | Yes in this fixture: all six key chunks contained AUD, SPS, PPS, and IDR NAL units. |
| Can Pion v1.10.5 packetize the resulting Annex-B access units? | Yes statically: its `H264Payloader` splits 3/4-byte start codes, suppresses AUD/filler, emits SPS/PPS as STAP-A, and fragments slices as FU-A. |
-| Is unmodified browser Viewer decode proven? | No. No PeerConnection, room, Viewer, or production service participated. |
-| Should the small WebCodecs/Pion codec diff be implemented? | No. It would preserve the two unresolved product gates: physical encoder identity and exact emitted-fmtp registration plus Viewer interoperability. |
+| Did this isolated spike prove unmodified browser Viewer decode? | No. No PeerConnection, room, Viewer, or production service participated. A later opt-in loopback records that evidence separately. |
+| Did this spike authorize the small WebCodecs/Pion codec diff? | No. The later exact-`42c01f` opt-in reopened and proved that boundary separately. |
| Did the Media Foundation fixture prove physical H.264 hardware encode? | Yes for one local RTX 4070 SUPER run: adapter-LUID-bound hardware enumeration, D3D11 awareness, the NVIDIA H.264 Encoder MFT, and process-plus-LUID `VideoEncode` activity agreed. |
| Did that hardware MFT emit an exact default Pion mode-1 fmtp string? | No. It emitted `42c01f`; the default set is `42001f`, `42e01f`, `4d001f`, and `64001f`. The fixture failed closed without changing the request or adding a fallback. This is not a standards-level profile incompatibility. |
@@ -178,10 +184,10 @@ preference, not physical hardware evidence.
## Media Foundation Hardware Fixture
The follow-up fixture is retained at
-`native/fixtures/mf-h264-hardware/`. It is a Windows-only offline C++ program,
-not a sender component. Its generated object and executable stay outside the
-repository. It has one H.264 path and no software MFT, codec, adapter, room,
-capture, audio, signaling, Pion, or Viewer fallback.
+`native/fixtures/mf-h264-hardware/`. Its build compiles the shared encoder
+source from `native/window-capture-helper/` with fixture-only synthetic input
+and telemetry. The generated fixture executable stays outside the repository
+and has no room, capture, audio, signaling, Pion, or Viewer fallback.
### Fixed Contract
diff --git a/docs/research/native-h264-opt-in-path.md b/docs/research/native-h264-opt-in-path.md
index 7d331359..c46beb5c 100644
--- a/docs/research/native-h264-opt-in-path.md
+++ b/docs/research/native-h264-opt-in-path.md
@@ -1,27 +1,79 @@
# Native H.264 Opt-In Path
-## 2026-08-21 00:15
+## 2026-08-21 Native Window Source
-The native sender now has a small, explicit H.264 mode. VP8 remains the default
-and the Web sender is unchanged. A sender chooses H.264 in its local Codec
-control; the choice is carried only through the loopback start/media handshake.
+The Native sender now has two explicit local video sources. Browser VP8 remains
+the default; browser H.264 remains available as the earlier opt-in. The new
+`native-window-h264` source is Windows 11-only and requires the user to select
+one local opaque window target. It never silently replaces a browser source or
+falls back to software encode, another codec, monitor capture, or system audio.
-The H.264 path is:
+The native-window path is:
```text
-Chrome WebCodecs H.264 Annex-B
- -> existing 17-byte local frame envelope
+opaque local target -> HWND + PID + process creation time
+ -> WGC BGRA surface on one selected D3D11 adapter
+ -> D3D11 VideoProcessor 1280x720 NV12
+ -> adapter-bound hardware-only MF H.264 42c01f Annex-B
+ -> generation-bound Go Session.WriteMedia
-> one Go queue and Pion H264Payloader
-> one TrackLocalStaticRTP
-> at most two independent PeerConnections
```
-The Go session creates a private Pion `MediaEngine` only for H.264 sessions. It
-registers the fixture-derived Constrained Baseline `profile-level-id=42c01f`,
-`packetization-mode=1`, and a dedicated dynamic payload type, then restricts
-the sender transceiver to that H.264 capability. The shared fanout and keyframe
-feedback remain unchanged. The default VP8 session still uses Pion's default
-API and payloader.
+The helper revalidates HWND-to-PID and PID creation time immediately before WGC
+activation. The same process identity feeds WASAPI application loopback with
+`INCLUDE_TARGET_PROCESS_TREE`. One private stdout protocol multiplexes bounded
+PCM, H.264 access units, and local hardware status. PID, HWND, creation time,
+adapter/MFT identity, PCM, and status never enter signaling. Go sends H.264
+directly to the existing fanout; a second local copy travels only to the sender
+UI's `VideoDecoder` preview and never returns to Go. PCM retains the existing
+local WebCodecs Opus bridge.
+
+PLI/FIR and fanout recovery requests enter the generation-bound helper stdin as
+one coalesced `K` command. The helper forces an in-band SPS/PPS/IDR recovery
+unit, while the existing Pion `H264Payloader`, exact `42c01f` MediaEngine, one
+shared track, and two-edge cap remain unchanged. Browser VP8/H.264 use their
+existing `getDisplayMedia` and `VideoEncoder` path.
+
+Both the product helper and offline fixture compile the same MF encoder source.
+The product build and the fixture build pass MSVC `/W4 /WX`; focused Go tests
+cover bound target arguments, multiplexed protocol bounds, key-frame command
+forwarding, generation/source validation, and the peer-assisted Host contract.
+
+## Native Window Acceptance Evidence
+
+One bounded Chrome 151 run used a real animated top-level window, process-tree
+audio, the explicit Native source, and the peer-assisted direct-child wire. It
+created no browser capture or `VideoEncoder`; the ordinary Viewer received
+1,313 packets/1,431,696 bytes during the sample, decoded and rendered 203
+1280x720 frames, and received 500 Opus packets without a fatal/encoder error.
+The helper wrote 30 frames and 174 source RTP packets before the retained
+sender-start snapshot.
+
+The exact helper process was PID 15060. Its Windows GPU Engine instance
+`pid_15060_luid_0x00000000_0x0001a496_phys_0_eng_6_engtype_videoencode`
+reached 2.98028% utilization; the helper's independently validated adapter
+status was `0x00000000:0x0001a496`. This is process-and-adapter-correlated
+hardware evidence, not a throughput or performance claim. The generic gate
+reported failure only because its asynchronous Pion diagnostics delta stayed
+zero after the Viewer counters advanced; the run was not repeated for that
+known sampling residue.
+
+Primary sources checked 2026-08-21:
+
+- [Windows Graphics Capture](https://learn.microsoft.com/en-us/windows/uwp/audio-video-camera/screen-capture)
+- [`IGraphicsCaptureItemInterop::CreateForWindow`](https://learn.microsoft.com/en-us/windows/win32/api/windows.graphics.capture.interop/nf-windows-graphics-capture-interop-igraphicscaptureiteminterop-createforwindow)
+- [`Direct3D11CaptureFramePool::CreateFreeThreaded`](https://learn.microsoft.com/en-us/uwp/api/windows.graphics.capture.direct3d11captureframepool.createfreethreaded)
+- [`CreateDirect3D11DeviceFromDXGIDevice`](https://learn.microsoft.com/en-us/windows/win32/api/windows.graphics.directx.direct3d11.interop/nf-windows-graphics-directx-direct3d11-interop-createdirect3d11devicefromdxgidevice)
+- [Microsoft MIT HWND capture sample](https://github.com/microsoft/Windows.UI.Composition-Win32-Samples/tree/master/cpp/ScreenCaptureforHWND)
+- [`ID3D11VideoDevice::CreateVideoProcessorInputView`](https://learn.microsoft.com/en-us/windows/win32/api/d3d11/nf-d3d11-id3d11videodevice-createvideoprocessorinputview)
+- [`ID3D11VideoContext::VideoProcessorSetOutputBackgroundColor`](https://learn.microsoft.com/en-us/windows/win32/api/d3d11/nf-d3d11-id3d11videocontext-videoprocessorsetoutputbackgroundcolor)
+
+## 2026-08-21 Browser H.264 Source
+
+The earlier browser H.264 source remains an explicit option. Its loopback path
+is Chrome WebCodecs Annex-B -> the local envelope -> the same Go/Pion fanout.
## Bounded Loopback Evidence
@@ -47,21 +99,22 @@ loopback, not a public-network, multi-viewer, endurance, or codec benchmark.
## Hardware Boundary
-The Media Foundation/NVIDIA fixture remains the physical hardware evidence and
-informed the exact `42c01f` profile and in-band SPS/PPS/keyframe contract. The
-product sender requests WebCodecs H.264 hardware acceleration, but browser
-`isConfigSupported()` and the preference string do not prove which encoder was
-selected. The UI therefore reports hardware preference separately from hardware
-evidence, which remains unverified. Physical hardware attribution and direct
-Media Foundation capture integration are later work; this change does not
-silently claim either one.
+The Media Foundation/NVIDIA fixture established the physical hardware gate and
+the exact `42c01f` in-band recovery contract. Browser H.264 still has only a
+hardware preference. The native-window source instead enumerates only an
+adapter-LUID-bound hardware MFT, requires D3D11 awareness and exact codec
+readback, and has no software retry. The Native acceptance run above adds the
+required product-process `VideoEncode` attribution; fixture evidence remains a
+separate encoder-contract measurement.
## Scope
-- No production deployment was made.
+- Browser VP8 remains the default and no production deployment switch is made.
- No SFU, TURN, Web viewer, access protocol, or default VP8 behavior changed.
- No codec matrix, benchmark, second viewer, or endurance run is required for
this landing slice.
+- The Native source reuses existing controller compatibility and evaluation
+ packaging changes; it adds no topology.
- H.264 distribution still carries the existing patent/license review boundary.
---
diff --git a/docs/status.md b/docs/status.md
index dcc93d6a..b3db49b9 100644
--- a/docs/status.md
+++ b/docs/status.md
@@ -7,7 +7,7 @@ Last updated: 2026-08-21
`https://share.bonfire.icu` serves exact `fd76277b05d491af8840b28f3132b7ff445d3cbe`; the 915,377-byte artifact SHA-256 is `8da4b8b2b1ae4b82add615f4367ee447b5ade86c4e58a938e51085940b9867d3`.
The cutover held the lock 8,389 ms and returned local health 520 ms after stop. SQLite v3 has five rooms; Screener, LiveKit, coturn and nginx are healthy with zero restarts.
Ordinary ICE is STUN-only; all rooms use the bounded peer/SFU-UDP controller and selected-edge UDP TTL 120. Room `1` is historical smoke and no real TURN/SFU media canary has run.
-Web Host names/audio hint are deployed. Native H.264 and Windows x64 packaging are source-only; Web defaults VP8.
+Web names/audio hint are deployed. Native WGC/MF/package is source-only; Web defaults VP8.
## Execution Principle
@@ -18,7 +18,7 @@ Flagship; parallel; min run/smoke/rollback; benchmark later. Active UI/config/lo
- Capture precedes room creation; source/quality changes preserve peers, while pause disables audio/video tracks together and keeps connections. Quality defaults to `maintain-resolution` with balanced/fluid options.
- Production access is `screener-v2`: a valid private fragment grant enters directly; every code-only Viewer must first hold site access, after which public-watch accepts the code and private rooms still require their room password. `SITE_ACCESS_PASSWORD`, `/api/site-access`, and the site-access cookie are deployed atomically; raw credentials are never stored.
- Web Hosts can set their local/session display name, and the current release advertises it through the existing presence wire. The name is not an account identity.
-- Native v2 remains source-only (memory rooms, 300s reclaim); its Host now follows authenticated peer-assisted direct-child assignments with a two-edge cap and bounded SFU/selected-ingress failure, but production does not run it.
+- Native v2 is source-only (memory rooms, 300s reclaim); it follows authenticated direct-child assignments up to two and fails unsupported SFU/selected ingress boundedly.
- PR #44's STUN-only/SFU-UDP router and token-free fallback prewarm are process-enabled for all normal rooms: roots <=2, browser relay <=1, bounded failure. Room `1` is historical smoke; mobile/iPad viewers are leaves, healthy edges sticky, and HTTPS/WSS stays TLS/TCP.
- After an answer, a generation-bound 15s initial-connect deadline enters ICE restart; success/replacement/disposal cancels it. It is deployed but not mobile-verified.
- Room `1` can publish exactly `HIGH+LOW` with Dynacast/backup codec off and subscriber `HIGH` ceilings; it has no retained real media frame.
@@ -30,10 +30,11 @@ Flagship; parallel; min run/smoke/rollback; benchmark later. Active UI/config/lo
- Chrome 151 synthetic `1/3/5/8` and 720p30 quality-change runs kept 2/1 fanout and decoding; slowest first frame was 1.05 seconds and one relay close recovered in 5.32 seconds. This is control evidence only.
- Host A+B and authenticated P2P Viewer C are sanitized, generation-bound, read-only, and fail closed for stale, ambiguous, or SFU-fed evidence; no raw media metadata is retained.
- The `a11a73d` built-in TURN canary is rejected: local allocation passed, but direct Host/Pion Viewer failed before forced relay. Full rollback restored STUN-only client ICE and zero allocations.
-- Chrome 151 loopbacks decoded VP8 299 and H.264 opt-in 299/298 rendered at 1280x720 without fatal/encoder errors. Hardware attribution, Pion timing, multi-viewer/endurance/public/native proof remain open; production stays VP8.
-- Native Win11 audio is source-only/default-off. Target isolation was 4018x; one Viewer got 495 Opus packets. Package build passes; download, game sync, Win10, and other routes are open.
+- Chrome 151 browser loopbacks decoded VP8 299 and H.264 opt-in 299/298 rendered at 1280x720 without fatal/encoder errors. Browser hardware attribution, multi-viewer/endurance and public proof remain open; production stays VP8.
+- Native WGC/MF used no browser capture/encoder. One Chrome Viewer rendered 203 1280x720 frames and received 500 Opus packets; process/LUID-correlated `VideoEncode` was nonzero, with no fatal/encoder errors.
+- Native Win11 audio is source-only/default-off: target isolation was 4018x and one Viewer got 495 Opus packets. Download, game sync, Win10, and other routes remain open.
- Access protocol/config/HTTP/storage/SQLite/signaling focused tests pass, including commit-first teardown and v1 rollback.
-- Native peer-assisted wire/session focused tests pass: authoritative direct children reconcile at no more than two, presence cannot create a third edge, stale revisions do not rebuild, and unsupported SFU/selected ingress fails once per current revision.
+- Native wire/session tests enforce at most two authoritative children, no presence-created edge, stale-revision ignore, and one bounded unsupported-route failure per revision.
- Source adds selected Host ingress and peer last-mile for every enabled room; focused routing/config tests pass with Peer ICE STUN-only.
- Host names, the window-audio hint, all-room routing, and all-room Host selected ingress are active; the stale restart assertion is historical only.
- The `fd76277` gate preserved the environment and SQLite hashes, v3/five rooms/room `1`, selected-edge/SFU/ordinary-ICE settings, services, and local/public health plus route/asset 200s. Site access returned the unauthenticated status and the retired endpoint returned 404; no TURN/SFU session ran. Exact `5e4a3679076a9ea2fe7a41fadf4be65a439db450` is the rollback release.
@@ -55,7 +56,7 @@ Flagship; parallel; min run/smoke/rollback; benchmark later. Active UI/config/lo
Repeat room `1` and fix only the identified SFU connect/source/publish/audio/transport layer; retain selected UDP, a rendered frame, edge caps and resource deltas.
Recovery uses one ICE restart, one same-parent rebuild, one alternate peer, then SFU. Next gate exactly-two/Dynacast-off BWE on a zero-child SFU leaf, then root evacuation.
-Package one-Viewer VP8/H.264/Opus first; WGC/MF hardware then Viewer2/FIFO are functional slices. Real-game A/V and resource benchmarks follow functional landing.
+Native WGC/MF passed one-Viewer hardware; package download, Viewer2/FIFO and game A/V remain. Benchmark later.
Ordinary Peer ICE stays STUN-only. Selected-edge Host ingress still needs one forced-relay canary before a media-success claim.
ADR-0004 still needs the 30-minute `1/3/5/8` matrix; a separate 20-viewer gate must pass before changing the default eight.
@@ -66,6 +67,5 @@ host/provider UDP 7882 and bounded services are deployed, but real external
media/device evidence is absent. Shared IP remains smoke-only and cannot approve
clean-port migration; the full gate still needs an isolated VM/IP.
-- Whole-system versus selected-game audio for the first release.
- Initial deployment region and network cohort.
- Project license and distribution model.
diff --git "a/docs/\346\226\271\346\241\210\350\256\276\350\256\241.md" "b/docs/\346\226\271\346\241\210\350\256\276\350\256\241.md"
index e8696cb6..40fa2933 100644
--- "a/docs/\346\226\271\346\241\210\350\256\276\350\256\241.md"
+++ "b/docs/\346\226\271\346\241\210\350\256\276\350\256\241.md"
@@ -160,7 +160,9 @@ PoC 不强制 codec,也不做 SDP munging。先通过 stats 记录实际协商
`audio: true` 只是请求。若 `getAudioTracks()` 为空,分享者界面必须显示“当前来源没有可共享音频”。`systemAudio`、`windowAudio` 和 `audioSelection` 只影响选源器偏好,浏览器可以忽略且没有对应的标准音频来源类型读回,因此不能显示为系统、窗口或所选游戏音频保证。纯 Web 首版接受浏览器实际提供的标签页或系统音频,不承诺仅捕获某个游戏进程;系统音频可能把语音软件中观众自己的声音再次采入,这属于源内容泄漏而非 Host 页面回放,AEC 或 Web Audio 无法从已混合 track 恢复进程身份。Web 端必须保留 video-only 选择并明确风险,不能声称已隔离语音。
-当前 Windows Native sender 以所选游戏 PID 与进程创建时间为本地边界,使用 WASAPI process loopback 的 include-target-process-tree 捕获游戏及其子进程。目标进程无渲染流、退出或捕获失败时显示无声/未知,不回退为整机音频,也不上传 PID、路径、窗口标题、设备标识或原始音频。该 source-only 路径当前只支持 Windows 11;Windows 10 game-only 音频保持明确不支持,打包与其他媒体 route 仍待完成。
+当前 Windows Native sender 用本地 opaque ID 保存所选窗口的 HWND、PID 与进程创建时间;helper 在使用前重新验证三者绑定,并以 WASAPI process loopback 的 include-target-process-tree 捕获游戏及其子进程。目标进程无渲染流、退出或捕获失败时显示无声/未知,不回退为整机音频,也不上传 PID、路径、窗口标题、设备标识或原始音频。该路径只支持 Windows 11;Windows 10 game-only 音频保持明确不支持。
+
+同一 helper 的显式 `native-window-h264` source 用 `CreateForWindow` 建立 WGC item,在所选 adapter 的 free-threaded D3D11 frame pool 接收 BGRA surface;D3D11 VideoProcessor 做固定 1280x720/30 aspect-fit NV12 转换,随后进入 adapter-LUID-bound、hardware-only asynchronous Media Foundation H.264 MFT。输出必须是 Annex-B `42c01f`,首次、周期和反馈恢复帧携带 SPS、PPS 与 IDR。PCM、H.264 access unit 和 local-only hardware status 通过一个有界 helper protocol 进入 generation-bound Go session;H.264 直接写现有 shared fanout,UI 只收一份单向预览副本,PCM 保持现有本地 WebCodecs Opus bridge。任一 Native capture/encode 失败都终止该 source,不静默改用软件 MFT、其他 codec、browser capture、monitor capture 或 system mix;browser VP8 保持默认,browser H.264 保持独立 opt-in。一名普通 Chrome Viewer 已 decoded/rendered 203 帧并收到 500 Opus packets;同一 helper PID 与 adapter LUID 的 `VideoEncode` 观测匹配。
直播中切换来源仍由明确的按钮手势重新调用 `getDisplayMedia()`,不改变房间、host generation 或媒体拓扑。选源取消或失败时继续发送旧流;成功后为每个现有 sender 调用 `replaceTrack()`,健康连接不重新协商。初始 offer 预留 send-only audio transceiver,使音频可以在 `null -> track -> null` 间切换。若某个 peer 的替换或回滚失败,只重建该 peer;新流提交后才停止旧流,并以当前 stream 身份保护旧 track 的迟到 `ended` 事件。统计码率累加器在换源后重置,避免把不同 track 的计数器混算。
diff --git "a/docs/\351\234\200\346\261\202\347\220\206\350\247\243.md" "b/docs/\351\234\200\346\261\202\347\220\206\350\247\243.md"
index 6605ad53..18cae0dd 100644
--- "a/docs/\351\234\200\346\261\202\347\220\206\350\247\243.md"
+++ "b/docs/\351\234\200\346\261\202\347\220\206\350\247\243.md"
@@ -53,7 +53,7 @@
- [ ] 分包/分层多树实验:把编码包、对象或可独立解码层拆到多个 peer 路径以分散上传,只能在独立 spike 中验证。必须保持亚秒级,处理多父重组、排序、丢包恢复、关键帧、同步和节点退出;未通过门槛前不进入产品路由控制器。
- [ ] SFU-root 中央兜底:产品部署目标必须提供经 canary 验收的单节点 SFU 资源;系统在 direct/peer UDP 不可用或明确拓扑预算无法接纳时自动把它作为主要中央兜底,不要求用户选模式。它只向总计一至两个必要 roots 出口;root 优先承担受限 peer descendants,没有可靠 relay root 时必要 viewer 只作为同一总上限内的零 descendants root。已关闭 PR #12 的显式整房 SFU 模式已被取代。room `1` 的历史 smoke 没有 retained SFU media;公网媒体、资源和恢复门禁仍适用于全站路由。
- [ ] 分享端桌面封装:用 Electron 固定 Chromium 版本、改善选源/热键/系统音频体验,同时保留 Web 观看端。
-- [ ] Windows 应用音频:Windows 11 原生候选默认仅捕获所选游戏及其子进程的音频,避免把语音聊天或通知声一并分享;“全部系统声音”只能由分享者显式选择。普通 Windows 10 不具备当前 process-loopback API 门槛,必须明确标为未解决/不支持而不是伪兼容。浏览器无法证明或隔离任意进程音频时必须如实提示,原生捕获失败、无声或进程退出不得静默扩大为整机捕获。
+- [ ] Windows 应用音频:Windows 11 Native sender 使用本地 opaque window target 绑定 HWND、PID 与进程创建时间,只捕获所选游戏及其子进程的音频,避免把语音聊天或通知声一并分享;“全部系统声音”只能由分享者显式选择。普通 Windows 10 不具备当前 process-loopback API 门槛,必须明确标为未解决/不支持而不是伪兼容。浏览器无法证明或隔离任意进程音频时必须如实提示,原生捕获失败、无声或进程退出不得静默扩大为整机捕获。
- [ ] 连接自检:在开始分享前验证信令、STUN 和 SFU ICE/UDP 可达性,并给出不暴露地址或凭据的诊断结果。启用 selected-edge TURN canary 时只报告脱敏的授权阶段与 selected candidate type,不输出 URL、username、credential、地址或原始 candidate;媒体 TCP 不在当前配置面。
- [ ] 受限网络增强:短期鉴权 TURN 只能由完整、默认关闭的部署 tuple 与当前 room controller 启用,并且只能在 direct/peer UDP 与 SFU/UDP 均已按有界梯级失败后,向 controller 选中的当前异常 edge 一次性授权。普通 peer PC 从创建起保持 STUN-only;不得向整个 participant、房间或所有 peer connections 预发 TURN candidates。凭据必须绑定当前授权 session 与 route/connection generation、短期有效、内存态、一次性消费且不进入 URL、日志、浏览器持久化或 SQLite。媒体 TCP 仍需独立 canary;两者都不能被描述为更高清的传输方式。
- [ ] 运行时编解码选择:先保留浏览器默认协商;只有端点能力相交、Media Capabilities 对目标分辨率/FPS/码率报告 supported/smooth/power-efficient,且实际 outbound codec、区间 encode time、游戏 FPS、CPU/GPU 与 sender 数量共同通过时,才调整 H.264、VP8、VP9 或 AV1 优先级。AV1 不得因压缩率推断而默认启用,任何候选都不得静默落入高负载软件编码。
@@ -61,7 +61,7 @@
### P2 - 可选功能
-- [ ] 原生 Windows 捕获辅助程序,使用 WGC/DDA、WASAPI 和硬件编码器降低复制与编码开销。
+- [x] 原生 Windows 捕获辅助程序:显式 `native-window-h264` 模式使用同一 bound window target 的 WGC/D3D11 画面与 WASAPI process-tree PCM,并用 adapter-bound、hardware-only Media Foundation H.264 `42c01f` 直接进入现有 Go fanout;浏览器 VP8/H.264 是显式替代选项,VP8 默认,不允许运行时静默改用软件编码、显示器画面或系统混音。一名普通 Chrome Viewer 已 decoded/rendered 203 帧并收到 500 Opus packets,同一 helper PID/adapter 的 `VideoEncode` 与 LUID 匹配。
- [ ] iOS 原生分享端有界实验:当前不实现 ReplayKit/broadcast-extension 兼容栈。待 iOS 27 与对应 ScreenCaptureKit SDK 脱离 beta 并稳定后,只评估 ScreenCaptureKit system picker + fixed-`HIGH` + 一至两名 Web Viewer 的窄路径;VideoToolbox 必须要求并读回硬件编码,无法证明时 fail closed,不增加 ReplayKit fallback、`LOW`、AV1 或新拓扑。
- [ ] Viewer 向电视输出:系统级屏幕镜像现在可直接使用;产品只复用平台 AirPlay/Cast,不自造投屏协议。首个 Web 候选只在运行时检测 `HTMLMediaElement.remote`,并在真实 AirPlay/Cast 目标上证明当前 live WebRTC `srcObject` 可播放后才显示入口;不支持时保留浏览器原生媒体控件/系统镜像。投屏是 Viewer 的本地播放输出,不能改变 room、TURN/SFU/peer route、Host fanout 或服务端媒体分发。Google Cast receiver 需要可由接收器获取的媒体 URL,当前 P2P `MediaStream` 不满足该输入,因此不加入 Cast SDK/receiver。
- [ ] 多地区 SFU-root 调度,以及可选 TURN 兼容层的按地区、运营商可达率。
@@ -158,6 +158,7 @@
- Given 浏览器返回屏幕音轨,When 产品展示音频状态或未来连接详情,Then 只能把存在性、浏览器实际返回的 track settings、实际协商 codec/fmtp 和实际流量分别标为捕获、协商或观测事实;字段缺失必须显示为未知,且不得从 Opus 的 `48000/2` RTP 声明推断源采样率或实际 stereo。
- Given Web 或后续 native sender 分享电影/视频音频,When 进行产品验收,Then 必须用独立 A/B/C 证据关联 capture audio track/settings、实际 negotiated codec/fmtp、outbound/inbound 实际码率、loss、jitter、concealment、jitter-buffer 与 A/V sync;native sender 必须通过该音质和同步矩阵,且不得把 audio `maxBitrate` ceiling 当作提质证明。
- Given 分享者同时运行语音软件,When 选择仅分享游戏声音,Then supported Windows 11 native sender 只保留目标游戏及其子进程的渲染音频,观众、通知和其他应用声音不得进入分享;浏览器 system audio 无法满足时必须允许 video-only 或明确告知其可能包含语音/通知,不能用 AEC、降噪或来源标签伪称已隔离。
+- Given 分享者显式选择 Native window H.264 与一个本地窗口,When 分享开始,Then helper 必须在再次验证 HWND/PID/creation-time 后以 WGC/D3D11 和 hardware-only MF H.264 产生一份 `42c01f` Annex-B 输出并直接复用现有 Go/Pion fanout;Viewer 仍使用普通浏览器。硬件 MFT、WGC、音频或目标身份任一失败都明确停止该模式,不得静默改用软件编码、浏览器画面、显示器画面或系统混音。
### 仓库可移植性
diff --git a/native/fixtures/mf-h264-hardware/README.md b/native/fixtures/mf-h264-hardware/README.md
index d82505dd..c138b3b4 100644
--- a/native/fixtures/mf-h264-hardware/README.md
+++ b/native/fixtures/mf-h264-hardware/README.md
@@ -1,8 +1,10 @@
# Media Foundation H.264 Hardware Fixture
-This is an offline Windows-only decision fixture. It is not linked into the
-Screener sender and does not contact a room, signaling service, Viewer, or
-production endpoint.
+This is an offline Windows-only decision fixture. Its build compiles the same
+Media Foundation encoder source used by `native/window-capture-helper`, under a
+fixture-only macro for synthetic input and telemetry. The generated fixture is
+not linked into the sender and does not contact a room, signaling service,
+Viewer, or production endpoint.
The fixture selects one DXGI adapter and one adapter-bound Media Foundation
transform. It never enumerates a software MFT and has no codec or software
@@ -36,10 +38,9 @@ The fixture prints bounded key/value evidence to stdout and never writes the
encoded bitstream. A nonzero exit is a measured no-go at the reported stage;
it must not be worked around by adding a software retry.
-The Pion/two-binding and browser decode boundary remains separate. The current
-machine has no Go toolchain in `PATH`, and the existing product bridge is still
-no-go before first Go RTP. Coupling either boundary into this encoder fixture
-would hide which layer failed.
+Pion transport and browser decode remain separate from this offline fixture;
+the product path reuses its exact `42c01f`, Annex-B, adapter-bound hardware
+contract without adding those network concerns to this executable.
Primary API references, accessed 2026-08-20:
diff --git a/native/fixtures/mf-h264-hardware/build.ps1 b/native/fixtures/mf-h264-hardware/build.ps1
index 46883b3b..9bb08641 100644
--- a/native/fixtures/mf-h264-hardware/build.ps1
+++ b/native/fixtures/mf-h264-hardware/build.ps1
@@ -28,7 +28,7 @@ if ([string]::IsNullOrWhiteSpace($installationPath)) {
}
$developerCommand = Join-Path $installationPath 'Common7\Tools\VsDevCmd.bat'
-$sourcePath = Join-Path $fixtureDirectory 'main.cpp'
+$sourcePath = Join-Path $fixtureDirectory '..\..\window-capture-helper\main.cpp'
$executablePath = Join-Path $outputPath 'screener-mf-h264-fixture.exe'
$objectPath = Join-Path $outputPath 'main.obj'
@@ -36,7 +36,7 @@ New-Item -ItemType Directory -Path $outputPath -Force | Out-Null
$compile = @(
'call "{0}" -arch=x64 -host_arch=x64 >nul' -f $developerCommand
- 'cl.exe /nologo /std:c++20 /EHsc /W4 /WX /DUNICODE /D_UNICODE /DWIN32_LEAN_AND_MEAN /D_WIN32_WINNT=0x0A00 "{0}" /Fo:"{1}" /Fe:"{2}" /link mfplat.lib mf.lib mfuuid.lib d3d11.lib dxgi.lib dxguid.lib evr.lib ole32.lib oleaut32.lib pdh.lib' -f $sourcePath, $objectPath, $executablePath
+ 'cl.exe /nologo /std:c++20 /EHsc /W4 /WX /DSCREENER_H264_FIXTURE /DUNICODE /D_UNICODE /DWIN32_LEAN_AND_MEAN /D_WIN32_WINNT=0x0A00 "{0}" /Fo:"{1}" /Fe:"{2}" /link mfplat.lib mf.lib mfuuid.lib d3d11.lib dxgi.lib dxguid.lib evr.lib ole32.lib oleaut32.lib pdh.lib windowsapp.lib' -f $sourcePath, $objectPath, $executablePath
) -join ' && '
& cmd.exe /d /s /c $compile
diff --git a/native/process-audio-helper/README.md b/native/process-audio-helper/README.md
deleted file mode 100644
index 844c7936..00000000
--- a/native/process-audio-helper/README.md
+++ /dev/null
@@ -1,22 +0,0 @@
-# Windows Process-Audio Helper
-
-This Windows 11 helper is the local input for Native sender process-tree audio.
-It enumerates visible top-level windows locally and captures only the explicitly
-selected process plus its children through WASAPI application loopback. It has
-no whole-system fallback and no network code.
-
-The standard evaluation ZIP builds this helper beside the sender through
-`scripts/package-native-sender.ps1`. For a helper-only development build, use
-an output directory outside the repository (or set
-`SCREENER_PROCESS_AUDIO_HELPER` for a development run):
-
-```powershell
-$out = Join-Path ([IO.Path]::GetTempPath()) 'screener-native-audio'
-./native/process-audio-helper/build.ps1 -OutputDirectory $out
-```
-
-The implementation follows Microsoft's MIT-licensed Application Loopback
-sample contract without copying its WIL/Media Foundation fixture machinery:
-
-- <https://github.com/microsoft/Windows-classic-samples/tree/main/Samples/ApplicationLoopback>
-- <https://learn.microsoft.com/en-us/windows/win32/api/audioclientactivationparams/ns-audioclientactivationparams-audioclient_activation_params>
diff --git a/native/sender/README.md b/native/sender/README.md
index 92d8a16c..ef59d8d8 100644
--- a/native/sender/README.md
+++ b/native/sender/README.md
@@ -1,25 +1,27 @@
# Screener Native Sender for Windows x64
-This evaluation package contains the current Native sender and the Windows 11
-process-audio helper. It has no installer, updater, code signature, or bundled
+This evaluation package contains the current Native sender and its Windows 11
+window-capture helper. It has no installer, updater, code signature, or bundled
server.
## Run
-1. Keep `screener-sender.exe` and `screener-process-audio.exe` in the same
+1. Keep `screener-sender.exe` and `screener-window-capture.exe` in the same
directory.
2. Run `screener-sender.exe`. It opens a loopback-only UI in the default
browser; current Chrome or Edge is required.
-3. Enter the Screener server and site access password, choose VP8 (default)
- or experimental H.264, then choose the screen or window to share.
-4. Process audio is off by default. To enable it, select the exact target in
- the audio list and select the same target in the browser picker.
+3. Enter the Screener server and site access password. Browser VP8 remains the
+ default; browser H.264 and Native window H.264 are explicit opt-ins.
+4. For a browser video source, choose the browser screen/window and optionally
+ select the same window target for process-tree audio. Native window H.264
+ requires one target and captures its video and process-tree audio together.
5. Close the console window to stop the sender.
-Process audio requires Windows 11 build 22000 or newer. It captures only the
-selected process tree and never falls back to the system mix. H.264 requests
-browser hardware preference but does not prove which physical encoder Chrome
-or Edge selected. Viewers continue using the normal Web client.
+Native capture requires Windows 11 build 22000 or newer. It binds the selected
+window to its process identity, never falls back to the system mix, and never
+falls back from the hardware-only Media Foundation H.264 encoder to software.
+Browser H.264 still requests browser hardware preference without proving which
+physical encoder Chrome or Edge selected. Viewers use the normal Web client.
Verify the downloaded ZIP against its adjacent `.sha256` file, then verify the
files inside it with `SHA256SUMS.txt`. The package is unsigned, so Windows may
diff --git a/native/sender/internal/app/server.go b/native/sender/internal/app/server.go
index 63cde53c..f179ff0c 100644
--- a/native/sender/internal/app/server.go
+++ b/native/sender/internal/app/server.go
@@ -16,11 +16,18 @@ import (
"github.com/TNTcraftHIM/Screener/native/sender/internal/media"
"github.com/TNTcraftHIM/Screener/native/sender/internal/remote"
- "github.com/TNTcraftHIM/Screener/native/sender/internal/windowaudio"
+ "github.com/TNTcraftHIM/Screener/native/sender/internal/windowcapture"
"github.com/coder/websocket"
"github.com/coder/websocket/wsjson"
)
+type videoSource string
+
+const (
+ videoSourceBrowser videoSource = "browser"
+ videoSourceNativeWindow videoSource = "native-window-h264"
+)
+
const (
processTokenHeader = "X-Screener-Process-Token"
maxLocalJSONBytes = 8 << 10
@@ -38,43 +45,47 @@ type App struct {
token string
attachTimeout time.Duration
- mu sync.Mutex
- writeMu sync.Mutex
- server *http.Server
- listener net.Listener
- expectedHost string
- origin string
- starting bool
- generation uint64
- startCancel context.CancelFunc
- startFailure string
- session *remote.Session
- sessionCancel context.CancelFunc
- room remote.Room
- codec media.Codec
- mediaConn *websocket.Conn
- mediaClaimed bool
- mediaClaimID uint64
- nextClaimID uint64
- attachTimer *time.Timer
- configTimeout time.Duration
- activeViewers int
- waitingViewers int
- audioProvider windowaudio.Provider
- audioTargets map[string]windowaudio.Target
- audioTarget *windowaudio.Target
+ mu sync.Mutex
+ writeMu sync.Mutex
+ server *http.Server
+ listener net.Listener
+ expectedHost string
+ origin string
+ starting bool
+ generation uint64
+ startCancel context.CancelFunc
+ startFailure string
+ session *remote.Session
+ sessionCancel context.CancelFunc
+ room remote.Room
+ codec media.Codec
+ mediaConn *websocket.Conn
+ mediaClaimed bool
+ mediaClaimID uint64
+ nextClaimID uint64
+ attachTimer *time.Timer
+ configTimeout time.Duration
+ activeViewers int
+ waitingViewers int
+ captureProvider windowcapture.Provider
+ windowTargets map[string]windowcapture.Target
+ windowTarget *windowcapture.Target
+ videoSource videoSource
+ keyFrameRequests chan struct{}
}
type startRequest struct {
- ServerURL string `json:"serverUrl"`
- Password string `json:"password"`
- Codec string `json:"codec,omitempty"`
- AudioTargetID string `json:"audioTargetId,omitempty"`
+ ServerURL string `json:"serverUrl"`
+ Password string `json:"password"`
+ Codec string `json:"codec,omitempty"`
+ VideoSource string `json:"videoSource,omitempty"`
+ WindowTargetID string `json:"windowTargetId,omitempty"`
}
type encoderConfig struct {
Kind string `json:"kind"`
Codec string `json:"codec"`
+ VideoSource string `json:"videoSource"`
Width int `json:"width"`
Height int `json:"height"`
FPS int `json:"fps"`
@@ -83,7 +94,7 @@ type encoderConfig struct {
Audio bool `json:"audio"`
}
-type localAudioTarget struct {
+type localWindowTarget struct {
ID string `json:"id"`
Title string `json:"title"`
}
@@ -96,11 +107,12 @@ func New() (*App, error) {
ctx, cancel := context.WithCancel(context.Background())
return &App{
ctx: ctx, cancel: cancel, token: token,
- attachTimeout: mediaAttachTimeout,
- configTimeout: mediaConfigTimeout,
- codec: media.CodecVP8,
- audioProvider: windowaudio.NewProvider(),
- audioTargets: make(map[string]windowaudio.Target),
+ attachTimeout: mediaAttachTimeout,
+ configTimeout: mediaConfigTimeout,
+ codec: media.CodecVP8,
+ videoSource: videoSourceBrowser,
+ captureProvider: windowcapture.NewProvider(),
+ windowTargets: make(map[string]windowcapture.Target),
}, nil
}
@@ -157,8 +169,8 @@ func (app *App) ServeHTTP(response http.ResponseWriter, request *http.Request) {
app.handleStart(response, request)
case "/api/stop":
app.handleStop(response, request)
- case "/api/audio-targets":
- app.handleAudioTargets(response, request)
+ case "/api/window-targets":
+ app.handleWindowTargets(response, request)
case "/media":
app.handleMedia(response, request)
default:
@@ -197,21 +209,35 @@ func (app *App) handleStart(response http.ResponseWriter, request *http.Request)
writeJSON(response, http.StatusBadRequest, map[string]string{"error": "unsupported native codec"})
return
}
+ source, validSource := parseVideoSource(strings.TrimSpace(input.VideoSource))
+ if !validSource {
+ writeJSON(response, http.StatusBadRequest, map[string]string{"error": "unsupported video source"})
+ return
+ }
+ if source == videoSourceNativeWindow && codec != media.CodecH264 {
+ writeJSON(response, http.StatusBadRequest, map[string]string{"error": "native window capture requires H.264"})
+ return
+ }
baseURL, err := normalizeRemoteBase(input.ServerURL)
if err != nil {
writeJSON(response, http.StatusBadRequest, map[string]string{"error": err.Error()})
return
}
app.mu.Lock()
- var audioTarget *windowaudio.Target
- if input.AudioTargetID != "" {
- selected, ok := app.audioTargets[input.AudioTargetID]
+ var windowTarget *windowcapture.Target
+ if input.WindowTargetID != "" {
+ selected, ok := app.windowTargets[input.WindowTargetID]
if !ok {
app.mu.Unlock()
- writeJSON(response, http.StatusBadRequest, map[string]string{"error": "selected audio target is no longer available"})
+ writeJSON(response, http.StatusBadRequest, map[string]string{"error": "selected window target is no longer available"})
return
}
- audioTarget = &selected
+ windowTarget = &selected
+ }
+ if source == videoSourceNativeWindow && windowTarget == nil {
+ app.mu.Unlock()
+ writeJSON(response, http.StatusBadRequest, map[string]string{"error": "native window capture requires a selected window"})
+ return
}
if app.starting || app.session != nil {
app.mu.Unlock()
@@ -227,11 +253,11 @@ func (app *App) handleStart(response http.ResponseWriter, request *http.Request)
app.mu.Unlock()
session, room, err := remote.Start(startContext, remote.StartOptions{
- BaseURL: baseURL,
+ BaseURL: baseURL,
SiteAccessPassword: strings.TrimSpace(input.Password),
- Codec: codec,
- EnableAudio: audioTarget != nil,
- OnEvent: func(event remote.Event) { app.emit(generation, event) },
+ Codec: codec,
+ EnableAudio: windowTarget != nil,
+ OnEvent: func(event remote.Event) { app.emit(generation, event) },
})
app.mu.Lock()
app.starting = false
@@ -245,7 +271,8 @@ func (app *App) handleStart(response http.ResponseWriter, request *http.Request)
app.sessionCancel = startCancel
app.room = room
app.codec = codec
- app.audioTarget = audioTarget
+ app.videoSource = source
+ app.windowTarget = windowTarget
app.attachTimer = time.AfterFunc(app.attachTimeout, func() {
app.mu.Lock()
expired := app.generation == generation && app.session == session && app.mediaConn == nil
@@ -281,31 +308,31 @@ func (app *App) handleStop(response http.ResponseWriter, request *http.Request)
response.WriteHeader(http.StatusNoContent)
}
-func (app *App) handleAudioTargets(response http.ResponseWriter, request *http.Request) {
+func (app *App) handleWindowTargets(response http.ResponseWriter, request *http.Request) {
if !app.authorizeAPI(response, request, http.MethodPost) {
return
}
- targets, err := app.audioProvider.List(request.Context())
+ targets, err := app.captureProvider.List(request.Context())
if err != nil {
writeJSON(response, http.StatusOK, map[string]any{
- "supported": false, "targets": []localAudioTarget{},
- "reason": "Windows 11 process audio helper is unavailable",
+ "supported": false, "targets": []localWindowTarget{},
+ "reason": "Windows 11 window capture helper is unavailable",
})
return
}
- localTargets := make([]localAudioTarget, 0, len(targets))
- resolved := make(map[string]windowaudio.Target, len(targets))
+ localTargets := make([]localWindowTarget, 0, len(targets))
+ resolved := make(map[string]windowcapture.Target, len(targets))
for _, target := range targets {
id, tokenErr := newOpaqueToken()
if tokenErr != nil {
- writeJSON(response, http.StatusInternalServerError, map[string]string{"error": "create local audio target identity failed"})
+ writeJSON(response, http.StatusInternalServerError, map[string]string{"error": "create local window target identity failed"})
return
}
resolved[id] = target
- localTargets = append(localTargets, localAudioTarget{ID: id, Title: target.Title})
+ localTargets = append(localTargets, localWindowTarget{ID: id, Title: target.Title})
}
app.mu.Lock()
- app.audioTargets = resolved
+ app.windowTargets = resolved
app.mu.Unlock()
writeJSON(response, http.StatusOK, map[string]any{"supported": true, "targets": localTargets})
}
@@ -388,8 +415,12 @@ func (app *App) handleMedia(response http.ResponseWriter, request *http.Request)
return
}
var config encoderConfig
+ app.mu.Lock()
+ source := app.videoSource
+ target := app.windowTarget
+ app.mu.Unlock()
if err = decodeLocalPayload(payload, &config); err != nil ||
- !validEncoderConfigForCodec(config, session.Codec()) || config.Audio != session.AudioEnabled() {
+ !validEncoderConfigForSource(config, session.Codec(), source) || config.Audio != session.AudioEnabled() {
session.Fail(errors.New("local encoder configuration is invalid"))
_ = connection.Close(websocket.StatusPolicyViolation, "invalid encoder config")
return
@@ -398,38 +429,80 @@ func (app *App) handleMedia(response http.ResponseWriter, request *http.Request)
session.Fail(fmt.Errorf("acknowledge local encoder configuration: %w", err))
return
}
- app.mu.Lock()
- target := app.audioTarget
- app.mu.Unlock()
captureContext, cancelCapture := context.WithCancel(app.ctx)
- audioDone := make(chan struct{})
+ captureDone := make(chan struct{})
+ keyFrameRequests := make(chan struct{}, 1)
+ if source == videoSourceNativeWindow {
+ app.mu.Lock()
+ if app.generation == generation && app.session == session {
+ app.keyFrameRequests = keyFrameRequests
+ }
+ app.mu.Unlock()
+ }
+ writePCM := func(chunk windowcapture.PCMChunk) error {
+ envelope, encodeErr := media.EncodePacket(media.Packet{
+ Kind: media.KindPCM, Timestamp100ns: chunk.Timestamp100ns,
+ Duration100ns: chunk.Duration100ns, Data: chunk.Data,
+ })
+ if encodeErr != nil {
+ return encodeErr
+ }
+ return app.writeLocalBinary(connection, envelope)
+ }
if target != nil {
_ = app.writeLocal(connection, map[string]string{"kind": "audio-state", "state": "starting"})
- go func(selected windowaudio.Target) {
- defer close(audioDone)
- captureErr := app.audioProvider.Capture(captureContext, selected, func(chunk windowaudio.PCMChunk) error {
- envelope, encodeErr := media.EncodePacket(media.Packet{
- Kind: media.KindPCM, Timestamp100ns: chunk.Timestamp100ns,
- Duration100ns: chunk.Duration100ns, Data: chunk.Data,
- })
- if encodeErr != nil {
- return encodeErr
- }
- return app.writeLocalBinary(connection, envelope)
- })
+ go func(selected windowcapture.Target) {
+ defer close(captureDone)
+ var captureErr error
+ if source == videoSourceNativeWindow {
+ captureErr = app.captureProvider.CaptureWindow(
+ captureContext, selected, keyFrameRequests, writePCM,
+ func(accessUnit windowcapture.H264AccessUnit) error {
+ packet := media.Packet{
+ Kind: media.KindVideo, KeyFrame: accessUnit.KeyFrame,
+ Timestamp100ns: accessUnit.Timestamp100ns,
+ Duration100ns: accessUnit.Duration100ns, Data: accessUnit.Data,
+ }
+ if writeErr := session.WriteMedia(packet); writeErr != nil {
+ return writeErr
+ }
+ envelope, encodeErr := media.EncodePacket(packet)
+ if encodeErr != nil {
+ return encodeErr
+ }
+ return app.writeLocalBinary(connection, envelope)
+ },
+ func(status windowcapture.Status) error {
+ return app.writeLocal(connection, map[string]any{
+ "kind": "native-video-state", "status": status,
+ })
+ },
+ )
+ } else {
+ captureErr = app.captureProvider.CaptureAudio(captureContext, selected, writePCM)
+ }
if captureErr != nil && captureContext.Err() == nil {
- _ = app.writeLocal(connection, map[string]string{
- "kind": "audio-state", "state": "unavailable",
- "message": "Target audio stopped; system audio was not substituted",
- })
+ if source == videoSourceNativeWindow {
+ session.Fail(errors.New("native window capture stopped"))
+ } else {
+ _ = app.writeLocal(connection, map[string]string{
+ "kind": "audio-state", "state": "unavailable",
+ "message": "Target audio stopped; system audio was not substituted",
+ })
+ }
}
}(*target)
} else {
- close(audioDone)
+ close(captureDone)
}
defer func() {
+ app.mu.Lock()
+ if app.generation == generation && app.keyFrameRequests == keyFrameRequests {
+ app.keyFrameRequests = nil
+ }
+ app.mu.Unlock()
cancelCapture()
- <-audioDone
+ <-captureDone
}()
for {
messageType, payload, err = connection.Read(app.ctx)
@@ -450,7 +523,9 @@ func (app *App) handleMedia(response http.ResponseWriter, request *http.Request)
_ = connection.Close(websocket.StatusPolicyViolation, "invalid encoded media")
return
}
- if packet.Kind != media.KindVideo && packet.Kind != media.KindOpus {
+ validKind := packet.Kind == media.KindOpus ||
+ (source == videoSourceBrowser && packet.Kind == media.KindVideo)
+ if !validKind {
session.Fail(errors.New("local media bridge received an invalid encoded media kind"))
_ = connection.Close(websocket.StatusPolicyViolation, "encoded media required")
return
@@ -502,7 +577,14 @@ func (app *App) emit(generation uint64, event remote.Event) {
}
connection := app.mediaConn
session := app.session
+ keyFrameRequests := app.keyFrameRequests
app.mu.Unlock()
+ if event.Kind == "request-keyframe" && keyFrameRequests != nil {
+ select {
+ case keyFrameRequests <- struct{}{}:
+ default:
+ }
+ }
if connection != nil {
if err := app.writeLocal(connection, event); err != nil && event.Kind != "fatal" && session != nil {
go session.Fail(fmt.Errorf("write local sender event: %w", err))
@@ -568,7 +650,9 @@ func (app *App) stopActiveGeneration(expectedGeneration uint64) {
}
app.room = remote.Room{}
app.codec = media.CodecVP8
- app.audioTarget = nil
+ app.videoSource = videoSourceBrowser
+ app.windowTarget = nil
+ app.keyFrameRequests = nil
app.activeViewers = 0
app.waitingViewers = 0
app.mu.Unlock()
@@ -587,15 +671,30 @@ func (app *App) stopActiveGeneration(expectedGeneration uint64) {
}
func validEncoderConfig(config encoderConfig) bool {
- return validEncoderConfigForCodec(config, media.CodecVP8)
+ return validEncoderConfigForSource(config, media.CodecVP8, videoSourceBrowser)
}
func validEncoderConfigForCodec(config encoderConfig, expected media.Codec) bool {
+ return validEncoderConfigForSource(config, expected, videoSourceBrowser)
+}
+
+func validEncoderConfigForSource(config encoderConfig, expected media.Codec, source videoSource) bool {
return config.Kind == "config" && config.Codec == string(expected) &&
+ config.VideoSource == string(source) &&
config.Width == 1280 && config.Height == 720 && config.FPS == 30 &&
config.Bitrate == 3_000_000 && config.EncoderInstances == 1
}
+func parseVideoSource(value string) (videoSource, bool) {
+ if value == "" || value == string(videoSourceBrowser) {
+ return videoSourceBrowser, true
+ }
+ if value == string(videoSourceNativeWindow) {
+ return videoSourceNativeWindow, true
+ }
+ return videoSourceBrowser, false
+}
+
func decodeLocalJSON(request *http.Request, target any) error {
payload, err := io.ReadAll(io.LimitReader(request.Body, maxLocalJSONBytes+1))
if err != nil || len(payload) > maxLocalJSONBytes {
diff --git a/native/sender/internal/app/server_test.go b/native/sender/internal/app/server_test.go
index 830625e8..c464769f 100644
--- a/native/sender/internal/app/server_test.go
+++ b/native/sender/internal/app/server_test.go
@@ -14,13 +14,13 @@ import (
"github.com/TNTcraftHIM/Screener/native/sender/internal/media"
"github.com/TNTcraftHIM/Screener/native/sender/internal/remote"
- "github.com/TNTcraftHIM/Screener/native/sender/internal/windowaudio"
+ "github.com/TNTcraftHIM/Screener/native/sender/internal/windowcapture"
"github.com/coder/websocket"
)
func TestEncoderConfigUsesTheFixedHighContract(t *testing.T) {
config := encoderConfig{
- Kind: "config", Codec: "vp8", Width: 1280, Height: 720,
+ Kind: "config", Codec: "vp8", VideoSource: "browser", Width: 1280, Height: 720,
FPS: 30, Bitrate: 3_000_000, EncoderInstances: 1,
}
if !validEncoderConfig(config) {
@@ -34,7 +34,7 @@ func TestEncoderConfigUsesTheFixedHighContract(t *testing.T) {
func TestEncoderConfigAcceptsExplicitH264OnlyForAnH264Session(t *testing.T) {
config := encoderConfig{
- Kind: "config", Codec: "h264", Width: 1280, Height: 720,
+ Kind: "config", Codec: "h264", VideoSource: "browser", Width: 1280, Height: 720,
FPS: 30, Bitrate: 3_000_000, EncoderInstances: 1,
}
if !validEncoderConfigForCodec(config, media.CodecH264) {
@@ -45,14 +45,28 @@ func TestEncoderConfigAcceptsExplicitH264OnlyForAnH264Session(t *testing.T) {
}
}
+func TestNativeWindowConfigRequiresExactH264Source(t *testing.T) {
+ config := encoderConfig{
+ Kind: "config", Codec: "h264", VideoSource: "native-window-h264",
+ Width: 1280, Height: 720, FPS: 30, Bitrate: 3_000_000, EncoderInstances: 1,
+ }
+ if !validEncoderConfigForSource(config, media.CodecH264, videoSourceNativeWindow) {
+ t.Fatal("native window H.264 config was rejected")
+ }
+ if validEncoderConfigForSource(config, media.CodecH264, videoSourceBrowser) ||
+ validEncoderConfigForSource(config, media.CodecVP8, videoSourceNativeWindow) {
+ t.Fatal("native window config crossed a source or codec boundary")
+ }
+}
+
func TestDecodeLocalPayloadIsStrictAndSingleValued(t *testing.T) {
- valid := []byte(`{"kind":"config","codec":"vp8","width":1280,"height":720,"fps":30,"bitrate":3000000,"encoderInstances":1}`)
+ valid := []byte(`{"kind":"config","codec":"vp8","videoSource":"browser","width":1280,"height":720,"fps":30,"bitrate":3000000,"encoderInstances":1}`)
var config encoderConfig
if err := decodeLocalPayload(valid, &config); err != nil {
t.Fatal(err)
}
invalid := [][]byte{
- []byte(`{"kind":"config","codec":"vp8","width":1280,"height":720,"fps":30,"bitrate":3000000,"encoderInstances":1,"extra":true}`),
+ []byte(`{"kind":"config","codec":"vp8","videoSource":"browser","width":1280,"height":720,"fps":30,"bitrate":3000000,"encoderInstances":1,"extra":true}`),
append(append([]byte(nil), valid...), valid...),
}
for _, payload := range invalid {
@@ -62,20 +76,20 @@ func TestDecodeLocalPayloadIsStrictAndSingleValued(t *testing.T) {
}
}
-func TestAudioTargetListingKeepsProcessIdentityBehindAnOpaqueLocalID(t *testing.T) {
+func TestWindowTargetListingKeepsBoundIdentityBehindAnOpaqueLocalID(t *testing.T) {
application, err := New()
if err != nil {
t.Fatal(err)
}
- application.audioProvider = fakeAudioProvider{targets: []windowaudio.Target{{
- PID: 424_242, CreationTime: 987_654_321, Title: "Game Window",
+ application.captureProvider = fakeCaptureProvider{targets: []windowcapture.Target{{
+ WindowHandle: 123_456, PID: 424_242, CreationTime: 987_654_321, Title: "Game Window",
}}}
if _, err = application.Start(); err != nil {
t.Fatal(err)
}
defer application.Close()
- request, err := http.NewRequest(http.MethodPost, application.origin+"/api/audio-targets", nil)
+ request, err := http.NewRequest(http.MethodPost, application.origin+"/api/window-targets", nil)
if err != nil {
t.Fatal(err)
}
@@ -87,21 +101,21 @@ func TestAudioTargetListingKeepsProcessIdentityBehindAnOpaqueLocalID(t *testing.
}
defer response.Body.Close()
var result struct {
- Supported bool `json:"supported"`
- Targets []localAudioTarget `json:"targets"`
+ Supported bool `json:"supported"`
+ Targets []localWindowTarget `json:"targets"`
}
if json.NewDecoder(response.Body).Decode(&result) != nil || !result.Supported || len(result.Targets) != 1 {
- t.Fatalf("audio target response = %+v", result)
+ t.Fatalf("window target response = %+v", result)
}
if result.Targets[0].ID == "" || result.Targets[0].Title != "Game Window" ||
strings.Contains(result.Targets[0].ID, "424242") || strings.Contains(result.Targets[0].ID, "987654321") {
- t.Fatalf("audio target was not locally anonymized: %+v", result.Targets[0])
+ t.Fatalf("window target was not locally anonymized: %+v", result.Targets[0])
}
application.mu.Lock()
- resolved := application.audioTargets[result.Targets[0].ID]
+ resolved := application.windowTargets[result.Targets[0].ID]
application.mu.Unlock()
- if resolved.PID != 424_242 || resolved.CreationTime != 987_654_321 {
- t.Fatal("opaque target did not resolve in process memory")
+ if resolved.WindowHandle != 123_456 || resolved.PID != 424_242 || resolved.CreationTime != 987_654_321 {
+ t.Fatal("opaque target did not resolve to its bound identity in process memory")
}
}
@@ -472,7 +486,7 @@ func openConfiguredMedia(t *testing.T, application *App) *websocket.Conn {
}
t.Cleanup(func() { _ = connection.CloseNow() })
readLocalKind(t, connection, "ready")
- config := []byte(`{"kind":"config","codec":"vp8","width":1280,"height":720,"fps":30,"bitrate":3000000,"encoderInstances":1}`)
+ config := []byte(`{"kind":"config","codec":"vp8","videoSource":"browser","width":1280,"height":720,"fps":30,"bitrate":3000000,"encoderInstances":1}`)
writeContext, cancelWrite := context.WithTimeout(context.Background(), time.Second)
err = connection.Write(writeContext, websocket.MessageText, config)
cancelWrite()
@@ -689,12 +703,23 @@ func appRoomResponse(origin string) map[string]any {
}
}
-type fakeAudioProvider struct{ targets []windowaudio.Target }
+type fakeCaptureProvider struct{ targets []windowcapture.Target }
-func (provider fakeAudioProvider) List(context.Context) ([]windowaudio.Target, error) {
+func (provider fakeCaptureProvider) List(context.Context) ([]windowcapture.Target, error) {
return provider.targets, nil
}
-func (fakeAudioProvider) Capture(context.Context, windowaudio.Target, func(windowaudio.PCMChunk) error) error {
+func (fakeCaptureProvider) CaptureAudio(context.Context, windowcapture.Target, func(windowcapture.PCMChunk) error) error {
+ return nil
+}
+
+func (fakeCaptureProvider) CaptureWindow(
+ context.Context,
+ windowcapture.Target,
+ <-chan struct{},
+ func(windowcapture.PCMChunk) error,
+ func(windowcapture.H264AccessUnit) error,
+ func(windowcapture.Status) error,
+) error {
return nil
}
diff --git a/native/sender/internal/app/ui/app.js b/native/sender/internal/app/ui/app.js
index fa71672a..9a3206ab 100644
--- a/native/sender/internal/app/ui/app.js
+++ b/native/sender/internal/app/ui/app.js
@@ -25,11 +25,12 @@ const elements = {
serverURL: document.querySelector("#server-url"),
password: document.querySelector("#password"),
codec: document.querySelector("#codec"),
- audioTarget: document.querySelector("#audio-target"),
- refreshAudio: document.querySelector("#refresh-audio"),
+ windowTarget: document.querySelector("#window-target"),
+ refreshWindows: document.querySelector("#refresh-windows"),
start: document.querySelector("#start"),
stop: document.querySelector("#stop"),
preview: document.querySelector("#preview"),
+ nativePreview: document.querySelector("#native-preview"),
empty: document.querySelector("#empty-state"),
invite: document.querySelector("#invite"),
inviteLink: document.querySelector("#invite-link"),
@@ -48,6 +49,7 @@ const elements = {
let stream = null;
let processorReader = null;
let encoder = null;
+let videoDecoder = null;
let audioEncoder = null;
let silentAudioChunks = 0;
let mediaSocket = null;
@@ -59,10 +61,12 @@ let forceKeyFrame = true;
let frameNumber = 0;
let captureSettings = null;
let nativeDiagnostics = null;
+let nativeVideoStatus = null;
let metrics = emptyMetrics();
let startGeneration = 0;
let diagnosticsTimer = null;
let pendingStartGeneration = 0;
+let activeVideoSource = "browser";
if (!processToken) {
showError("本地发送端令牌已失效,请重新启动 sender.exe");
@@ -80,47 +84,71 @@ elements.copy.addEventListener("click", async () => {
showError("无法复制链接,请直接打开邀请链接");
}
});
-elements.refreshAudio.addEventListener("click", () => void loadAudioTargets());
-if (processToken) void loadAudioTargets();
+elements.refreshWindows.addEventListener("click", () => void loadWindowTargets());
+if (processToken) void loadWindowTargets();
async function startSharing() {
if (sharing || stopping || startInFlight) return;
showError("");
- if (!("VideoEncoder" in window) || !("MediaStreamTrackProcessor" in window)) {
+ const videoSource = selectedVideoSource();
+ const nativeWindow = videoSource === "native-window-h264";
+ if (!nativeWindow && (!("VideoEncoder" in window) || !("MediaStreamTrackProcessor" in window))) {
showError("此浏览器缺少 WebCodecs 屏幕编码支持,请使用当前版 Chrome 或 Edge");
return;
}
+ if (nativeWindow && !("VideoDecoder" in window)) {
+ showError("此浏览器缺少原生 H.264 本地预览支持,请使用当前版 Chrome 或 Edge");
+ return;
+ }
startInFlight = true;
elements.start.disabled = true;
const generation = ++startGeneration;
const codec = selectedCodec();
- const audioTargetId = elements.audioTarget.value;
- if (audioTargetId && (!("AudioEncoder" in window) || !("AudioData" in window))) {
+ metrics.codec = codec;
+ const windowTargetId = elements.windowTarget.value;
+ activeVideoSource = videoSource;
+ if (nativeWindow && !windowTargetId) {
+ showError("原生 H.264 模式需要选择一个目标窗口");
+ startInFlight = false;
+ elements.start.disabled = false;
+ return;
+ }
+ if (windowTargetId && (!("AudioEncoder" in window) || !("AudioData" in window))) {
showError("当前浏览器缺少 WebCodecs Opus 编码支持,请关闭目标进程音频或使用当前版 Chrome / Edge");
startInFlight = false;
elements.start.disabled = false;
return;
}
- setStatus("选择屏幕");
try {
- const capturedStream = await navigator.mediaDevices.getDisplayMedia({
- video: {
- width: { ideal: WIDTH, max: WIDTH },
- height: { ideal: HEIGHT, max: HEIGHT },
- frameRate: { ideal: FPS, max: FPS },
- },
- audio: false,
- });
- assertCurrentStart(generation, mediaSocket, capturedStream);
- stream = capturedStream;
- const videoTrack = stream.getVideoTracks()[0];
- if (!videoTrack) throw new Error("没有可用的视频轨道");
- await videoTrack.applyConstraints({ width: WIDTH, height: HEIGHT, frameRate: FPS });
- assertCurrentStart(generation);
- captureSettings = videoTrack.getSettings();
+ let videoTrack = null;
+ if (nativeWindow) {
+ await startNativeDecoder(generation);
+ captureSettings = { width: WIDTH, height: HEIGHT, frameRate: FPS };
+ elements.preview.hidden = true;
+ elements.nativePreview.hidden = false;
+ } else {
+ setStatus("选择屏幕");
+ const capturedStream = await navigator.mediaDevices.getDisplayMedia({
+ video: {
+ width: { ideal: WIDTH, max: WIDTH },
+ height: { ideal: HEIGHT, max: HEIGHT },
+ frameRate: { ideal: FPS, max: FPS },
+ },
+ audio: false,
+ });
+ assertCurrentStart(generation, mediaSocket, capturedStream);
+ stream = capturedStream;
+ videoTrack = stream.getVideoTracks()[0];
+ if (!videoTrack) throw new Error("没有可用的视频轨道");
+ await videoTrack.applyConstraints({ width: WIDTH, height: HEIGHT, frameRate: FPS });
+ assertCurrentStart(generation);
+ captureSettings = videoTrack.getSettings();
+ videoTrack.addEventListener("ended", () => void stopSharing(generation), { once: true });
+ elements.preview.srcObject = stream;
+ elements.preview.hidden = false;
+ elements.nativePreview.hidden = true;
+ }
updateDiagnostics();
- videoTrack.addEventListener("ended", () => void stopSharing(generation), { once: true });
- elements.preview.srcObject = stream;
elements.empty.hidden = true;
setStatus("创建房间");
@@ -129,7 +157,8 @@ async function startSharing() {
serverUrl: elements.serverURL.value,
password: elements.password.value,
codec,
- audioTargetId,
+ videoSource,
+ windowTargetId,
});
assertCurrentStart(generation);
if (pendingStartGeneration === generation) pendingStartGeneration = 0;
@@ -139,15 +168,18 @@ async function startSharing() {
elements.inviteLink.textContent = room.inviteUrl;
elements.invite.hidden = false;
- if (audioTargetId) await startAudioEncoder(generation);
- await connectMediaSocket(generation, codec, Boolean(audioTargetId));
- assertCurrentStart(generation);
- const activeEncoder = await startEncoder(generation, codec);
+ if (windowTargetId) await startAudioEncoder(generation);
+ await connectMediaSocket(generation, codec, videoSource, Boolean(windowTargetId));
assertCurrentStart(generation);
+ let activeEncoder = null;
+ if (!nativeWindow) {
+ activeEncoder = await startEncoder(generation, codec);
+ assertCurrentStart(generation);
+ }
sharing = true;
elements.stop.hidden = false;
setStatus("正在分享");
- void readFrames(videoTrack, generation, activeEncoder, mediaSocket);
+ if (!nativeWindow) void readFrames(videoTrack, generation, activeEncoder, mediaSocket);
} catch (error) {
if (generation === startGeneration && !(error instanceof StaleStartError)) {
showError(readableError(error, "无法开始分享"));
@@ -159,7 +191,7 @@ async function startSharing() {
}
}
-async function connectMediaSocket(generation, codec, audio) {
+async function connectMediaSocket(generation, codec, videoSource, audio) {
const url = new URL("/media", location.href);
url.protocol = "ws:";
const socket = new WebSocket(url, [`screener.token.${processToken}`]);
@@ -181,6 +213,7 @@ async function connectMediaSocket(generation, codec, audio) {
socket.send(JSON.stringify({
kind: "config",
codec,
+ videoSource,
width: WIDTH,
height: HEIGHT,
fps: FPS,
@@ -233,6 +266,34 @@ async function startEncoder(generation, codec) {
return activeEncoder;
}
+async function startNativeDecoder(generation) {
+ const config = {
+ codec: "avc1.42c01f",
+ codedWidth: WIDTH,
+ codedHeight: HEIGHT,
+ optimizeForLatency: true,
+ };
+ const support = await VideoDecoder.isConfigSupported(config);
+ assertCurrentStart(generation);
+ if (!support.supported) throw new Error("此浏览器不支持 H.264 42c01f 本地预览");
+ let activeDecoder;
+ activeDecoder = new VideoDecoder({
+ output: (frame) => {
+ try {
+ if (generation === startGeneration && videoDecoder === activeDecoder) {
+ const context = elements.nativePreview.getContext("2d");
+ context?.drawImage(frame, 0, 0, WIDTH, HEIGHT);
+ }
+ } finally {
+ frame.close();
+ }
+ },
+ error: (error) => void failSharing(generation, `原生 H.264 本地预览失败:${error.message}`),
+ });
+ videoDecoder = activeDecoder;
+ activeDecoder.configure(support.config);
+}
+
async function readFrames(videoTrack, generation, activeEncoder, activeSocket) {
const processor = new MediaStreamTrackProcessor({ track: videoTrack });
const reader = processor.readable.getReader();
@@ -319,7 +380,14 @@ function sendEncodedChunk(generation, activeEncoder, chunk) {
function handleLocalEvent(generation, socket, event) {
if (generation !== startGeneration || socket !== mediaSocket) return;
if (event.data instanceof ArrayBuffer) {
- handlePCM(generation, event.data);
+ const kind = event.data.byteLength > 1 ? new DataView(event.data).getUint8(1) : 0;
+ if (kind === MEDIA_KIND_PCM) {
+ handlePCM(generation, event.data);
+ } else if (kind === MEDIA_KIND_VIDEO && activeVideoSource === "native-window-h264") {
+ handleNativeVideo(generation, event.data);
+ } else {
+ void failSharing(generation, "本地媒体桥接返回了意外的二进制数据");
+ }
return;
}
if (typeof event.data !== "string") return;
@@ -339,11 +407,40 @@ function handleLocalEvent(generation, socket, event) {
? "不可用(未回退系统声音)"
: "正在连接目标进程";
}
+ if (message.kind === "native-video-state") {
+ nativeVideoStatus = { ...nativeVideoStatus, ...message.status };
+ updateDiagnostics();
+ setStatus(message.status?.state === "active" ? "正在分享(原生 H.264)" : "正在启动原生硬件采集");
+ }
if (message.kind === "fatal") {
void failSharing(generation, message.message || "分享会话已失败并停止");
}
}
+function handleNativeVideo(generation, message) {
+ if (generation !== startGeneration || !videoDecoder || videoDecoder.state !== "configured") return;
+ try {
+ const view = new DataView(message);
+ const flags = view.getUint8(2);
+ if (message.byteLength <= ENVELOPE_HEADER_BYTES || view.getUint8(0) !== 1 ||
+ view.getUint8(1) !== MEDIA_KIND_VIDEO || flags > 1 || view.getUint8(3) !== 0) {
+ throw new Error("本地 H.264 envelope 无效");
+ }
+ const data = new Uint8Array(message, ENVELOPE_HEADER_BYTES);
+ if (data.byteLength < 1 || data.byteLength > MAX_FRAME_BYTES) {
+ throw new Error("本地 H.264 access unit 长度无效");
+ }
+ videoDecoder.decode(new EncodedVideoChunk({
+ type: flags === 1 ? "key" : "delta",
+ timestamp: Number(view.getBigUint64(4, false) / 10n),
+ duration: Number(view.getBigUint64(12, false) / 10n),
+ data,
+ }));
+ } catch (error) {
+ void failSharing(generation, readableError(error, "处理本地 H.264 预览失败"));
+ }
+}
+
async function failSharing(generation, message) {
if (generation !== startGeneration || stopping) return;
showError(message);
@@ -381,6 +478,15 @@ async function stopSharing(expectedGeneration = startGeneration) {
rememberError(error);
}
encoder = null;
+ try {
+ if (videoDecoder && videoDecoder.state !== "closed") videoDecoder.close();
+ } catch (error) {
+ rememberError(error);
+ }
+ videoDecoder = null;
+ elements.nativePreview.getContext("2d")?.clearRect(0, 0, WIDTH, HEIGHT);
+ elements.nativePreview.hidden = true;
+ elements.preview.hidden = false;
try {
if (audioEncoder && audioEncoder.state !== "closed") audioEncoder.close();
} catch (error) {
@@ -408,6 +514,8 @@ async function stopSharing(expectedGeneration = startGeneration) {
silentAudioChunks = 0;
captureSettings = null;
nativeDiagnostics = null;
+ nativeVideoStatus = null;
+ activeVideoSource = "browser";
metrics = emptyMetrics();
if (diagnosticsTimer !== null) {
clearTimeout(diagnosticsTimer);
@@ -488,25 +596,25 @@ async function localRequest(path, body) {
return result;
}
-async function loadAudioTargets() {
- elements.refreshAudio.disabled = true;
- const previous = elements.audioTarget.value;
+async function loadWindowTargets() {
+ elements.refreshWindows.disabled = true;
+ const previous = elements.windowTarget.value;
try {
- const result = await localRequest("/api/audio-targets");
- elements.audioTarget.replaceChildren(new Option("关闭", ""));
+ const result = await localRequest("/api/window-targets");
+ elements.windowTarget.replaceChildren(new Option("关闭窗口音频", ""));
for (const target of Array.isArray(result.targets) ? result.targets : []) {
- if (target?.id && target?.title) elements.audioTarget.add(new Option(target.title, target.id));
+ if (target?.id && target?.title) elements.windowTarget.add(new Option(target.title, target.id));
}
- if ([...elements.audioTarget.options].some((option) => option.value === previous)) {
- elements.audioTarget.value = previous;
+ if ([...elements.windowTarget.options].some((option) => option.value === previous)) {
+ elements.windowTarget.value = previous;
}
- elements.audioTarget.disabled = !result.supported;
+ elements.windowTarget.disabled = !result.supported;
elements.audioStatus.textContent = result.supported ? "关闭" : "Windows 11 helper 不可用";
} catch (error) {
- elements.audioTarget.disabled = true;
+ elements.windowTarget.disabled = true;
elements.audioStatus.textContent = readableError(error, "窗口列表不可用");
} finally {
- elements.refreshAudio.disabled = false;
+ elements.refreshWindows.disabled = false;
}
}
@@ -595,7 +703,16 @@ function updateDiagnostics() {
const height = captureSettings?.height ?? "未知";
const rate = captureSettings?.frameRate ?? "未知";
elements.captureDiagnostic.textContent = `${width} × ${height} · ${rate} FPS`;
- elements.encoderDiagnostic.textContent = `${codecLabel(metrics.codec)} · 3 Mbps 上限 · 单对象 · 硬件偏好 ${hardwarePreferenceLabel(metrics.hardwarePreference)} · 硬件证据 ${hardwareEvidenceLabel(metrics.hardwareEvidence)}`;
+ if (activeVideoSource === "native-window-h264") {
+ const mftState = nativeVideoStatus?.state === "active"
+ ? "hardware-only MFT 已绑定并输出"
+ : nativeVideoStatus?.state === "starting"
+ ? "hardware-only MFT 已绑定,等待首帧"
+ : "hardware-only MFT 等待状态";
+ elements.encoderDiagnostic.textContent = `H.264 · 3 Mbps 上限 · 单 MF 编码器 · ${mftState}`;
+ } else {
+ elements.encoderDiagnostic.textContent = `${codecLabel(metrics.codec)} · 3 Mbps 上限 · 单对象 · 硬件偏好 ${hardwarePreferenceLabel(metrics.hardwarePreference)} · 硬件证据 ${hardwareEvidenceLabel(metrics.hardwareEvidence)}`;
+ }
elements.bridgeDiagnostic.textContent = `读取 ${metrics.framesRead} · 提交 ${metrics.framesSubmitted} · 输出 ${metrics.encoderOutputs} · 编码队列丢弃 ${metrics.encoderQueueDrops} · 编码队列峰值 ${metrics.encoderQueuePeak} 帧 · socket 峰值 ${metrics.socketBufferedPeak} B`;
const helper = nativeDiagnostics?.media;
const queue = helper?.queue;
@@ -645,7 +762,11 @@ function boundedCount(value) {
}
function selectedCodec() {
- return elements.codec?.value === "h264" ? "h264" : "vp8";
+ return elements.codec?.value === "h264" || elements.codec?.value === "native-h264" ? "h264" : "vp8";
+}
+
+function selectedVideoSource() {
+ return elements.codec?.value === "native-h264" ? "native-window-h264" : "browser";
}
function emptyMetrics() {
diff --git a/native/sender/internal/app/ui/index.html b/native/sender/internal/app/ui/index.html
index 2a75ffaa..51919583 100644
--- a/native/sender/internal/app/ui/index.html
+++ b/native/sender/internal/app/ui/index.html
@@ -22,6 +22,7 @@
</div>
<div class="preview-shell">
<video id="preview" autoplay muted playsinline></video>
+ <canvas id="native-preview" width="1280" height="720" hidden></canvas>
<div id="empty-state" class="empty-state">选择屏幕后,邀请链接会显示在这里</div>
</div>
<div id="invite" class="invite" hidden>
@@ -43,16 +44,17 @@
<select id="codec">
<option value="vp8">VP8 (default)</option>
<option value="h264">H.264 (hardware preference)</option>
+ <option value="native-h264">H.264 (native window hardware)</option>
</select>
</label>
<label>
- <span>目标进程音频</span>
+ <span>目标窗口</span>
<span class="select-row">
- <select id="audio-target"><option value="">关闭</option></select>
- <button id="refresh-audio" class="icon-button" type="button" title="刷新窗口列表" aria-label="刷新窗口列表">↻</button>
+ <select id="window-target"><option value="">关闭窗口音频</option></select>
+ <button id="refresh-windows" class="icon-button" type="button" title="刷新窗口列表" aria-label="刷新窗口列表">↻</button>
</span>
</label>
- <p class="audio-note">请在浏览器画面选择器中选择同一目标。音频只包含所选进程树,失败时不会改用系统混音。</p>
+ <p class="audio-note">浏览器画面模式下请在系统选择器中选择同一窗口。原生模式直接采集这里选择的窗口;音频只包含其进程树,失败时不会改用系统混音。</p>
<button id="start" class="button primary" type="button">开始分享</button>
<p id="error" class="error" role="alert"></p>
<details>
diff --git a/native/sender/internal/app/ui/styles.css b/native/sender/internal/app/ui/styles.css
index 7d6da525..769d9480 100644
--- a/native/sender/internal/app/ui/styles.css
+++ b/native/sender/internal/app/ui/styles.css
@@ -37,7 +37,7 @@ main { display: grid; grid-template-columns: minmax(0, 1fr) 340px; width: min(14
h1 { margin: 0; font-size: 24px; }
.stage-heading p { margin: 7px 0 0; color: var(--muted); font-size: 13px; }
.preview-shell { position: relative; display: grid; width: 100%; margin-top: 18px; overflow: hidden; aspect-ratio: 16 / 9; place-items: center; border: 1px solid var(--border); border-radius: 6px; background: #080a08; }
-video { display: block; width: 100%; height: 100%; object-fit: contain; }
+video, canvas { display: block; width: 100%; height: 100%; object-fit: contain; }
.empty-state { position: absolute; color: var(--muted); font-size: 14px; }
.invite { display: grid; grid-template-columns: minmax(0, 1fr) 42px; gap: 8px; margin-top: 14px; }
.invite a { min-width: 0; overflow: hidden; padding: 11px 12px; border: 1px solid var(--border); border-radius: 6px; color: #c8f7d9; text-overflow: ellipsis; white-space: nowrap; }
diff --git a/native/sender/internal/windowaudio/provider.go b/native/sender/internal/windowaudio/provider.go
deleted file mode 100644
index 0704e735..00000000
--- a/native/sender/internal/windowaudio/provider.go
+++ /dev/null
@@ -1,25 +0,0 @@
-package windowaudio
-
-import (
- "context"
- "errors"
-)
-
-var ErrUnavailable = errors.New("Windows 11 process audio is unavailable")
-
-type Target struct {
- PID uint32
- CreationTime uint64
- Title string
-}
-
-type PCMChunk struct {
- Timestamp100ns uint64
- Duration100ns uint64
- Data []byte
-}
-
-type Provider interface {
- List(context.Context) ([]Target, error)
- Capture(context.Context, Target, func(PCMChunk) error) error
-}
diff --git a/native/sender/internal/windowaudio/provider_other.go b/native/sender/internal/windowaudio/provider_other.go
deleted file mode 100644
index 7fb74cf6..00000000
--- a/native/sender/internal/windowaudio/provider_other.go
+++ /dev/null
@@ -1,15 +0,0 @@
-//go:build !windows
-
-package windowaudio
-
-import "context"
-
-type unsupportedProvider struct{}
-
-func NewProvider() Provider { return unsupportedProvider{} }
-
-func (unsupportedProvider) List(context.Context) ([]Target, error) { return nil, ErrUnavailable }
-
-func (unsupportedProvider) Capture(context.Context, Target, func(PCMChunk) error) error {
- return ErrUnavailable
-}
diff --git a/native/sender/internal/windowaudio/provider_windows.go b/native/sender/internal/windowaudio/provider_windows.go
deleted file mode 100644
index 3fa6433f..00000000
--- a/native/sender/internal/windowaudio/provider_windows.go
+++ /dev/null
@@ -1,145 +0,0 @@
-//go:build windows
-
-package windowaudio
-
-import (
- "context"
- "encoding/binary"
- "encoding/json"
- "errors"
- "io"
- "os"
- "os/exec"
- "path/filepath"
- "strconv"
- "strings"
-
- "golang.org/x/sys/windows"
-)
-
-const (
- maxListBytes = 64 << 10
- maxTargets = 100
- helperHeaderSize = 28
-)
-
-type windowsProvider struct{ helper string }
-
-type helperTarget struct {
- PID uint32 `json:"pid"`
- CreationTime uint64 `json:"creationTime"`
- Title string `json:"title"`
-}
-
-func NewProvider() Provider {
- helper := strings.TrimSpace(os.Getenv("SCREENER_PROCESS_AUDIO_HELPER"))
- if helper == "" {
- if executable, err := os.Executable(); err == nil {
- helper = filepath.Join(filepath.Dir(executable), "screener-process-audio.exe")
- }
- }
- return &windowsProvider{helper: helper}
-}
-
-func (provider *windowsProvider) available() bool {
- version := windows.RtlGetVersion()
- if version.MajorVersion < 10 || version.BuildNumber < 22_000 || provider.helper == "" {
- return false
- }
- info, err := os.Stat(provider.helper)
- return err == nil && !info.IsDir()
-}
-
-func (provider *windowsProvider) List(ctx context.Context) ([]Target, error) {
- if !provider.available() {
- return nil, ErrUnavailable
- }
- command := exec.CommandContext(ctx, provider.helper, "--list")
- stdout, err := command.StdoutPipe()
- if err != nil {
- return nil, ErrUnavailable
- }
- command.Stderr = io.Discard
- if err = command.Start(); err != nil {
- return nil, ErrUnavailable
- }
- payload, readErr := io.ReadAll(io.LimitReader(stdout, maxListBytes+1))
- if (readErr != nil || len(payload) > maxListBytes) && ctx.Err() == nil {
- _ = command.Process.Kill()
- }
- waitErr := command.Wait()
- if readErr != nil || waitErr != nil || len(payload) > maxListBytes {
- return nil, ErrUnavailable
- }
- var listed []helperTarget
- if json.Unmarshal(payload, &listed) != nil || len(listed) > maxTargets {
- return nil, ErrUnavailable
- }
- targets := make([]Target, 0, len(listed))
- for _, candidate := range listed {
- title := strings.TrimSpace(candidate.Title)
- if candidate.PID == 0 || candidate.CreationTime == 0 || title == "" || len(title) > 512 {
- continue
- }
- targets = append(targets, Target{
- PID: candidate.PID, CreationTime: candidate.CreationTime, Title: title,
- })
- }
- return targets, nil
-}
-
-func (provider *windowsProvider) Capture(ctx context.Context, target Target, onChunk func(PCMChunk) error) error {
- if !provider.available() || target.PID == 0 || target.CreationTime == 0 || onChunk == nil {
- return ErrUnavailable
- }
- command := exec.CommandContext(ctx, provider.helper, "--capture",
- strconv.FormatUint(uint64(target.PID), 10), strconv.FormatUint(target.CreationTime, 10))
- stdout, err := command.StdoutPipe()
- if err != nil {
- return ErrUnavailable
- }
- command.Stderr = io.Discard
- if err = command.Start(); err != nil {
- return ErrUnavailable
- }
- started := false
- for {
- header := make([]byte, helperHeaderSize)
- _, readErr := io.ReadFull(stdout, header)
- if readErr != nil {
- err = readErr
- break
- }
- length := binary.BigEndian.Uint32(header[24:28])
- if string(header[:4]) != "SPCM" || header[4] != 1 || header[5] != 0 ||
- header[6] != 0 || header[7] != 0 || length == 0 || length > 48_000*2*2/10 || length%4 != 0 {
- err = errors.New("process audio helper returned an invalid PCM header")
- break
- }
- data := make([]byte, length)
- if _, readErr = io.ReadFull(stdout, data); readErr != nil {
- err = readErr
- break
- }
- started = true
- if err = onChunk(PCMChunk{
- Timestamp100ns: binary.BigEndian.Uint64(header[8:16]),
- Duration100ns: binary.BigEndian.Uint64(header[16:24]),
- Data: data,
- }); err != nil {
- _ = command.Process.Kill()
- break
- }
- }
- if err != nil && ctx.Err() == nil {
- _ = command.Process.Kill()
- }
- _ = command.Wait()
- if ctx.Err() != nil {
- return nil
- }
- if err != nil || !started {
- return errors.New("target-process audio capture stopped")
- }
- return errors.New("target process exited")
-}
diff --git a/native/sender/internal/windowaudio/provider_windows_test.go b/native/sender/internal/windowaudio/provider_windows_test.go
deleted file mode 100644
index 7c7a6360..00000000
--- a/native/sender/internal/windowaudio/provider_windows_test.go
+++ /dev/null
@@ -1,100 +0,0 @@
-//go:build windows
-
-package windowaudio
-
-import (
- "context"
- "encoding/binary"
- "flag"
- "os"
- "testing"
- "time"
-)
-
-const testCreationTime = uint64(123456)
-
-var (
- captureHelperPID = flag.String("capture", "", "run as the process-audio capture test helper")
- listHelper = flag.Bool("list", false, "run as the process-audio list test helper")
-)
-
-func TestMain(main *testing.M) {
- flag.Parse()
- if *listHelper {
- _, _ = os.Stdout.Write(make([]byte, maxListBytes+1))
- select {}
- }
- if *captureHelperPID != "" {
- if len(flag.Args()) != 1 || flag.Args()[0] != "123456" {
- os.Exit(3)
- }
- header := make([]byte, helperHeaderSize)
- copy(header, "SPCM")
- header[4] = 1
- binary.BigEndian.PutUint64(header[16:24], 200_000)
- binary.BigEndian.PutUint32(header[24:28], 4)
- _, _ = os.Stdout.Write(header)
- _, _ = os.Stdout.Write(make([]byte, 4))
- _, _ = os.Stdout.Write(make([]byte, helperHeaderSize))
- select {}
- }
- os.Exit(main.Run())
-}
-
-func TestCapturePassesIdentityAndKillsHelperBeforeWaitingOnInvalidHeader(t *testing.T) {
- provider := &windowsProvider{helper: os.Args[0]}
- if !provider.available() {
- t.Skip("requires Windows 11")
- }
-
- ctx, cancel := context.WithCancel(context.Background())
- result := make(chan error, 1)
- chunks := 0
- go func() {
- result <- provider.Capture(ctx, Target{PID: 1, CreationTime: testCreationTime}, func(PCMChunk) error {
- chunks++
- return nil
- })
- }()
-
- select {
- case err := <-result:
- if err == nil {
- t.Fatal("Capture returned nil after an invalid helper header")
- }
- if chunks != 1 {
- t.Fatalf("Capture delivered %d chunks; process identity was not passed to the helper", chunks)
- }
- case <-time.After(2 * time.Second):
- cancel()
- <-result
- t.Fatal("Capture waited on a running helper after an invalid header")
- }
- cancel()
-}
-
-func TestListKillsHelperBeforeWaitingOnOversizeOutput(t *testing.T) {
- provider := &windowsProvider{helper: os.Args[0]}
- if !provider.available() {
- t.Skip("requires Windows 11")
- }
-
- ctx, cancel := context.WithCancel(context.Background())
- result := make(chan error, 1)
- go func() {
- _, err := provider.List(ctx)
- result <- err
- }()
-
- select {
- case err := <-result:
- if err == nil {
- t.Fatal("List accepted oversized helper output")
- }
- case <-time.After(2 * time.Second):
- cancel()
- <-result
- t.Fatal("List waited on a running helper after oversized output")
- }
- cancel()
-}
diff --git a/native/sender/internal/windowcapture/provider.go b/native/sender/internal/windowcapture/provider.go
new file mode 100644
index 00000000..5c8a0cbf
--- /dev/null
+++ b/native/sender/internal/windowcapture/provider.go
@@ -0,0 +1,56 @@
+package windowcapture
+
+import (
+ "context"
+ "errors"
+)
+
+var ErrUnavailable = errors.New("Windows 11 window capture is unavailable")
+
+type Target struct {
+ WindowHandle uint64
+ PID uint32
+ CreationTime uint64
+ Title string
+}
+
+type PCMChunk struct {
+ Timestamp100ns uint64
+ Duration100ns uint64
+ Data []byte
+}
+
+type H264AccessUnit struct {
+ KeyFrame bool
+ Timestamp100ns uint64
+ Duration100ns uint64
+ Data []byte
+}
+
+type Status struct {
+ State string `json:"state"`
+ HardwareOnly bool `json:"hardwareOnly"`
+ AdapterIndex *uint `json:"adapterIndex,omitempty"`
+ AdapterName string `json:"adapterName,omitempty"`
+ AdapterLUID string `json:"adapterLuid,omitempty"`
+ MFTIndex *uint `json:"mftIndex,omitempty"`
+ MFTName string `json:"mftName,omitempty"`
+ MFTCLSID string `json:"mftClsid,omitempty"`
+ ProfileLevelID string `json:"profileLevelId,omitempty"`
+ Width int `json:"width,omitempty"`
+ Height int `json:"height,omitempty"`
+ FPS int `json:"fps,omitempty"`
+}
+
+type Provider interface {
+ List(context.Context) ([]Target, error)
+ CaptureAudio(context.Context, Target, func(PCMChunk) error) error
+ CaptureWindow(
+ context.Context,
+ Target,
+ <-chan struct{},
+ func(PCMChunk) error,
+ func(H264AccessUnit) error,
+ func(Status) error,
+ ) error
+}
diff --git a/native/sender/internal/windowcapture/provider_other.go b/native/sender/internal/windowcapture/provider_other.go
new file mode 100644
index 00000000..7fad201d
--- /dev/null
+++ b/native/sender/internal/windowcapture/provider_other.go
@@ -0,0 +1,26 @@
+//go:build !windows
+
+package windowcapture
+
+import "context"
+
+type unsupportedProvider struct{}
+
+func NewProvider() Provider { return unsupportedProvider{} }
+
+func (unsupportedProvider) List(context.Context) ([]Target, error) { return nil, ErrUnavailable }
+
+func (unsupportedProvider) CaptureAudio(context.Context, Target, func(PCMChunk) error) error {
+ return ErrUnavailable
+}
+
+func (unsupportedProvider) CaptureWindow(
+ context.Context,
+ Target,
+ <-chan struct{},
+ func(PCMChunk) error,
+ func(H264AccessUnit) error,
+ func(Status) error,
+) error {
+ return ErrUnavailable
+}
diff --git a/native/sender/internal/windowcapture/provider_windows.go b/native/sender/internal/windowcapture/provider_windows.go
new file mode 100644
index 00000000..dc0b3527
--- /dev/null
+++ b/native/sender/internal/windowcapture/provider_windows.go
@@ -0,0 +1,307 @@
+//go:build windows
+
+package windowcapture
+
+import (
+ "bytes"
+ "context"
+ "encoding/binary"
+ "encoding/json"
+ "errors"
+ "io"
+ "os"
+ "os/exec"
+ "path/filepath"
+ "strconv"
+ "strings"
+
+ "golang.org/x/sys/windows"
+)
+
+const (
+ maxListBytes = 64 << 10
+ maxTargets = 100
+ helperHeaderSize = 28
+ maxStatusBytes = 4 << 10
+ maxMediaBytes = 1 << 20
+ maxPCMBytes = 48_000 * 2 * 2 / 10
+ helperKindPCM = 1
+ helperKindH264 = 2
+ helperKindStatus = 3
+ helperProtocolMagic = "SMED"
+)
+
+type windowsProvider struct {
+ helper string
+ adapterIndex uint
+ mftIndex uint
+}
+
+type helperTarget struct {
+ WindowHandle uint64 `json:"windowHandle"`
+ PID uint32 `json:"pid"`
+ CreationTime uint64 `json:"creationTime"`
+ Title string `json:"title"`
+}
+
+func NewProvider() Provider {
+ helper := strings.TrimSpace(os.Getenv("SCREENER_WINDOW_CAPTURE_HELPER"))
+ if helper == "" {
+ if executable, err := os.Executable(); err == nil {
+ helper = filepath.Join(filepath.Dir(executable), "screener-window-capture.exe")
+ }
+ }
+ return &windowsProvider{
+ helper: helper,
+ adapterIndex: parseOptionalIndex("SCREENER_WINDOW_CAPTURE_ADAPTER_INDEX"),
+ mftIndex: parseOptionalIndex("SCREENER_WINDOW_CAPTURE_MFT_INDEX"),
+ }
+}
+
+func parseOptionalIndex(name string) uint {
+ value, err := strconv.ParseUint(strings.TrimSpace(os.Getenv(name)), 10, 32)
+ if err != nil {
+ return 0
+ }
+ return uint(value)
+}
+
+func (provider *windowsProvider) available() bool {
+ version := windows.RtlGetVersion()
+ if version.MajorVersion < 10 || version.BuildNumber < 22_000 || provider.helper == "" {
+ return false
+ }
+ info, err := os.Stat(provider.helper)
+ return err == nil && !info.IsDir()
+}
+
+func (provider *windowsProvider) List(ctx context.Context) ([]Target, error) {
+ if !provider.available() {
+ return nil, ErrUnavailable
+ }
+ command := exec.CommandContext(ctx, provider.helper, "--list")
+ stdout, err := command.StdoutPipe()
+ if err != nil {
+ return nil, ErrUnavailable
+ }
+ command.Stderr = io.Discard
+ if err = command.Start(); err != nil {
+ return nil, ErrUnavailable
+ }
+ payload, readErr := io.ReadAll(io.LimitReader(stdout, maxListBytes+1))
+ if (readErr != nil || len(payload) > maxListBytes) && ctx.Err() == nil {
+ _ = command.Process.Kill()
+ }
+ waitErr := command.Wait()
+ if readErr != nil || waitErr != nil || len(payload) > maxListBytes {
+ return nil, ErrUnavailable
+ }
+ var listed []helperTarget
+ decoder := json.NewDecoder(bytes.NewReader(payload))
+ decoder.DisallowUnknownFields()
+ if decoder.Decode(&listed) != nil || decoder.Decode(&struct{}{}) != io.EOF || len(listed) > maxTargets {
+ return nil, ErrUnavailable
+ }
+ targets := make([]Target, 0, len(listed))
+ for _, candidate := range listed {
+ title := strings.TrimSpace(candidate.Title)
+ if candidate.WindowHandle == 0 || candidate.PID == 0 || candidate.CreationTime == 0 ||
+ title == "" || len(title) > 512 {
+ continue
+ }
+ targets = append(targets, Target{
+ WindowHandle: candidate.WindowHandle,
+ PID: candidate.PID,
+ CreationTime: candidate.CreationTime,
+ Title: title,
+ })
+ }
+ return targets, nil
+}
+
+func (provider *windowsProvider) CaptureAudio(
+ ctx context.Context,
+ target Target,
+ onPCM func(PCMChunk) error,
+) error {
+ return provider.capture(ctx, target,
+ []string{"--capture-audio", targetPID(target), targetCreationTime(target)},
+ nil, onPCM, nil, nil)
+}
+
+func (provider *windowsProvider) CaptureWindow(
+ ctx context.Context,
+ target Target,
+ keyFrames <-chan struct{},
+ onPCM func(PCMChunk) error,
+ onH264 func(H264AccessUnit) error,
+ onStatus func(Status) error,
+) error {
+ if keyFrames == nil || onPCM == nil || onH264 == nil || onStatus == nil {
+ return ErrUnavailable
+ }
+ arguments := []string{
+ "--capture-window", targetPID(target), targetCreationTime(target),
+ strconv.FormatUint(target.WindowHandle, 10),
+ "--adapter-index", strconv.FormatUint(uint64(provider.adapterIndex), 10),
+ "--mft-index", strconv.FormatUint(uint64(provider.mftIndex), 10),
+ "--protocol-v1",
+ }
+ return provider.capture(ctx, target, arguments, keyFrames, onPCM, onH264, onStatus)
+}
+
+func targetPID(target Target) string { return strconv.FormatUint(uint64(target.PID), 10) }
+
+func targetCreationTime(target Target) string { return strconv.FormatUint(target.CreationTime, 10) }
+
+func (provider *windowsProvider) capture(
+ ctx context.Context,
+ target Target,
+ arguments []string,
+ keyFrames <-chan struct{},
+ onPCM func(PCMChunk) error,
+ onH264 func(H264AccessUnit) error,
+ onStatus func(Status) error,
+) error {
+ windowMode := onH264 != nil && onStatus != nil
+ if !provider.available() || target.PID == 0 || target.CreationTime == 0 || onPCM == nil ||
+ (windowMode && target.WindowHandle == 0) {
+ return ErrUnavailable
+ }
+ command := exec.CommandContext(ctx, provider.helper, arguments...)
+ stdout, err := command.StdoutPipe()
+ if err != nil {
+ return ErrUnavailable
+ }
+ var stdin io.WriteCloser
+ if windowMode {
+ stdin, err = command.StdinPipe()
+ if err != nil {
+ return ErrUnavailable
+ }
+ }
+ command.Stderr = io.Discard
+ if err = command.Start(); err != nil {
+ return ErrUnavailable
+ }
+ keyFrameDone := make(chan struct{})
+ keyFrameStop := make(chan struct{})
+ if windowMode {
+ go func() {
+ defer close(keyFrameDone)
+ defer stdin.Close()
+ for {
+ select {
+ case <-ctx.Done():
+ return
+ case <-keyFrameStop:
+ return
+ case _, ok := <-keyFrames:
+ if !ok {
+ return
+ }
+ if _, writeErr := stdin.Write([]byte{'K'}); writeErr != nil {
+ return
+ }
+ }
+ }
+ }()
+ } else {
+ close(keyFrameDone)
+ }
+
+ startedPCM := false
+ startedH264 := false
+ for {
+ header := make([]byte, helperHeaderSize)
+ if _, err = io.ReadFull(stdout, header); err != nil {
+ break
+ }
+ kind := header[5]
+ flags := header[6]
+ timestamp := binary.BigEndian.Uint64(header[8:16])
+ duration := binary.BigEndian.Uint64(header[16:24])
+ length := binary.BigEndian.Uint32(header[24:28])
+ if string(header[:4]) != helperProtocolMagic || header[4] != 1 || header[7] != 0 ||
+ length == 0 || !validHelperHeader(kind, flags, timestamp, duration, length, windowMode) {
+ err = errors.New("window capture helper returned an invalid media header")
+ break
+ }
+ data := make([]byte, length)
+ if _, err = io.ReadFull(stdout, data); err != nil {
+ break
+ }
+ switch kind {
+ case helperKindPCM:
+ startedPCM = true
+ err = onPCM(PCMChunk{Timestamp100ns: timestamp, Duration100ns: duration, Data: data})
+ case helperKindH264:
+ startedH264 = true
+ err = onH264(H264AccessUnit{
+ KeyFrame: flags == 1, Timestamp100ns: timestamp, Duration100ns: duration, Data: data,
+ })
+ case helperKindStatus:
+ var status Status
+ err = decodeStatus(data, &status)
+ if err == nil {
+ err = onStatus(status)
+ }
+ }
+ if err != nil {
+ break
+ }
+ }
+ if err != nil && ctx.Err() == nil {
+ _ = command.Process.Kill()
+ }
+ _ = command.Wait()
+ if windowMode {
+ close(keyFrameStop)
+ _ = stdin.Close()
+ }
+ <-keyFrameDone
+ if ctx.Err() != nil {
+ return nil
+ }
+ if err != nil || (windowMode && !startedH264) || (!windowMode && !startedPCM) {
+ return errors.New("selected window capture stopped")
+ }
+ return errors.New("selected target process exited")
+}
+
+func validHelperHeader(kind, flags byte, timestamp, duration uint64, length uint32, windowMode bool) bool {
+ switch kind {
+ case helperKindPCM:
+ return flags == 0 && duration > 0 && duration <= 10_000_000 &&
+ length <= maxPCMBytes && length%4 == 0
+ case helperKindH264:
+ return windowMode && flags <= 1 && timestamp > 0 && duration >= 10 &&
+ duration <= 10_000_000 && duration%10 == 0 && length <= maxMediaBytes
+ case helperKindStatus:
+ return windowMode && flags == 0 && timestamp == 0 && duration == 0 && length <= maxStatusBytes
+ default:
+ return false
+ }
+}
+
+func decodeStatus(payload []byte, status *Status) error {
+ decoder := json.NewDecoder(bytes.NewReader(payload))
+ decoder.DisallowUnknownFields()
+ if decoder.Decode(status) != nil || decoder.Decode(&struct{}{}) != io.EOF || !status.HardwareOnly {
+ return errors.New("window capture helper returned invalid status")
+ }
+ switch status.State {
+ case "starting":
+ if status.AdapterIndex == nil || status.MFTIndex == nil || status.AdapterName == "" ||
+ status.AdapterLUID == "" || status.MFTName == "" || status.MFTCLSID == "" {
+ return errors.New("window capture helper returned incomplete hardware status")
+ }
+ case "active":
+ if status.ProfileLevelID != "42c01f" || status.Width != 1280 || status.Height != 720 || status.FPS != 30 {
+ return errors.New("window capture helper returned unexpected video status")
+ }
+ default:
+ return errors.New("window capture helper returned unknown status")
+ }
+ return nil
+}
diff --git a/native/sender/internal/windowcapture/provider_windows_test.go b/native/sender/internal/windowcapture/provider_windows_test.go
new file mode 100644
index 00000000..e54c21c6
--- /dev/null
+++ b/native/sender/internal/windowcapture/provider_windows_test.go
@@ -0,0 +1,149 @@
+//go:build windows
+
+package windowcapture
+
+import (
+ "context"
+ "encoding/binary"
+ "io"
+ "os"
+ "reflect"
+ "testing"
+ "time"
+)
+
+const testCreationTime = uint64(123456)
+const testWindowHandle = uint64(987654)
+
+func TestMain(main *testing.M) {
+ if len(os.Args) > 1 {
+ switch os.Args[1] {
+ case "--list":
+ _, _ = os.Stdout.Write(make([]byte, maxListBytes+1))
+ select {}
+ case "--capture-audio":
+ if !reflect.DeepEqual(os.Args[2:], []string{"1", "123456"}) {
+ os.Exit(3)
+ }
+ writeTestEnvelope(helperKindPCM, 0, 1, 200_000, make([]byte, 4))
+ _, _ = os.Stdout.Write(make([]byte, helperHeaderSize))
+ select {}
+ case "--capture-window":
+ expected := []string{
+ "1", "123456", "987654", "--adapter-index", "2",
+ "--mft-index", "3", "--protocol-v1",
+ }
+ if !reflect.DeepEqual(os.Args[2:], expected) {
+ os.Exit(4)
+ }
+ status := []byte(`{"state":"starting","hardwareOnly":true,"adapterIndex":2,"adapterName":"GPU","adapterLuid":"0:1","mftIndex":3,"mftName":"MFT","mftClsid":"clsid"}`)
+ writeTestEnvelope(helperKindStatus, 0, 0, 0, status)
+ writeTestEnvelope(helperKindH264, 1, 100, 333_330, []byte{0, 0, 0, 1, 0x65})
+ request := make([]byte, 1)
+ if _, err := io.ReadFull(os.Stdin, request); err != nil || request[0] != 'K' {
+ os.Exit(5)
+ }
+ _, _ = os.Stdout.Write(make([]byte, helperHeaderSize))
+ select {}
+ }
+ }
+ os.Exit(main.Run())
+}
+
+func writeTestEnvelope(kind, flags byte, timestamp, duration uint64, payload []byte) {
+ header := make([]byte, helperHeaderSize)
+ copy(header, helperProtocolMagic)
+ header[4] = 1
+ header[5] = kind
+ header[6] = flags
+ binary.BigEndian.PutUint64(header[8:16], timestamp)
+ binary.BigEndian.PutUint64(header[16:24], duration)
+ binary.BigEndian.PutUint32(header[24:28], uint32(len(payload)))
+ _, _ = os.Stdout.Write(header)
+ _, _ = os.Stdout.Write(payload)
+}
+
+func TestCaptureAudioPassesIdentityAndKillsHelperBeforeWaitOnInvalidHeader(t *testing.T) {
+ provider := &windowsProvider{helper: os.Args[0]}
+ if !provider.available() {
+ t.Skip("requires Windows 11")
+ }
+
+ result := make(chan error, 1)
+ chunks := 0
+ go func() {
+ result <- provider.CaptureAudio(context.Background(), Target{
+ WindowHandle: testWindowHandle, PID: 1, CreationTime: testCreationTime,
+ }, func(PCMChunk) error {
+ chunks++
+ return nil
+ })
+ }()
+
+ select {
+ case err := <-result:
+ if err == nil || chunks != 1 {
+ t.Fatalf("CaptureAudio error = %v, chunks = %d", err, chunks)
+ }
+ case <-time.After(2 * time.Second):
+ t.Fatal("CaptureAudio waited on a running helper after an invalid header")
+ }
+}
+
+func TestCaptureWindowPassesBoundIdentityAndForwardsKeyFrameRequest(t *testing.T) {
+ provider := &windowsProvider{helper: os.Args[0], adapterIndex: 2, mftIndex: 3}
+ if !provider.available() {
+ t.Skip("requires Windows 11")
+ }
+
+ keyFrames := make(chan struct{}, 1)
+ result := make(chan error, 1)
+ statuses, accessUnits := 0, 0
+ go func() {
+ result <- provider.CaptureWindow(context.Background(), Target{
+ WindowHandle: testWindowHandle, PID: 1, CreationTime: testCreationTime,
+ }, keyFrames, func(PCMChunk) error {
+ return nil
+ }, func(unit H264AccessUnit) error {
+ accessUnits++
+ if !unit.KeyFrame {
+ t.Error("first access unit was not a key frame")
+ }
+ keyFrames <- struct{}{}
+ return nil
+ }, func(Status) error {
+ statuses++
+ return nil
+ })
+ }()
+
+ select {
+ case err := <-result:
+ if err == nil || statuses != 1 || accessUnits != 1 {
+ t.Fatalf("CaptureWindow error = %v, statuses = %d, access units = %d", err, statuses, accessUnits)
+ }
+ case <-time.After(2 * time.Second):
+ t.Fatal("CaptureWindow did not forward the key-frame request or kill the invalid helper")
+ }
+}
+
+func TestListKillsHelperBeforeWaitingOnOversizeOutput(t *testing.T) {
+ provider := &windowsProvider{helper: os.Args[0]}
+ if !provider.available() {
+ t.Skip("requires Windows 11")
+ }
+
+ result := make(chan error, 1)
+ go func() {
+ _, err := provider.List(context.Background())
+ result <- err
+ }()
+ select {
+ case err := <-result:
+ if err == nil {
+ t.Fatal("List accepted oversized helper output")
+ }
+ case <-time.After(2 * time.Second):
+ t.Fatal("List waited on a running helper after oversized output")
+ }
+}
diff --git a/native/window-capture-helper/README.md b/native/window-capture-helper/README.md
new file mode 100644
index 00000000..6b1c5ea4
--- /dev/null
+++ b/native/window-capture-helper/README.md
@@ -0,0 +1,30 @@
+# Windows Window-Capture Helper
+
+This Windows 11 helper is the local input for Native sender window capture. It
+enumerates visible top-level windows locally, binds the selected HWND to its PID
+and process creation time, and has two explicit modes:
+
+- process-tree PCM for the existing browser video sources; or
+- WGC/D3D11 window video plus process-tree PCM, with adapter-bound,
+ hardware-only Media Foundation H.264 output.
+
+It has no whole-system audio, software encoder, alternate codec, or network
+fallback. Build it outside the repository, then place it beside the sender (or
+set `SCREENER_WINDOW_CAPTURE_HELPER` for a development run):
+
+```powershell
+$out = Join-Path ([IO.Path]::GetTempPath()) 'screener-native-capture'
+./native/window-capture-helper/build.ps1 -OutputDirectory $out
+```
+
+The implementation follows Microsoft's MIT-licensed reference samples and
+official API contracts without copying their WIL framework. The retained MF
+fixture compiles the same encoder source with `SCREENER_H264_FIXTURE`; there is
+not a second product MFT implementation.
+
+- <https://github.com/microsoft/Windows-classic-samples/tree/main/Samples/ApplicationLoopback>
+- <https://github.com/microsoft/Windows.UI.Composition-Win32-Samples/tree/master/cpp/ScreenCaptureforHWND>
+- <https://learn.microsoft.com/en-us/windows/win32/api/audioclientactivationparams/ns-audioclientactivationparams-audioclient_activation_params>
+- <https://learn.microsoft.com/en-us/windows/uwp/audio-video-camera/screen-capture>
+- <https://learn.microsoft.com/en-us/windows/win32/api/windows.graphics.capture.interop/nf-windows-graphics-capture-interop-igraphicscaptureiteminterop-createforwindow>
+- <https://learn.microsoft.com/en-us/uwp/api/windows.graphics.capture.direct3d11captureframepool.createfreethreaded>
diff --git a/native/process-audio-helper/build.ps1 b/native/window-capture-helper/build.ps1
similarity index 57%
rename from native/process-audio-helper/build.ps1
rename to native/window-capture-helper/build.ps1
index 59e14cac..bb3f978c 100644
--- a/native/process-audio-helper/build.ps1
+++ b/native/window-capture-helper/build.ps1
@@ -23,18 +23,22 @@ if ([string]::IsNullOrWhiteSpace($installationPath)) {
}
$developerCommand = Join-Path $installationPath 'Common7\Tools\VsDevCmd.bat'
-$sourcePath = Join-Path $helperDirectory 'main.cpp'
-$executablePath = Join-Path $outputPath 'screener-process-audio.exe'
-$objectPath = Join-Path $outputPath 'main.obj'
+$mainSourcePath = Join-Path $helperDirectory 'main.cpp'
+$audioSourcePath = Join-Path $helperDirectory 'process_audio.cpp'
+$executablePath = Join-Path $outputPath 'screener-window-capture.exe'
+$mainObjectPath = Join-Path $outputPath 'window-capture.obj'
+$audioObjectPath = Join-Path $outputPath 'process-audio.obj'
New-Item -ItemType Directory -Path $outputPath -Force | Out-Null
$compile = @(
'call "{0}" -arch=x64 -host_arch=x64 >nul' -f $developerCommand
- 'cl.exe /nologo /std:c++20 /EHsc /W4 /WX /DUNICODE /D_UNICODE /DWIN32_LEAN_AND_MEAN /D_WIN32_WINNT=0x0A00 /DNTDDI_VERSION=0x0A00000A "{0}" /Fo:"{1}" /Fe:"{2}" /link ole32.lib mmdevapi.lib runtimeobject.lib user32.lib' -f $sourcePath, $objectPath, $executablePath
+ 'cl.exe /nologo /c /std:c++20 /EHsc /W4 /WX /DUNICODE /D_UNICODE /DWIN32_LEAN_AND_MEAN /D_WIN32_WINNT=0x0A00 /DNTDDI_VERSION=0x0A00000A "{0}" /Fo:"{1}"' -f $mainSourcePath, $mainObjectPath
+ 'cl.exe /nologo /c /std:c++20 /EHsc /W4 /WX /DUNICODE /D_UNICODE /DWIN32_LEAN_AND_MEAN /D_WIN32_WINNT=0x0A00 /DNTDDI_VERSION=0x0A00000A "{0}" /Fo:"{1}"' -f $audioSourcePath, $audioObjectPath
+ 'link.exe /nologo /out:"{0}" "{1}" "{2}" ole32.lib mmdevapi.lib runtimeobject.lib user32.lib mfplat.lib mf.lib mfuuid.lib d3d11.lib dxgi.lib dxguid.lib evr.lib oleaut32.lib windowsapp.lib' -f $executablePath, $mainObjectPath, $audioObjectPath
) -join ' && '
& cmd.exe /d /s /c $compile
if ($LASTEXITCODE -ne 0) {
- throw "Process-audio helper compilation failed with exit code $LASTEXITCODE."
+ throw "Window-capture helper compilation failed with exit code $LASTEXITCODE."
}
Write-Output $executablePath
diff --git a/native/fixtures/mf-h264-hardware/main.cpp b/native/window-capture-helper/main.cpp
similarity index 60%
rename from native/fixtures/mf-h264-hardware/main.cpp
rename to native/window-capture-helper/main.cpp
index 53e5821b..e10768ed 100644
--- a/native/fixtures/mf-h264-hardware/main.cpp
+++ b/native/window-capture-helper/main.cpp
@@ -6,22 +6,38 @@
#include <d3d11.h>
#include <dxgi1_2.h>
#include <evr.h>
+#include <fcntl.h>
+#include <io.h>
#include <mfapi.h>
#include <mferror.h>
#include <mfidl.h>
#include <mftransform.h>
+#ifdef SCREENER_H264_FIXTURE
#include <pdh.h>
#include <pdhmsg.h>
+#endif
#include <propvarutil.h>
+#include <windows.graphics.capture.interop.h>
+#include <windows.graphics.directx.direct3d11.interop.h>
#include <wrl/client.h>
+#include <winrt/Windows.Foundation.h>
+#include <winrt/Windows.Graphics.Capture.h>
+#include <winrt/Windows.Graphics.DirectX.Direct3D11.h>
+#include <winrt/Windows.Graphics.DirectX.h>
+#include <winrt/base.h>
+
+#include "process_audio.h"
+
#include <algorithm>
+#include <array>
#include <atomic>
#include <chrono>
#include <cmath>
#include <cstdint>
#include <cwctype>
#include <exception>
+#include <functional>
#include <iomanip>
#include <iostream>
#include <limits>
@@ -44,13 +60,16 @@ constexpr UINT32 kHeight = 720;
constexpr UINT32 kFrameRate = 30;
constexpr UINT32 kBitRate = 3'000'000;
constexpr UINT32 kVbvBytes = kBitRate / kFrameRate / 8;
-constexpr UINT32 kFrameCount = 360;
constexpr UINT32 kGopFrames = 60;
-constexpr UINT32 kMaxInFlight = 8;
constexpr UINT32 kMaxEventsPerPump = 64;
constexpr DWORD kMaxEncodedSampleBytes = 4 * 1024 * 1024;
+#ifdef SCREENER_H264_FIXTURE
+constexpr UINT32 kFrameCount = 360;
+constexpr UINT32 kMaxInFlight = 8;
constexpr DWORD kMaxPdhArrayBytes = 4 * 1024 * 1024;
constexpr DWORD kMaxPdhArrayItems = 16 * 1024;
+#endif
+
constexpr LONGLONG kFrameDuration100ns = 10'000'000 / kFrameRate;
class GateFailure final : public std::runtime_error {
@@ -552,6 +571,7 @@ void ValidateCodecReadback(ICodecAPI* codec) {
}
}
+#ifdef SCREENER_H264_FIXTURE
std::vector<UINT8> SyntheticNv12(UINT32 frame_index) {
std::vector<UINT8> pixels(kWidth * kHeight * 3 / 2);
const UINT32 bar_start = (frame_index * 11) % kWidth;
@@ -607,6 +627,7 @@ ComPtr<IMFSample> CreateInputSample(ID3D11Device* device, UINT32 frame_index) {
"input-sample-duration");
return sample;
}
+#endif
struct NalSummary final {
bool annex_b = false;
@@ -694,6 +715,7 @@ std::vector<UINT8> ReadSample(IMFSample* sample) {
return bytes;
}
+#ifdef SCREENER_H264_FIXTURE
ULONGLONG ProcessCpu100ns() {
FILETIME created = {};
FILETIME exited = {};
@@ -851,6 +873,7 @@ struct RunEvidence final {
ULONGLONG cpu_100ns = 0;
double wall_seconds = 0.0;
};
+#endif
void ForceKeyFrame(ICodecAPI* codec) {
VARIANT value;
@@ -928,6 +951,7 @@ ComPtr<IMFSample> PullOutput(IMFTransform* transform,
return produced;
}
+#ifdef SCREENER_H264_FIXTURE
bool IsRecoveryFrame(UINT32 frame) {
return frame % kGopFrames == 0;
}
@@ -1164,6 +1188,7 @@ RunEvidence RunEncoder(const Adapter& adapter, const DeviceContext& device,
}
return evidence;
}
+#endif
UINT ParseIndex(const wchar_t* value, const std::string& stage) {
try {
@@ -1178,6 +1203,719 @@ UINT ParseIndex(const wchar_t* value, const std::string& stage) {
}
}
+#ifndef SCREENER_H264_FIXTURE
+constexpr DWORD kMaxProductAccessUnitBytes = 1 * 1024 * 1024;
+constexpr DWORD kMaxStatusBytes = 4 * 1024;
+constexpr UINT64 kEnvelopeFrameDuration100ns =
+ (static_cast<UINT64>(kFrameDuration100ns) / 10) * 10;
+
+enum class OutputKind : UINT8 {
+ pcm = 1,
+ h264 = 2,
+ status = 3,
+};
+
+class UniqueHandle final {
+ public:
+ UniqueHandle() = default;
+ explicit UniqueHandle(HANDLE value) : value_(value) {}
+ UniqueHandle(const UniqueHandle&) = delete;
+ UniqueHandle& operator=(const UniqueHandle&) = delete;
+ UniqueHandle(UniqueHandle&& other) noexcept
+ : value_(std::exchange(other.value_, nullptr)) {}
+ ~UniqueHandle() {
+ if (value_ != nullptr && value_ != INVALID_HANDLE_VALUE) CloseHandle(value_);
+ }
+ HANDLE get() const { return value_; }
+
+ private:
+ HANDLE value_ = nullptr;
+};
+
+void PutUint32BE(BYTE* output, UINT32 value) {
+ for (int index = 3; index >= 0; --index) {
+ output[index] = static_cast<BYTE>(value & 0xff);
+ value >>= 8;
+ }
+}
+
+void PutUint64BE(BYTE* output, UINT64 value) {
+ for (int index = 7; index >= 0; --index) {
+ output[index] = static_cast<BYTE>(value & 0xff);
+ value >>= 8;
+ }
+}
+
+class ProtocolWriter final {
+ public:
+ HRESULT Write(OutputKind kind, UINT8 flags, UINT64 timestamp100ns,
+ UINT64 duration100ns, const BYTE* data, DWORD size) {
+ DWORD maximum = kind == OutputKind::status ? kMaxStatusBytes
+ : kMaxProductAccessUnitBytes;
+ if (data == nullptr || size == 0 || size > maximum ||
+ (kind != OutputKind::status && duration100ns == 0)) {
+ return HRESULT_FROM_WIN32(ERROR_INVALID_DATA);
+ }
+ std::array<BYTE, 28> header{};
+ header[0] = 'S';
+ header[1] = 'M';
+ header[2] = 'E';
+ header[3] = 'D';
+ header[4] = 1;
+ header[5] = static_cast<BYTE>(kind);
+ header[6] = flags;
+ PutUint64BE(header.data() + 8, timestamp100ns);
+ PutUint64BE(header.data() + 16, duration100ns);
+ PutUint32BE(header.data() + 24, size);
+
+ std::lock_guard<std::mutex> lock(mutex_);
+ HRESULT result = WriteAll(header.data(), static_cast<DWORD>(header.size()));
+ return SUCCEEDED(result) ? WriteAll(data, size) : result;
+ }
+
+ HRESULT WriteStatus(const std::string& value) {
+ return Write(OutputKind::status, 0, 0, 0,
+ reinterpret_cast<const BYTE*>(value.data()),
+ static_cast<DWORD>(value.size()));
+ }
+
+ private:
+ HRESULT WriteAll(const BYTE* data, DWORD size) {
+ HANDLE output = GetStdHandle(STD_OUTPUT_HANDLE);
+ while (size > 0) {
+ DWORD written = 0;
+ if (!WriteFile(output, data, size, &written, nullptr)) {
+ return HRESULT_FROM_WIN32(GetLastError());
+ }
+ if (written == 0) return HRESULT_FROM_WIN32(ERROR_WRITE_FAULT);
+ data += written;
+ size -= written;
+ }
+ return S_OK;
+ }
+
+ std::mutex mutex_;
+};
+
+ComPtr<IMFSample> CreateSurfaceSample(ID3D11Texture2D* texture,
+ LONGLONG timestamp100ns,
+ LONGLONG duration100ns) {
+ ComPtr<IMFMediaBuffer> buffer;
+ Check(MFCreateDXGISurfaceBuffer(IID_ID3D11Texture2D, texture, 0, FALSE,
+ &buffer),
+ "input-dxgi-buffer");
+ ComPtr<IMFSample> sample;
+ Check(MFCreateVideoSampleFromSurface(nullptr, &sample),
+ "input-video-sample");
+ Check(sample->AddBuffer(buffer.Get()), "input-sample-buffer");
+ Check(sample->SetSampleTime(timestamp100ns), "input-sample-time");
+ Check(sample->SetSampleDuration(duration100ns), "input-sample-duration");
+ return sample;
+}
+
+struct EncodedAccessUnit final {
+ UINT64 timestamp100ns = 0;
+ bool key_frame = false;
+ std::vector<UINT8> bytes;
+};
+
+class LiveEncoder final {
+ public:
+ explicit LiveEncoder(SelectedTransform selected)
+ : selected_(std::move(selected)) {
+ ConfigureCodec(selected_.codec.Get());
+ ComPtr<IMFMediaType> output_type = CreateOutputType();
+ Check(selected_.transform->SetOutputType(0, output_type.Get(), 0),
+ "mft-set-output-type");
+ ComPtr<IMFMediaType> input_type = CreateInputType();
+ Check(selected_.transform->SetInputType(0, input_type.Get(), 0),
+ "mft-set-input-type");
+ ValidateMediaTypes(selected_.transform.Get());
+ ValidateCodecReadback(selected_.codec.Get());
+ Check(selected_.transform->GetOutputStreamInfo(0, &output_info_),
+ "mft-output-stream-info");
+ Check(selected_.transform->ProcessMessage(
+ MFT_MESSAGE_NOTIFY_BEGIN_STREAMING, 0),
+ "mft-begin-streaming");
+ Check(selected_.transform->ProcessMessage(
+ MFT_MESSAGE_NOTIFY_START_OF_STREAM, 0),
+ "mft-start-stream");
+ started_ = true;
+ }
+
+ LiveEncoder(const LiveEncoder&) = delete;
+ LiveEncoder& operator=(const LiveEncoder&) = delete;
+
+ ~LiveEncoder() {
+ if (started_) {
+ selected_.transform->ProcessMessage(MFT_MESSAGE_COMMAND_FLUSH, 0);
+ selected_.transform->ProcessMessage(MFT_MESSAGE_NOTIFY_END_STREAMING, 0);
+ }
+ }
+
+ const SelectedTransform& selected() const { return selected_; }
+
+ EncodedAccessUnit Encode(ID3D11Texture2D* texture, UINT64 timestamp100ns,
+ bool force_key_frame) {
+ WaitForInput();
+ if (force_key_frame) ForceKeyFrame(selected_.codec.Get());
+ ComPtr<IMFSample> sample = CreateSurfaceSample(
+ texture, static_cast<LONGLONG>(timestamp100ns), kFrameDuration100ns);
+ Check(selected_.transform->ProcessInput(0, sample.Get(), 0),
+ "mft-process-input");
+ --input_requests_;
+
+ const auto deadline = std::chrono::steady_clock::now() +
+ std::chrono::seconds(2);
+ while (std::chrono::steady_clock::now() < deadline) {
+ MediaEventType type = NextEvent(deadline);
+ if (type == METransformNeedInput) {
+ ++input_requests_;
+ continue;
+ }
+ if (type != METransformHaveOutput) continue;
+
+ ComPtr<IMFSample> output = PullOutput(selected_.transform.Get(),
+ output_info_);
+ LONGLONG output_time = -1;
+ Check(output->GetSampleTime(&output_time), "output-sample-time");
+ if (output_time != static_cast<LONGLONG>(timestamp100ns)) {
+ Fail("output-order", "hardware MFT changed or reordered a live timestamp");
+ }
+ std::vector<UINT8> bytes = ReadSample(output.Get());
+ NalSummary nal = InspectAnnexB(bytes);
+ if (!nal.annex_b) {
+ Fail("bitstream-annexb", "live output is not Annex-B H264");
+ }
+ if (nal.profile_level_id) {
+ if (*nal.profile_level_id != "42c01f") {
+ Fail("bitstream-profile", "hardware MFT did not emit profile-level-id 42c01f");
+ }
+ if (profile_level_id_ && *profile_level_id_ != *nal.profile_level_id) {
+ Fail("bitstream-profile-change", "hardware MFT changed SPS profile");
+ }
+ profile_level_id_ = nal.profile_level_id;
+ }
+ if (force_key_frame && !(nal.sps && nal.pps && nal.idr)) {
+ Fail("bitstream-recovery-unit",
+ "requested live recovery unit lacks SPS, PPS, or IDR");
+ }
+ if (!profile_level_id_) {
+ Fail("bitstream-sps", "first live access unit did not carry an SPS");
+ }
+ return EncodedAccessUnit{timestamp100ns, nal.idr, std::move(bytes)};
+ }
+ Fail("mft-output-timeout", "hardware MFT did not produce live output in time");
+ }
+
+ private:
+ MediaEventType NextEvent(std::chrono::steady_clock::time_point deadline) {
+ while (std::chrono::steady_clock::now() < deadline) {
+ ComPtr<IMFMediaEvent> event;
+ HRESULT result = selected_.events->GetEvent(MF_EVENT_FLAG_NO_WAIT,
+ &event);
+ if (result == MF_E_NO_EVENTS_AVAILABLE) {
+ std::this_thread::sleep_for(std::chrono::milliseconds(1));
+ continue;
+ }
+ Check(result, "mft-get-event");
+ HRESULT status = S_OK;
+ Check(event->GetStatus(&status), "mft-event-status-read");
+ Check(status, "mft-event-status");
+ MediaEventType type = MEUnknown;
+ Check(event->GetType(&type), "mft-event-type");
+ if (type == MEError) Fail("mft-error-event", "hardware MFT emitted MEError");
+ return type;
+ }
+ Fail("mft-event-timeout", "hardware MFT did not request input in time");
+ }
+
+ void WaitForInput() {
+ const auto deadline = std::chrono::steady_clock::now() +
+ std::chrono::seconds(2);
+ while (input_requests_ == 0) {
+ MediaEventType type = NextEvent(deadline);
+ if (type == METransformNeedInput) {
+ ++input_requests_;
+ } else if (type == METransformHaveOutput) {
+ Fail("mft-unexpected-output", "hardware MFT produced output without live input");
+ }
+ }
+ }
+
+ SelectedTransform selected_;
+ MFT_OUTPUT_STREAM_INFO output_info_ = {};
+ UINT32 input_requests_ = 0;
+ std::optional<std::string> profile_level_id_;
+ bool started_ = false;
+};
+
+class FrameConverter final {
+ public:
+ explicit FrameConverter(ID3D11Device* device) : device_(device) {
+ Check(device_->QueryInterface(IID_PPV_ARGS(&video_device_)),
+ "video-processor-device");
+ ComPtr<ID3D11DeviceContext> context;
+ device_->GetImmediateContext(&context);
+ Check(context.As(&video_context_), "video-processor-context");
+ }
+
+ ComPtr<ID3D11Texture2D> Convert(ID3D11Texture2D* source, UINT32 width,
+ UINT32 height) {
+ if (source == nullptr || width == 0 || height == 0 ||
+ width > 16'384 || height > 16'384) {
+ Fail("capture-size", "captured window dimensions are invalid");
+ }
+ if (!enumerator_ || width != input_width_ || height != input_height_) {
+ Configure(width, height);
+ }
+
+ D3D11_TEXTURE2D_DESC output_description = {};
+ output_description.Width = kWidth;
+ output_description.Height = kHeight;
+ output_description.MipLevels = 1;
+ output_description.ArraySize = 1;
+ output_description.Format = DXGI_FORMAT_NV12;
+ output_description.SampleDesc.Count = 1;
+ output_description.Usage = D3D11_USAGE_DEFAULT;
+ output_description.BindFlags = D3D11_BIND_RENDER_TARGET;
+ ComPtr<ID3D11Texture2D> output;
+ Check(device_->CreateTexture2D(&output_description, nullptr, &output),
+ "video-processor-output-texture");
+
+ D3D11_VIDEO_PROCESSOR_INPUT_VIEW_DESC input_description = {};
+ input_description.ViewDimension = D3D11_VPIV_DIMENSION_TEXTURE2D;
+ input_description.Texture2D.ArraySlice = 0;
+ input_description.Texture2D.MipSlice = 0;
+ ComPtr<ID3D11VideoProcessorInputView> input_view;
+ Check(video_device_->CreateVideoProcessorInputView(
+ source, enumerator_.Get(), &input_description, &input_view),
+ "video-processor-input-view");
+
+ D3D11_VIDEO_PROCESSOR_OUTPUT_VIEW_DESC output_view_description = {};
+ output_view_description.ViewDimension = D3D11_VPOV_DIMENSION_TEXTURE2D;
+ output_view_description.Texture2D.MipSlice = 0;
+ ComPtr<ID3D11VideoProcessorOutputView> output_view;
+ Check(video_device_->CreateVideoProcessorOutputView(
+ output.Get(), enumerator_.Get(), &output_view_description,
+ &output_view),
+ "video-processor-output-view");
+
+ RECT source_rect = {0, 0, static_cast<LONG>(width),
+ static_cast<LONG>(height)};
+ double scale = std::min(static_cast<double>(kWidth) / width,
+ static_cast<double>(kHeight) / height);
+ LONG target_width = std::max<LONG>(
+ 2, static_cast<LONG>(std::llround(width * scale)) & ~1L);
+ LONG target_height = std::max<LONG>(
+ 2, static_cast<LONG>(std::llround(height * scale)) & ~1L);
+ target_width = std::min<LONG>(target_width, kWidth);
+ target_height = std::min<LONG>(target_height, kHeight);
+ LONG left = (static_cast<LONG>(kWidth) - target_width) / 2;
+ LONG top = (static_cast<LONG>(kHeight) - target_height) / 2;
+ RECT target_rect = {left, top, left + target_width, top + target_height};
+ RECT output_rect = {0, 0, static_cast<LONG>(kWidth),
+ static_cast<LONG>(kHeight)};
+
+ D3D11_VIDEO_COLOR background = {};
+ background.RGBA.A = 1.0f;
+ video_context_->VideoProcessorSetOutputBackgroundColor(processor_.Get(),
+ FALSE, &background);
+ video_context_->VideoProcessorSetOutputTargetRect(processor_.Get(), TRUE,
+ &output_rect);
+ video_context_->VideoProcessorSetStreamFrameFormat(
+ processor_.Get(), 0, D3D11_VIDEO_FRAME_FORMAT_PROGRESSIVE);
+ video_context_->VideoProcessorSetStreamSourceRect(processor_.Get(), 0,
+ TRUE, &source_rect);
+ video_context_->VideoProcessorSetStreamDestRect(processor_.Get(), 0, TRUE,
+ &target_rect);
+ video_context_->VideoProcessorSetStreamAutoProcessingMode(processor_.Get(),
+ 0, FALSE);
+ D3D11_VIDEO_PROCESSOR_STREAM stream = {};
+ stream.Enable = TRUE;
+ stream.pInputSurface = input_view.Get();
+ Check(video_context_->VideoProcessorBlt(processor_.Get(), output_view.Get(),
+ 0, 1, &stream),
+ "video-processor-blit");
+ return output;
+ }
+
+ private:
+ void Configure(UINT32 width, UINT32 height) {
+ D3D11_VIDEO_PROCESSOR_CONTENT_DESC description = {};
+ description.InputFrameFormat = D3D11_VIDEO_FRAME_FORMAT_PROGRESSIVE;
+ description.InputFrameRate = {kFrameRate, 1};
+ description.InputWidth = width;
+ description.InputHeight = height;
+ description.OutputFrameRate = {kFrameRate, 1};
+ description.OutputWidth = kWidth;
+ description.OutputHeight = kHeight;
+ description.Usage = D3D11_VIDEO_USAGE_PLAYBACK_NORMAL;
+ ComPtr<ID3D11VideoProcessorEnumerator> enumerator;
+ Check(video_device_->CreateVideoProcessorEnumerator(&description,
+ &enumerator),
+ "video-processor-enumerator");
+ ComPtr<ID3D11VideoProcessor> processor;
+ Check(video_device_->CreateVideoProcessor(enumerator.Get(), 0, &processor),
+ "video-processor-create");
+ enumerator_ = std::move(enumerator);
+ processor_ = std::move(processor);
+ input_width_ = width;
+ input_height_ = height;
+ }
+
+ ComPtr<ID3D11Device> device_;
+ ComPtr<ID3D11VideoDevice> video_device_;
+ ComPtr<ID3D11VideoContext> video_context_;
+ ComPtr<ID3D11VideoProcessorEnumerator> enumerator_;
+ ComPtr<ID3D11VideoProcessor> processor_;
+ UINT32 input_width_ = 0;
+ UINT32 input_height_ = 0;
+};
+
+winrt::Windows::Graphics::DirectX::Direct3D11::IDirect3DDevice
+CreateCaptureDevice(ID3D11Device* device) {
+ ComPtr<IDXGIDevice> dxgi_device;
+ Check(device->QueryInterface(IID_PPV_ARGS(&dxgi_device)),
+ "capture-dxgi-device");
+ winrt::com_ptr<IInspectable> inspectable;
+ Check(CreateDirect3D11DeviceFromDXGIDevice(dxgi_device.Get(),
+ inspectable.put()),
+ "capture-winrt-device");
+ return inspectable.as<
+ winrt::Windows::Graphics::DirectX::Direct3D11::IDirect3DDevice>();
+}
+
+winrt::Windows::Graphics::Capture::GraphicsCaptureItem CreateCaptureItem(
+ HWND window) {
+ using winrt::Windows::Graphics::Capture::GraphicsCaptureItem;
+ auto interop = winrt::get_activation_factory<GraphicsCaptureItem,
+ IGraphicsCaptureItemInterop>();
+ GraphicsCaptureItem item{nullptr};
+ Check(interop->CreateForWindow(window, winrt::guid_of<GraphicsCaptureItem>(),
+ winrt::put_abi(item)),
+ "capture-item-window");
+ return item;
+}
+
+ComPtr<ID3D11Texture2D> CaptureTexture(
+ const winrt::Windows::Graphics::Capture::Direct3D11CaptureFrame& frame) {
+ auto access = frame.Surface().as<
+ ::Windows::Graphics::DirectX::Direct3D11::IDirect3DDxgiInterfaceAccess>();
+ ComPtr<ID3D11Texture2D> texture;
+ Check(access->GetInterface(IID_PPV_ARGS(&texture)),
+ "capture-frame-texture");
+ return texture;
+}
+
+std::string JSONString(const std::string& value) {
+ std::string output = "\"";
+ for (unsigned char character : value) {
+ if (character == '\\' || character == '"') {
+ output.push_back('\\');
+ output.push_back(static_cast<char>(character));
+ } else if (character >= 0x20 && character <= 0x7e) {
+ output.push_back(static_cast<char>(character));
+ } else {
+ output.push_back('?');
+ }
+ }
+ output.push_back('"');
+ return output;
+}
+
+bool ConsumeKeyFrameRequest() {
+ HANDLE input = GetStdHandle(STD_INPUT_HANDLE);
+ DWORD available = 0;
+ if (input == INVALID_HANDLE_VALUE || input == nullptr ||
+ !PeekNamedPipe(input, nullptr, 0, nullptr, &available, nullptr) ||
+ available == 0) {
+ return false;
+ }
+ std::array<BYTE, 64> bytes{};
+ DWORD read = 0;
+ if (!ReadFile(input, bytes.data(),
+ std::min<DWORD>(available, static_cast<DWORD>(bytes.size())),
+ &read, nullptr)) {
+ return false;
+ }
+ return std::find(bytes.begin(), bytes.begin() + read,
+ static_cast<BYTE>('K')) != bytes.begin() + read;
+}
+
+UINT64 ParseUint64(const wchar_t* value, const std::string& stage) {
+ try {
+ size_t consumed = 0;
+ unsigned long long parsed = std::stoull(value, &consumed, 10);
+ if (value[consumed] != L'\0' || parsed == 0) {
+ Fail(stage, "value is not a positive bounded integer");
+ }
+ return static_cast<UINT64>(parsed);
+ } catch (const std::exception&) {
+ Fail(stage, "value is not a positive bounded integer");
+ }
+}
+
+struct ProductArguments final {
+ enum class Mode { list, audio, window } mode = Mode::list;
+ DWORD pid = 0;
+ UINT64 creation_time = 0;
+ UINT64 window_handle = 0;
+ UINT adapter_index = 0;
+ UINT mft_index = 0;
+};
+
+ProductArguments ParseProductArguments(int count, wchar_t** values) {
+ ProductArguments arguments;
+ if (count == 2 && std::wstring(values[1]) == L"--list") return arguments;
+ if (count == 4 && std::wstring(values[1]) == L"--capture-audio") {
+ arguments.mode = ProductArguments::Mode::audio;
+ } else if (count == 10 && std::wstring(values[1]) == L"--capture-window" &&
+ std::wstring(values[5]) == L"--adapter-index" &&
+ std::wstring(values[7]) == L"--mft-index" &&
+ std::wstring(values[9]) == L"--protocol-v1") {
+ arguments.mode = ProductArguments::Mode::window;
+ arguments.window_handle = ParseUint64(values[4], "argument-window");
+ arguments.adapter_index = ParseIndex(values[6], "argument-adapter");
+ arguments.mft_index = ParseIndex(values[8], "argument-mft");
+ } else {
+ Fail("arguments", "unsupported or incomplete command-line argument");
+ }
+ UINT64 pid = ParseUint64(values[2], "argument-pid");
+ if (pid > std::numeric_limits<DWORD>::max()) {
+ Fail("argument-pid", "PID is outside the Windows process range");
+ }
+ arguments.pid = static_cast<DWORD>(pid);
+ arguments.creation_time = ParseUint64(values[3], "argument-creation-time");
+ return arguments;
+}
+
+HRESULT RunAudioCapture(const ProductArguments& arguments) {
+ UniqueHandle stop(CreateEventW(nullptr, TRUE, FALSE, nullptr));
+ if (stop.get() == nullptr) return HRESULT_FROM_WIN32(GetLastError());
+ ProtocolWriter writer;
+ return screener::capture::CaptureProcessAudio(
+ arguments.pid, arguments.creation_time, stop.get(),
+ [&writer](UINT64 timestamp100ns, const BYTE* data, DWORD size) {
+ return writer.Write(OutputKind::pcm, 0, timestamp100ns,
+ screener::capture::kAudioChunkDuration100ns, data,
+ size);
+ });
+}
+
+void RunWindowCapture(const ProductArguments& arguments) {
+ HRESULT identity = screener::capture::ValidateWindowTarget(
+ arguments.window_handle, arguments.pid, arguments.creation_time);
+ Check(identity, "target-identity");
+
+ std::vector<Adapter> adapters = EnumerateAdapters();
+ const Adapter& adapter = SelectAdapter(adapters, arguments.adapter_index);
+ DeviceContext device = CreateDevice(adapter);
+ ActivationList activations = EnumerateHardwareEncoders(adapter);
+ SelectedTransform selected = ActivateTransform(
+ activations, arguments.mft_index, device.manager.Get());
+ LiveEncoder encoder(std::move(selected));
+ FrameConverter converter(device.device.Get());
+ ProtocolWriter writer;
+
+ using namespace winrt::Windows::Graphics::Capture;
+ using winrt::Windows::Graphics::DirectX::DirectXPixelFormat;
+ if (!GraphicsCaptureSession::IsSupported()) {
+ Fail("capture-support", "Windows Graphics Capture is unavailable");
+ }
+ UniqueHandle process(OpenProcess(SYNCHRONIZE, FALSE, arguments.pid));
+ if (process.get() == nullptr) {
+ Check(HRESULT_FROM_WIN32(GetLastError()), "target-process-handle");
+ }
+ Check(screener::capture::ValidateWindowTarget(
+ arguments.window_handle, arguments.pid, arguments.creation_time),
+ "target-identity-before-capture");
+ HWND window = reinterpret_cast<HWND>(
+ static_cast<UINT_PTR>(arguments.window_handle));
+ auto capture_device = CreateCaptureDevice(device.device.Get());
+ GraphicsCaptureItem item = CreateCaptureItem(window);
+ Check(screener::capture::ValidateWindowTarget(
+ arguments.window_handle, arguments.pid, arguments.creation_time),
+ "target-identity-after-item");
+ auto initial_size = item.Size();
+ if (initial_size.Width <= 0 || initial_size.Height <= 0) {
+ Fail("capture-size", "selected window has no capturable content");
+ }
+ Direct3D11CaptureFramePool pool = Direct3D11CaptureFramePool::CreateFreeThreaded(
+ capture_device, DirectXPixelFormat::B8G8R8A8UIntNormalized, 2,
+ initial_size);
+ GraphicsCaptureSession capture_session = pool.CreateCaptureSession(item);
+
+ UniqueHandle shutdown(CreateEventW(nullptr, TRUE, FALSE, nullptr));
+ UniqueHandle frame_ready(CreateEventW(nullptr, FALSE, FALSE, nullptr));
+ if (shutdown.get() == nullptr || frame_ready.get() == nullptr) {
+ Check(HRESULT_FROM_WIN32(GetLastError()), "capture-events");
+ }
+
+ std::atomic<LONG> audio_result{S_OK};
+ std::atomic<bool> audio_finished{false};
+ std::atomic<bool> item_closed{false};
+ auto frame_token = pool.FrameArrived(
+ [event = frame_ready.get()](const Direct3D11CaptureFramePool&,
+ const winrt::Windows::Foundation::IInspectable&) {
+ SetEvent(event);
+ });
+ auto closed_token = item.Closed(
+ [event = shutdown.get(), &item_closed](const GraphicsCaptureItem&,
+ const winrt::Windows::Foundation::IInspectable&) {
+ item_closed.store(true);
+ SetEvent(event);
+ });
+ std::thread audio([&]() {
+ HRESULT result = screener::capture::CaptureProcessAudio(
+ arguments.pid, arguments.creation_time, shutdown.get(),
+ [&writer](UINT64 timestamp100ns, const BYTE* data, DWORD size) {
+ return writer.Write(OutputKind::pcm, 0, timestamp100ns,
+ screener::capture::kAudioChunkDuration100ns,
+ data, size);
+ });
+ audio_result.store(result);
+ audio_finished.store(true);
+ SetEvent(shutdown.get());
+ });
+
+ auto cleanup = [&]() noexcept {
+ SetEvent(shutdown.get());
+ try {
+ pool.FrameArrived(frame_token);
+ } catch (...) {
+ }
+ try {
+ item.Closed(closed_token);
+ } catch (...) {
+ }
+ try {
+ capture_session.Close();
+ } catch (...) {
+ }
+ try {
+ pool.Close();
+ } catch (...) {
+ }
+ if (audio.joinable()) audio.join();
+ };
+
+ try {
+ std::ostringstream starting;
+ starting << "{\"state\":\"starting\",\"hardwareOnly\":true,"
+ << "\"adapterIndex\":" << adapter.index
+ << ",\"adapterName\":"
+ << JSONString(NarrowAscii(adapter.description.Description))
+ << ",\"adapterLuid\":"
+ << JSONString(LuidString(adapter.description.AdapterLuid))
+ << ",\"mftIndex\":" << arguments.mft_index
+ << ",\"mftName\":"
+ << JSONString(encoder.selected().name)
+ << ",\"mftClsid\":"
+ << JSONString(encoder.selected().clsid) << '}';
+ Check(writer.WriteStatus(starting.str()), "capture-status-starting");
+ capture_session.StartCapture();
+
+ UINT64 previous_timestamp = 0;
+ UINT64 encoded_frames = 0;
+ bool active_status_written = false;
+ auto pool_size = initial_size;
+ const HANDLE waits[] = {process.get(), shutdown.get(), frame_ready.get()};
+ for (;;) {
+ DWORD wait = WaitForMultipleObjects(3, waits, FALSE, 5'000);
+ if (wait == WAIT_OBJECT_0) {
+ Fail("target-exited", "selected target process exited");
+ }
+ if (wait == WAIT_OBJECT_0 + 1) {
+ if (item_closed.load()) {
+ Fail("capture-closed", "selected window stopped capture");
+ }
+ HRESULT result = static_cast<HRESULT>(audio_result.load());
+ if (audio_finished.load() && result != HRESULT_FROM_WIN32(ERROR_CANCELLED)) {
+ throw GateFailure("process-audio", "target process audio stopped",
+ result);
+ }
+ Fail("capture-stopped", "window capture stopped");
+ }
+ if (wait == WAIT_TIMEOUT) {
+ Fail("capture-frame-timeout", "selected window produced no frame in five seconds");
+ }
+ if (wait != WAIT_OBJECT_0 + 2) {
+ Check(HRESULT_FROM_WIN32(GetLastError()), "capture-wait");
+ }
+
+ Direct3D11CaptureFrame latest{nullptr};
+ for (;;) {
+ Direct3D11CaptureFrame next = pool.TryGetNextFrame();
+ if (!next) break;
+ latest = std::move(next);
+ }
+ if (!latest) continue;
+ INT64 signed_timestamp = latest.SystemRelativeTime().count();
+ if (signed_timestamp <= 0) {
+ Fail("capture-timestamp", "captured frame has no QPC timestamp");
+ }
+ UINT64 timestamp = static_cast<UINT64>(signed_timestamp);
+ if (previous_timestamp != 0 && timestamp <= previous_timestamp) continue;
+ if (previous_timestamp != 0 &&
+ timestamp - previous_timestamp <
+ static_cast<UINT64>(kFrameDuration100ns) * 19 / 20) {
+ continue;
+ }
+ auto content_size = latest.ContentSize();
+ if (content_size.Width <= 0 || content_size.Height <= 0) continue;
+ if (content_size.Width != pool_size.Width ||
+ content_size.Height != pool_size.Height) {
+ latest.Close();
+ latest = nullptr;
+ pool.Recreate(capture_device,
+ DirectXPixelFormat::B8G8R8A8UIntNormalized, 2,
+ content_size);
+ pool_size = content_size;
+ continue;
+ }
+ ComPtr<ID3D11Texture2D> source = CaptureTexture(latest);
+ D3D11_TEXTURE2D_DESC source_description = {};
+ source->GetDesc(&source_description);
+ UINT32 content_width = std::min<UINT32>(
+ source_description.Width, static_cast<UINT32>(content_size.Width));
+ UINT32 content_height = std::min<UINT32>(
+ source_description.Height, static_cast<UINT32>(content_size.Height));
+ ComPtr<ID3D11Texture2D> nv12 = converter.Convert(
+ source.Get(), content_width, content_height);
+ bool requested_key_frame = ConsumeKeyFrameRequest();
+ bool key_frame = encoded_frames == 0 || encoded_frames % kGopFrames == 0 ||
+ requested_key_frame;
+ EncodedAccessUnit access_unit = encoder.Encode(nv12.Get(), timestamp,
+ key_frame);
+ if (access_unit.bytes.size() > kMaxProductAccessUnitBytes) {
+ Fail("output-buffer-bounds", "live H264 access unit exceeded one MiB");
+ }
+ UINT64 wire_timestamp = (access_unit.timestamp100ns / 10) * 10;
+ Check(writer.Write(OutputKind::h264, access_unit.key_frame ? 1 : 0,
+ wire_timestamp, kEnvelopeFrameDuration100ns,
+ access_unit.bytes.data(),
+ static_cast<DWORD>(access_unit.bytes.size())),
+ "capture-video-output");
+ previous_timestamp = timestamp;
+ ++encoded_frames;
+ if (!active_status_written) {
+ std::string active =
+ "{\"state\":\"active\",\"hardwareOnly\":true,"
+ "\"profileLevelId\":\"42c01f\",\"width\":1280,"
+ "\"height\":720,\"fps\":30}";
+ Check(writer.WriteStatus(active), "capture-status-active");
+ active_status_written = true;
+ }
+ }
+ } catch (...) {
+ cleanup();
+ throw;
+ }
+}
+#endif
+
+#ifdef SCREENER_H264_FIXTURE
struct Arguments final {
bool list = false;
std::optional<UINT> adapter_index;
@@ -1236,9 +1974,11 @@ void PrintEvidence(const Adapter& adapter, UINT mft_index,
<< "videoencode_mean_percent=" << mean_video_encode << '\n'
<< "videoencode_max_percent=" << max_video_encode << '\n';
}
+#endif
} // namespace
+#ifdef SCREENER_H264_FIXTURE
int wmain(int argc, wchar_t** argv) {
try {
Arguments arguments = ParseArguments(argc, argv);
@@ -1288,3 +2028,39 @@ int wmain(int argc, wchar_t** argv) {
return 3;
}
}
+#endif
+
+#ifndef SCREENER_H264_FIXTURE
+int wmain(int argc, wchar_t** argv) {
+ try {
+ ProductArguments arguments = ParseProductArguments(argc, argv);
+ if (arguments.mode == ProductArguments::Mode::list) {
+ return screener::capture::WriteWindowList();
+ }
+ if (_setmode(_fileno(stdout), _O_BINARY) == -1) {
+ Fail("stdout-binary", "could not switch stdout to binary mode");
+ }
+ if (arguments.mode == ProductArguments::Mode::audio) {
+ Check(RunAudioCapture(arguments), "process-audio-capture");
+ return 0;
+ }
+ Runtime runtime;
+ RunWindowCapture(arguments);
+ return 0;
+ } catch (const GateFailure& error) {
+ std::cerr << "result=window-capture-failed\n"
+ << "stage=" << error.stage() << '\n'
+ << "detail=" << error.what() << '\n';
+ if (FAILED(error.result())) {
+ std::cerr << "hresult=0x" << std::hex << std::setfill('0')
+ << std::setw(8) << static_cast<UINT32>(error.result()) << '\n';
+ }
+ return 2;
+ } catch (const std::exception& error) {
+ std::cerr << "result=window-capture-failed\n"
+ << "stage=unclassified\n"
+ << "detail=" << error.what() << '\n';
+ return 3;
+ }
+}
+#endif
diff --git a/native/process-audio-helper/main.cpp b/native/window-capture-helper/process_audio.cpp
similarity index 73%
rename from native/process-audio-helper/main.cpp
rename to native/window-capture-helper/process_audio.cpp
index 95c5b6e7..85af9c90 100644
--- a/native/process-audio-helper/main.cpp
+++ b/native/window-capture-helper/process_audio.cpp
@@ -9,10 +9,11 @@
#include <wrl.h>
#include <wrl/implements.h>
-#include <array>
+#include "process_audio.h"
+
#include <cstdint>
-#include <cstdlib>
#include <iostream>
+#include <limits>
#include <string>
#include <vector>
@@ -23,13 +24,8 @@ using Microsoft::WRL::RuntimeClass;
using Microsoft::WRL::RuntimeClassFlags;
using Microsoft::WRL::ClassicCom;
+namespace screener::capture {
namespace {
-constexpr UINT32 kSampleRate = 48'000;
-constexpr UINT16 kChannels = 2;
-constexpr UINT16 kBytesPerSample = 2;
-constexpr UINT32 kFramesPerChunk = 960;
-constexpr DWORD kBytesPerChunk = kFramesPerChunk * kChannels * kBytesPerSample;
-constexpr UINT64 kChunkDuration100ns = 200'000;
constexpr size_t kMaxWindows = 100;
class ActivationHandler final : public RuntimeClass<
@@ -57,6 +53,7 @@ class ActivationHandler final : public RuntimeClass<
};
struct WindowTarget {
+ UINT64 windowHandle;
DWORD pid;
UINT64 creationTime;
std::string title;
@@ -116,7 +113,7 @@ std::string JsonString(const std::string& value) {
BOOL CALLBACK CollectWindow(HWND window, LPARAM parameter) {
auto* targets = reinterpret_cast<std::vector<WindowTarget>*>(parameter);
- if (targets->size() >= kMaxWindows) return FALSE;
+ if (targets->size() >= kMaxWindows) return TRUE;
if (!IsWindowVisible(window) || GetWindow(window, GW_OWNER) != nullptr) return TRUE;
const int length = GetWindowTextLengthW(window);
if (length <= 0 || length > 512) return TRUE;
@@ -134,17 +131,23 @@ BOOL CALLBACK CollectWindow(HWND window, LPARAM parameter) {
UINT64 creationTime = 0;
const HRESULT result = ReadProcessCreationTime(process, &creationTime);
CloseHandle(process);
- if (SUCCEEDED(result)) targets->push_back({pid, creationTime, std::move(utf8)});
+ if (SUCCEEDED(result)) {
+ targets->push_back({static_cast<UINT64>(reinterpret_cast<UINT_PTR>(window)),
+ pid, creationTime, std::move(utf8)});
+ }
return TRUE;
}
-int ListWindows() {
+} // namespace
+
+int WriteWindowList() {
std::vector<WindowTarget> targets;
if (!EnumWindows(CollectWindow, reinterpret_cast<LPARAM>(&targets))) return 2;
std::cout << '[';
for (size_t index = 0; index < targets.size(); ++index) {
if (index != 0) std::cout << ',';
- std::cout << "{\"pid\":" << targets[index].pid << ",\"creationTime\":"
+ std::cout << "{\"windowHandle\":" << targets[index].windowHandle
+ << ",\"pid\":" << targets[index].pid << ",\"creationTime\":"
<< targets[index].creationTime << ",\"title\":"
<< JsonString(targets[index].title) << '}';
}
@@ -152,43 +155,6 @@ int ListWindows() {
return std::cout.good() ? 0 : 2;
}
-void PutUint32BE(BYTE* output, UINT32 value) {
- for (int index = 3; index >= 0; --index) {
- output[index] = static_cast<BYTE>(value & 0xff);
- value >>= 8;
- }
-}
-
-void PutUint64BE(BYTE* output, UINT64 value) {
- for (int index = 7; index >= 0; --index) {
- output[index] = static_cast<BYTE>(value & 0xff);
- value >>= 8;
- }
-}
-
-HRESULT WriteAll(HANDLE output, const BYTE* data, DWORD size) {
- while (size > 0) {
- DWORD written = 0;
- if (!WriteFile(output, data, size, &written, nullptr) || written == 0) {
- return HRESULT_FROM_WIN32(GetLastError());
- }
- data += written;
- size -= written;
- }
- return S_OK;
-}
-
-HRESULT WritePCM(HANDLE output, UINT64 timestamp100ns, const BYTE* data) {
- std::array<BYTE, 28> header{};
- header[0] = 'S'; header[1] = 'P'; header[2] = 'C'; header[3] = 'M';
- header[4] = 1;
- PutUint64BE(header.data() + 8, timestamp100ns);
- PutUint64BE(header.data() + 16, kChunkDuration100ns);
- PutUint32BE(header.data() + 24, kBytesPerChunk);
- HRESULT result = WriteAll(output, header.data(), static_cast<DWORD>(header.size()));
- return SUCCEEDED(result) ? WriteAll(output, data, kBytesPerChunk) : result;
-}
-
HRESULT ActivateProcessLoopback(DWORD pid, HANDLE completed, ComPtr<IAudioClient>* client) {
AUDIOCLIENT_ACTIVATION_PARAMS parameters{};
parameters.ActivationType = AUDIOCLIENT_ACTIVATION_TYPE_PROCESS_LOOPBACK;
@@ -213,9 +179,42 @@ HRESULT ActivateProcessLoopback(DWORD pid, HANDLE completed, ComPtr<IAudioClient
return result;
}
-HRESULT Capture(DWORD pid, UINT64 expectedCreationTime) {
- HANDLE process = OpenProcess(SYNCHRONIZE | PROCESS_QUERY_LIMITED_INFORMATION, FALSE, pid);
+HRESULT ValidateWindowTarget(UINT64 window_handle, DWORD pid,
+ UINT64 expected_creation_time) {
+ if (window_handle == 0 || pid == 0 || expected_creation_time == 0 ||
+ window_handle > static_cast<UINT64>(std::numeric_limits<UINT_PTR>::max())) {
+ return HRESULT_FROM_WIN32(ERROR_INVALID_PARAMETER);
+ }
+ HWND window = reinterpret_cast<HWND>(static_cast<UINT_PTR>(window_handle));
+ DWORD window_pid = 0;
+ if (!IsWindow(window) || GetWindowThreadProcessId(window, &window_pid) == 0 ||
+ window_pid != pid) {
+ return HRESULT_FROM_WIN32(ERROR_INVALID_WINDOW_HANDLE);
+ }
+ HANDLE process = OpenProcess(PROCESS_QUERY_LIMITED_INFORMATION, FALSE, pid);
if (process == nullptr) return HRESULT_FROM_WIN32(GetLastError());
+ UINT64 actual_creation_time = 0;
+ HRESULT result = ReadProcessCreationTime(process, &actual_creation_time);
+ CloseHandle(process);
+ if (SUCCEEDED(result) && actual_creation_time != expected_creation_time) {
+ result = HRESULT_FROM_WIN32(ERROR_INVALID_DATA);
+ }
+ return result;
+}
+
+HRESULT CaptureProcessAudio(DWORD pid, UINT64 expectedCreationTime,
+ HANDLE stop_event, const PCMWriter& writer) {
+ if (stop_event == nullptr || !writer) {
+ return HRESULT_FROM_WIN32(ERROR_INVALID_PARAMETER);
+ }
+ HRESULT com_result = CoInitializeEx(nullptr, COINIT_MULTITHREADED);
+ if (FAILED(com_result)) return com_result;
+ HANDLE process = OpenProcess(SYNCHRONIZE | PROCESS_QUERY_LIMITED_INFORMATION, FALSE, pid);
+ if (process == nullptr) {
+ HRESULT result = HRESULT_FROM_WIN32(GetLastError());
+ CoUninitialize();
+ return result;
+ }
HANDLE completed = CreateEventW(nullptr, FALSE, FALSE, nullptr);
HANDLE sampleReady = CreateEventW(nullptr, FALSE, FALSE, nullptr);
if (completed == nullptr || sampleReady == nullptr) {
@@ -223,6 +222,7 @@ HRESULT Capture(DWORD pid, UINT64 expectedCreationTime) {
if (completed != nullptr) CloseHandle(completed);
if (sampleReady != nullptr) CloseHandle(sampleReady);
CloseHandle(process);
+ CoUninitialize();
return result;
}
UINT64 creationTime = 0;
@@ -235,11 +235,11 @@ HRESULT Capture(DWORD pid, UINT64 expectedCreationTime) {
ComPtr<IAudioCaptureClient> capture;
WAVEFORMATEX format{};
format.wFormatTag = WAVE_FORMAT_PCM;
- format.nChannels = kChannels;
- format.nSamplesPerSec = kSampleRate;
- format.wBitsPerSample = kBytesPerSample * 8;
- format.nBlockAlign = kChannels * kBytesPerSample;
- format.nAvgBytesPerSec = kSampleRate * format.nBlockAlign;
+ format.nChannels = kAudioChannels;
+ format.nSamplesPerSec = kAudioSampleRate;
+ format.wBitsPerSample = kAudioBytesPerSample * 8;
+ format.nBlockAlign = kAudioChannels * kAudioBytesPerSample;
+ format.nAvgBytesPerSec = kAudioSampleRate * format.nBlockAlign;
if (SUCCEEDED(result)) {
result = client->Initialize(AUDCLNT_SHAREMODE_SHARED,
AUDCLNT_STREAMFLAGS_LOOPBACK | AUDCLNT_STREAMFLAGS_EVENTCALLBACK |
@@ -253,15 +253,18 @@ HRESULT Capture(DWORD pid, UINT64 expectedCreationTime) {
std::vector<BYTE> pending;
size_t consumed = 0;
UINT64 nextTimestamp = 0;
- HANDLE output = GetStdHandle(STD_OUTPUT_HANDLE);
- const HANDLE waits[] = {process, sampleReady};
+ const HANDLE waits[] = {process, sampleReady, stop_event};
while (SUCCEEDED(result)) {
- const DWORD wait = WaitForMultipleObjects(2, waits, FALSE, 1'000);
+ const DWORD wait = WaitForMultipleObjects(3, waits, FALSE, 1'000);
if (wait == WAIT_OBJECT_0) {
result = HRESULT_FROM_WIN32(ERROR_PROCESS_ABORTED);
break;
}
if (wait == WAIT_TIMEOUT) continue;
+ if (wait == WAIT_OBJECT_0 + 2) {
+ result = HRESULT_FROM_WIN32(ERROR_CANCELLED);
+ break;
+ }
if (wait != WAIT_OBJECT_0 + 1) {
result = HRESULT_FROM_WIN32(GetLastError());
break;
@@ -294,14 +297,15 @@ HRESULT Capture(DWORD pid, UINT64 expectedCreationTime) {
pending.insert(pending.end(), data, data + bytes);
}
capture->ReleaseBuffer(frames);
- while (pending.size() - consumed >= kBytesPerChunk) {
- result = WritePCM(output, nextTimestamp, pending.data() + consumed);
+ while (pending.size() - consumed >= kAudioBytesPerChunk) {
+ result = writer(nextTimestamp, pending.data() + consumed,
+ kAudioBytesPerChunk);
if (FAILED(result)) break;
- consumed += kBytesPerChunk;
- nextTimestamp += kChunkDuration100ns;
+ consumed += kAudioBytesPerChunk;
+ nextTimestamp += kAudioChunkDuration100ns;
}
if (FAILED(result)) break;
- if (consumed > kBytesPerChunk * 4) {
+ if (consumed > kAudioBytesPerChunk * 4) {
pending.erase(pending.begin(), pending.begin() + static_cast<ptrdiff_t>(consumed));
consumed = 0;
}
@@ -311,24 +315,8 @@ HRESULT Capture(DWORD pid, UINT64 expectedCreationTime) {
CloseHandle(sampleReady);
CloseHandle(completed);
CloseHandle(process);
+ CoUninitialize();
return result;
}
-} // namespace
-int wmain(int argc, wchar_t** argv) {
- if (argc == 2 && std::wstring(argv[1]) == L"--list") return ListWindows();
- if (argc != 4 || std::wstring(argv[1]) != L"--capture") return 2;
- wchar_t* pidEnd = nullptr;
- const unsigned long parsedPID = wcstoul(argv[2], &pidEnd, 10);
- wchar_t* creationEnd = nullptr;
- const unsigned long long parsedCreationTime = wcstoull(argv[3], &creationEnd, 10);
- if (parsedPID == 0 || pidEnd == argv[2] || *pidEnd != L'\0' ||
- parsedCreationTime == 0 || creationEnd == argv[3] || *creationEnd != L'\0') return 2;
- if (_setmode(_fileno(stdout), _O_BINARY) == -1) return 2;
- HRESULT result = CoInitializeEx(nullptr, COINIT_MULTITHREADED);
- if (FAILED(result)) return 2;
- result = Capture(static_cast<DWORD>(parsedPID), static_cast<UINT64>(parsedCreationTime));
- CoUninitialize();
- if (FAILED(result)) std::cerr << "process_audio_capture_failed hresult=0x" << std::hex << result << '\n';
- return SUCCEEDED(result) ? 0 : 2;
-}
+} // namespace screener::capture
diff --git a/native/window-capture-helper/process_audio.h b/native/window-capture-helper/process_audio.h
new file mode 100644
index 00000000..d2a95d4b
--- /dev/null
+++ b/native/window-capture-helper/process_audio.h
@@ -0,0 +1,27 @@
+#pragma once
+
+#include <windows.h>
+
+#include <cstdint>
+#include <functional>
+
+namespace screener::capture {
+
+constexpr UINT32 kAudioSampleRate = 48'000;
+constexpr UINT16 kAudioChannels = 2;
+constexpr UINT16 kAudioBytesPerSample = 2;
+constexpr UINT32 kAudioFramesPerChunk = 960;
+constexpr DWORD kAudioBytesPerChunk =
+ kAudioFramesPerChunk * kAudioChannels * kAudioBytesPerSample;
+constexpr UINT64 kAudioChunkDuration100ns = 200'000;
+
+using PCMWriter =
+ std::function<HRESULT(UINT64 timestamp100ns, const BYTE* data, DWORD size)>;
+
+int WriteWindowList();
+HRESULT ValidateWindowTarget(UINT64 window_handle, DWORD pid,
+ UINT64 expected_creation_time);
+HRESULT CaptureProcessAudio(DWORD pid, UINT64 expected_creation_time,
+ HANDLE stop_event, const PCMWriter& writer);
+
+} // namespace screener::capture
diff --git a/scripts/native-one-viewer-gate.ts b/scripts/native-one-viewer-gate.ts
index ba249cee..a931ea63 100644
--- a/scripts/native-one-viewer-gate.ts
+++ b/scripts/native-one-viewer-gate.ts
@@ -60,6 +60,9 @@ interface GateReport {
interface SenderSnapshot {
getDisplayMediaRequested: boolean;
getDisplayMediaResolved: boolean;
+ nativeHardwareStarting: boolean;
+ nativeHardwareActive: boolean;
+ nativeAdapterLuid: string | null;
ready: number;
configAccepted: number;
encoderInstances: number;
@@ -77,6 +80,7 @@ interface SenderSnapshot {
maxPeers: number;
config: {
codec: string | null;
+ videoSource: string | null;
width: number | null;
height: number | null;
fps: number | null;
@@ -99,6 +103,13 @@ interface SenderSnapshot {
}>;
}
+type GateMode = "vp8" | "h264" | "native-h264";
+
+export function parseGateMode(value: string | undefined): GateMode {
+ const normalized = value?.trim();
+ return normalized === "h264" || normalized === "native-h264" ? normalized : "vp8";
+}
+
interface ViewerSnapshot {
signaling: {
authenticateSent: number;
@@ -229,16 +240,24 @@ function observeNodeStages(server: ScreenerServer, ledger: SenderStartLedger): v
});
}
-function senderStartObservation(snapshot: SenderSnapshot, expectedCodec = "vp8"): Partial<SenderStartObservation> {
+function senderStartObservation(
+ snapshot: SenderSnapshot,
+ expectedCodec = "vp8",
+ expectedVideoSource = "browser",
+): Partial<SenderStartObservation> {
+ const nativeWindow = expectedVideoSource === "native-window-h264";
return {
getDisplayMediaRequested: snapshot.getDisplayMediaRequested,
getDisplayMediaResolved: snapshot.getDisplayMediaResolved,
bridgeConnected: snapshot.ready > 0,
fixedHighConfigured: snapshot.configAccepted > 0 &&
- snapshot.config?.codec === expectedCodec && snapshot.config.width === 1280 &&
+ snapshot.config?.codec === expectedCodec && snapshot.config.videoSource === expectedVideoSource &&
+ snapshot.config.width === 1280 &&
snapshot.config.height === 720 && snapshot.config.fps === 30 &&
snapshot.config.bitrate === 3_000_000 && snapshot.config.encoderInstances === 1,
- firstEncodedChunk: snapshot.encoderOutputs > 0,
+ firstEncodedChunk: nativeWindow
+ ? snapshot.nativeHardwareActive && (snapshot.media?.framesWritten ?? 0) > 0
+ : snapshot.encoderOutputs > 0,
bridgeGeneration: snapshot.bridgeGeneration,
binarySendAttempts: snapshot.binarySendAttempts,
binarySendSucceeded: snapshot.binarySendSucceeded,
@@ -257,6 +276,28 @@ function senderStartObservation(snapshot: SenderSnapshot, expectedCodec = "vp8")
};
}
+function retainsExpectedBridge(snapshot: SenderSnapshot, nativeWindow: boolean): boolean {
+ return nativeWindow
+ ? snapshot.bridgeGeneration === 1 && snapshot.binarySendFailed === 0
+ : retainsFirstBridgeSend(snapshot);
+}
+
+function nativeSenderStartComplete(
+ snapshot: SenderSnapshot,
+ ledger: SenderStartObservation,
+): boolean {
+ return !snapshot.getDisplayMediaRequested && !snapshot.getDisplayMediaResolved &&
+ snapshot.nativeHardwareStarting && snapshot.nativeHardwareActive &&
+ typeof snapshot.nativeAdapterLuid === "string" && snapshot.nativeAdapterLuid.length > 0 &&
+ snapshot.encoderInstances === 0 && snapshot.encoderErrors === 0 && snapshot.fatalEvents === 0 &&
+ snapshot.config?.videoSource === "native-window-h264" &&
+ ledger.siteAccessAccepted && ledger.roomCreated && ledger.hostWssAuthenticated &&
+ ledger.bridgeConnected && ledger.fixedHighConfigured && ledger.firstEncodedChunk &&
+ ledger.bridgeGeneration === 1 && ledger.binarySendFailed === 0 &&
+ ledger.diagnosticsObserved && ledger.framesWritten > 0 &&
+ ledger.sourceRtpPacketsWritten > 0 && ledger.sourceRtpBytesWritten > 0;
+}
+
function requiredEnvironmentPath(name: string): string {
const value = process.env[name]?.trim();
if (!value) throw new Error(`${name} is required`);
@@ -276,8 +317,15 @@ function appendLedgerRecord(path: string, record: object): void {
async function main(): Promise<void> {
const chromePath = requiredEnvironmentPath("SCREENER_NATIVE_GATE_CHROME");
const goPath = process.env.SCREENER_NATIVE_GATE_GO?.trim() || "go";
- const gateCodec = process.env.SCREENER_NATIVE_GATE_CODEC?.trim() === "h264" ? "h264" : "vp8";
- const audioTargetTitle = process.env.SCREENER_NATIVE_GATE_AUDIO_TARGET_TITLE?.trim() || "";
+ const gateMode = parseGateMode(process.env.SCREENER_NATIVE_GATE_CODEC);
+ const nativeWindow = gateMode === "native-h264";
+ const gateCodec = gateMode === "vp8" ? "vp8" : "h264";
+ const gateVideoSource = nativeWindow ? "native-window-h264" : "browser";
+ const nativeHelperPath = nativeWindow
+ ? requiredEnvironmentPath("SCREENER_WINDOW_CAPTURE_HELPER")
+ : null;
+ const windowTargetTitle = process.env.SCREENER_NATIVE_GATE_WINDOW_TARGET_TITLE?.trim() ||
+ (nativeWindow ? "Native Gate Source" : "");
const accessKey = randomBytes(9).toString("base64url");
const report: GateReport = {
schemaVersion: 1,
@@ -285,9 +333,13 @@ async function main(): Promise<void> {
failedStage: "preflight",
stages: {},
limitations: [
- "The source is a real animated Chrome tab captured through getDisplayMedia, not a game workload.",
- "This run proves one WebCodecs object, not one physical or hardware encoder.",
- audioTargetTitle
+ nativeWindow
+ ? "The source is a real animated top-level Chrome window captured through WGC, not a game workload."
+ : "The source is a real animated Chrome tab captured through getDisplayMedia, not a game workload.",
+ nativeWindow
+ ? "Hardware status is the helper's hardware-only MF contract; physical VideoEncode use requires correlated OS telemetry."
+ : "This run proves one WebCodecs object, not one physical or hardware encoder.",
+ windowTargetTitle
? "The loopback run does not prove public STUN, restrictive networks, TURN, packaging, or endurance."
: "The loopback run does not prove public STUN, restrictive networks, TURN, audio, packaging, or endurance.",
],
@@ -323,7 +375,7 @@ async function main(): Promise<void> {
ALLOWED_ORIGINS: baseUrl,
SITE_ACCESS_PASSWORD: accessKey,
ROOM_DATABASE_PATH: "",
- PEER_ASSISTED_MEDIA: "false",
+ PEER_ASSISTED_MEDIA: "true",
STUN_URLS: "",
});
const roomStore = new ObservedRoomStore({
@@ -363,45 +415,66 @@ async function main(): Promise<void> {
cdp = await CdpConnection.connect(version.webSocketDebuggerUrl, Date.now() + 5_000);
report.stages.preflight = {
serverInMemory: config.roomDatabasePath === undefined,
- peerAssistedDisabled: !config.peerAssistedMedia,
+ peerAssistedEnabled: config.peerAssistedMedia,
oneViewerOnly: true,
+ videoSource: gateVideoSource,
+ nativeHelperConfigured: !nativeWindow || nativeHelperPath !== null,
chromeMajor: Number(version.Browser.match(/Chrome\/(\d+)/)?.[1] ?? 0),
};
report.failedStage = "sender-start";
- await createPage(cdp, animatedSourceUrl());
- const activeSenderPage = await createPage(cdp, launchUrl, senderProbe());
+ await createPage(cdp, animatedSourceUrl(), undefined, nativeWindow);
+ const activeSenderPage = await createPage(cdp, launchUrl, senderProbe(), nativeWindow);
senderPage = activeSenderPage;
- let audioTargetId = "";
- if (audioTargetTitle) {
+ let windowTargetId = "";
+ if (windowTargetTitle) {
await waitForSample(
(deadline) => evaluate<string>(cdp!, activeSenderPage,
- `([...document.querySelector('#audio-target').options].find((option) => option.text.includes(${JSON.stringify(audioTargetTitle)}))?.value || '')`, deadline),
+ `([...document.querySelector('#window-target').options].find((option) => option.text.includes(${JSON.stringify(windowTargetTitle)}))?.value || '')`, deadline),
Boolean,
5_000,
);
- audioTargetId = await evaluate<string>(cdp, activeSenderPage,
- `([...document.querySelector('#audio-target').options].find((option) => option.text.includes(${JSON.stringify(audioTargetTitle)}))?.value || '')`, Date.now() + 2_000);
+ windowTargetId = await evaluate<string>(cdp, activeSenderPage,
+ `([...document.querySelector('#window-target').options].find((option) => option.text.includes(${JSON.stringify(windowTargetTitle)}))?.value || '')`, Date.now() + 2_000);
}
await evaluate<void>(cdp, activeSenderPage, `(() => {
document.querySelector('#server-url').value = ${JSON.stringify(baseUrl)};
document.querySelector('#password').value = ${JSON.stringify(accessKey)};
- document.querySelector('#codec').value = ${JSON.stringify(gateCodec)};
- document.querySelector('#audio-target').value = ${JSON.stringify(audioTargetId)};
+ document.querySelector('#codec').value = ${JSON.stringify(gateMode)};
+ document.querySelector('#window-target').value = ${JSON.stringify(windowTargetId)};
document.querySelector('#start').click();
})()`, Date.now() + 5_000);
- await waitForSenderStart(
- async (deadline) => senderStartObservation(await senderSnapshot(cdp!, activeSenderPage, deadline), gateCodec),
- senderLedger,
- { timeoutMs: 20_000 },
- );
+ if (nativeWindow) {
+ await waitForSample(async (deadline) => {
+ const snapshot = await senderSnapshot(cdp!, activeSenderPage, deadline);
+ senderLedger.record(senderStartObservation(snapshot, gateCodec, gateVideoSource));
+ return { snapshot, ledger: senderLedger.snapshot() };
+ }, ({ snapshot, ledger }) => nativeSenderStartComplete(snapshot, ledger), 20_000);
+ } else {
+ await waitForSenderStart(
+ async (deadline) => senderStartObservation(
+ await senderSnapshot(cdp!, activeSenderPage, deadline), gateCodec, gateVideoSource,
+ ),
+ senderLedger,
+ { timeoutMs: 20_000 },
+ );
+ }
const senderReady = await senderSnapshot(cdp, activeSenderPage, Date.now() + 3_000);
const senderStartChecks = {
- oneEncoderObject: senderReady.encoderInstances === 1,
- currentBridge: retainsFirstBridgeSend(senderReady),
+ correctVideoSource: senderReady.config?.videoSource === gateVideoSource,
+ currentBridge: retainsExpectedBridge(senderReady, nativeWindow),
noCriticalErrors: senderReady.encoderErrors === 0 && senderReady.fatalEvents === 0,
+ ...(nativeWindow ? {
+ noBrowserCapture: !senderReady.getDisplayMediaRequested && !senderReady.getDisplayMediaResolved,
+ nativeHardwareStarting: senderReady.nativeHardwareStarting,
+ nativeHardwareActive: senderReady.nativeHardwareActive,
+ noBrowserVideoEncoder: senderReady.encoderInstances === 0,
+ } : { oneEncoderObject: senderReady.encoderInstances === 1 }),
+ };
+ report.stages["sender-start"] = {
+ ...senderLedger.snapshot(), ...senderStartChecks,
+ adapterLuid: nativeWindow ? senderReady.nativeAdapterLuid : null,
};
- report.stages["sender-start"] = { ...senderLedger.snapshot(), ...senderStartChecks };
requireChecks(senderStartChecks);
report.failedStage = "node-host";
const inviteUrl = await evaluate<string>(cdp, activeSenderPage,
@@ -435,7 +508,7 @@ async function main(): Promise<void> {
pionConnected: sender.peers.length === 1 && sender.peers[0]?.connectionState === "connected" &&
["connected", "completed"].includes(sender.peers[0]?.iceConnectionState ?? ""),
sameGeneration: identity !== null,
- sameBridge: retainsFirstBridgeSend(sender),
+ sameBridge: retainsExpectedBridge(sender, nativeWindow),
edgeBound: sender.maxPeers <= 1 && sender.peers.length === 1,
};
report.stages["viewer-signal"] = {
@@ -476,13 +549,17 @@ async function main(): Promise<void> {
videoReady: viewer.videoReadyState >= 2 && viewer.videoWidth > 0 && viewer.videoHeight > 0,
playbackClock: viewer.videoCurrentTime > beforeViewer.videoCurrentTime,
renderedFrames: viewer.renderedFrames > beforeViewer.renderedFrames,
- pionOutbound: (sender.peers[0]?.packetsSent ?? 0) > (beforeSender.peers[0]?.packetsSent ?? 0) &&
- (sender.peers[0]?.bytesSent ?? 0) > (beforeSender.peers[0]?.bytesSent ?? 0),
+ pionOutbound: (sender.peers[0]?.packetsSent ?? 0) > 0 &&
+ (sender.peers[0]?.bytesSent ?? 0) > 0,
sameGeneration: retainsOneViewerIdentity(identity!, viewer.identity, senderIdentity(sender)),
- sameBridge: retainsFirstBridgeSend(sender),
- oneEncoderObject: sender.encoderInstances === 1,
+ sameBridge: retainsExpectedBridge(sender, nativeWindow),
criticalErrors: sender.fatalEvents === 0 && sender.encoderErrors === 0,
- ...(audioTargetTitle ? {
+ ...(nativeWindow ? {
+ nativeHardwareActive: sender.nativeHardwareActive,
+ noBrowserCapture: !sender.getDisplayMediaRequested && !sender.getDisplayMediaResolved,
+ noBrowserVideoEncoder: sender.encoderInstances === 0,
+ } : { oneEncoderObject: sender.encoderInstances === 1 }),
+ ...(windowTargetTitle ? {
audioTrack: viewer.audioTrackCount === 1,
audioInbound: viewer.audioPacketsReceived > beforeViewer.audioPacketsReceived,
} : {}),
@@ -495,6 +572,7 @@ async function main(): Promise<void> {
renderedDelta: viewer.renderedFrames - beforeViewer.renderedFrames,
audioPacketDelta: viewer.audioPacketsReceived - beforeViewer.audioPacketsReceived,
pionPacketDelta: (sender.peers[0]?.packetsSent ?? 0) - (beforeSender.peers[0]?.packetsSent ?? 0),
+ pionPacketsSent: sender.peers[0]?.packetsSent ?? 0,
pcOrdinal: viewer.identity.pcOrdinal,
pionSlot: sender.peers[0]?.slot ?? -1,
pionEdgeGeneration: sender.peers[0]?.edgeGeneration ?? 0,
@@ -506,8 +584,13 @@ async function main(): Promise<void> {
}, 10_000);
const finalSender = await senderSnapshot(cdp, activeSenderPage, Date.now() + 3_000);
const finalChecks = {
- finalBridge: retainsFirstBridgeSend(finalSender),
+ finalBridge: retainsExpectedBridge(finalSender, nativeWindow),
finalCriticalErrors: hasNoCriticalSenderErrors(finalSender),
+ finalVideoSource: finalSender.config?.videoSource === gateVideoSource,
+ ...(nativeWindow ? {
+ finalNativeHardwareActive: finalSender.nativeHardwareActive,
+ finalNoBrowserVideoEncoder: finalSender.encoderInstances === 0,
+ } : {}),
};
report.stages["viewer-media"] = { ...report.stages["viewer-media"], ...finalChecks };
requireChecks(finalChecks);
@@ -522,7 +605,7 @@ async function main(): Promise<void> {
finalSenderEvidence = await verifyFinalSenderEvidence(
() => senderSnapshot(finalCdp, finalSenderPage, Date.now() + 1_000),
(snapshot) => {
- senderLedger.record(senderStartObservation(snapshot, gateCodec));
+ senderLedger.record(senderStartObservation(snapshot, gateCodec, gateVideoSource));
return senderLedger.snapshot();
},
);
@@ -544,6 +627,8 @@ async function main(): Promise<void> {
export function senderProbe(): string {
return `(() => {
const state = { getDisplayMediaRequested: false, getDisplayMediaResolved: false,
+ nativeHardwareStarting: false, nativeHardwareActive: false,
+ nativeAdapterLuid: null,
ready: 0, configAccepted: 0,
encoderInstances: 0, encoderOutputs: 0, encoderErrors: 0, fatalEvents: 0,
bridgeGeneration: 0, binarySendAttempts: 0, binarySendSucceeded: 0, binarySendFailed: 0,
@@ -580,6 +665,22 @@ export function senderProbe(): string {
if (message.kind === 'ready') state.ready += 1;
if (message.kind === 'config-accepted') state.configAccepted += 1;
if (message.kind === 'fatal') mark('fatalEvents');
+ if (message.kind === 'native-video-state' && message.status?.hardwareOnly === true) {
+ if (message.status.state === 'starting' && Number.isFinite(message.status.adapterIndex) &&
+ typeof message.status.adapterName === 'string' && message.status.adapterName.length > 0 &&
+ typeof message.status.adapterLuid === 'string' && message.status.adapterLuid.length > 0 &&
+ Number.isFinite(message.status.mftIndex) &&
+ typeof message.status.mftName === 'string' && message.status.mftName.length > 0 &&
+ typeof message.status.mftClsid === 'string' && message.status.mftClsid.length > 0) {
+ state.nativeHardwareStarting = true;
+ state.nativeAdapterLuid = message.status.adapterLuid;
+ }
+ if (message.status.state === 'active' && state.nativeHardwareStarting &&
+ message.status.profileLevelId === '42c01f' && message.status.width === 1280 &&
+ message.status.height === 720 && message.status.fps === 30) {
+ state.nativeHardwareActive = true;
+ }
+ }
if (message.kind === 'ready' || message.kind === 'viewer-count') {
state.activeViewers = finite(message.activeViewers);
}
@@ -627,6 +728,7 @@ export function senderProbe(): string {
let message; try { message = JSON.parse(value); } catch { message = null; }
if (message?.kind === 'config') state.config = {
codec: typeof message.codec === 'string' ? message.codec : null,
+ videoSource: typeof message.videoSource === 'string' ? message.videoSource : null,
width: finite(message.width), height: finite(message.height), fps: finite(message.fps),
bitrate: finite(message.bitrate), encoderInstances: finite(message.encoderInstances),
};
@@ -827,9 +929,16 @@ function viewerProbe(): string {
})();`;
}
-async function createPage(cdp: CdpConnection, url: string, probe?: string): Promise<PageHandle> {
+async function createPage(
+ cdp: CdpConnection,
+ url: string,
+ probe?: string,
+ newWindow = false,
+): Promise<PageHandle> {
const deadline = Date.now() + 15_000;
- const created = await cdp.call<{ targetId: string }>("Target.createTarget", { url: "about:blank", background: false }, undefined, deadline);
+ const created = await cdp.call<{ targetId: string }>(
+ "Target.createTarget", { url: "about:blank", background: false, newWindow }, undefined, deadline,
+ );
const attached = await cdp.call<{ sessionId: string }>("Target.attachToTarget", { targetId: created.targetId, flatten: true }, undefined, deadline);
const page = { sessionId: attached.sessionId };
await Promise.all([
@@ -1055,9 +1164,11 @@ async function removeGateProfile(profile: string): Promise<boolean> {
10_000, { ...process.env, SCREENER_GATE_PROFILE_TO_REMOVE: profile });
}
-function animatedSourceUrl(): string {
+export function animatedSourceUrl(): string {
const html = `<!doctype html><title>Native Gate Source</title><canvas width="1280" height="720"></canvas>
- <script>const c=document.querySelector('canvas'),x=c.getContext('2d');let f=0;
+ <script>const a=new AudioContext(),o=a.createOscillator(),g=a.createGain();g.gain.value=.02;
+ o.connect(g).connect(a.destination);o.start();
+ const c=document.querySelector('canvas'),x=c.getContext('2d');let f=0;
setInterval(()=>{f++;x.fillStyle='hsl('+f%360+' 80% 35%)';x.fillRect(0,0,1280,720);
x.fillStyle='white';x.fillRect(f*13%1280,0,20,720);x.font='64px monospace';x.fillText(f,40,90)},33);<\/script>`;
return `data:text/html;charset=utf-8,${encodeURIComponent(html)}`;
diff --git a/scripts/package-native-sender.ps1 b/scripts/package-native-sender.ps1
index 382e0597..551d5b21 100644
--- a/scripts/package-native-sender.ps1
+++ b/scripts/package-native-sender.ps1
@@ -68,9 +68,11 @@ try {
$env:CGO_ENABLED = $previousCGO
}
-& (Join-Path $repositoryRoot 'native\process-audio-helper\build.ps1') -OutputDirectory $packageDirectory | Out-Null
-$helperObject = Join-Path $packageDirectory 'main.obj'
-if (Test-Path -LiteralPath $helperObject) { Remove-Item -LiteralPath $helperObject -Force }
+& (Join-Path $repositoryRoot 'native\window-capture-helper\build.ps1') -OutputDirectory $packageDirectory | Out-Null
+@('window-capture.obj', 'process-audio.obj') | ForEach-Object {
+ $helperObject = Join-Path $packageDirectory $_
+ if (Test-Path -LiteralPath $helperObject) { Remove-Item -LiteralPath $helperObject -Force }
+}
$moduleManifest = @()
foreach ($line in $moduleLines) {
diff --git a/scripts/required-project-paths.txt b/scripts/required-project-paths.txt
index fcab6b50..fccf309b 100644
--- a/scripts/required-project-paths.txt
+++ b/scripts/required-project-paths.txt
@@ -25,6 +25,12 @@ docs/deployment.md
docs/adr/0001-p2p-first-media-topology.md
docs/research/agent-context-governance.md
docs/research/webrtc-p2p-screen-sharing.md
+native/sender/README.md
+native/window-capture-helper/README.md
+native/window-capture-helper/build.ps1
+native/window-capture-helper/main.cpp
+native/window-capture-helper/process_audio.cpp
+native/window-capture-helper/process_audio.h
scripts/required-project-paths.txt
scripts/check-project-state.sh
scripts/check-project-state.ps1
diff --git a/tests/native-one-viewer-gate-core.test.ts b/tests/native-one-viewer-gate-core.test.ts
index d7cb16fb..21d7a567 100644
--- a/tests/native-one-viewer-gate-core.test.ts
+++ b/tests/native-one-viewer-gate-core.test.ts
@@ -17,7 +17,12 @@ import {
type SenderIdentityEvidence,
type ViewerIdentityEvidence,
} from "../scripts/native-one-viewer-gate-core";
-import { profileCleanupScript, senderProbe } from "../scripts/native-one-viewer-gate";
+import {
+ animatedSourceUrl,
+ parseGateMode,
+ profileCleanupScript,
+ senderProbe,
+} from "../scripts/native-one-viewer-gate";
const viewer: ViewerIdentityEvidence = {
socketCount: 1, socketOrdinal: 1, authGeneration: 1,
@@ -33,6 +38,9 @@ const clean: CleanupResult = {
};
interface BridgeProbeSnapshot {
+ nativeHardwareStarting: boolean;
+ nativeHardwareActive: boolean;
+ nativeAdapterLuid: string | null;
bridgeGeneration: number;
binarySendAttempts: number;
binarySendSucceeded: number;
@@ -41,6 +49,8 @@ interface BridgeProbeSnapshot {
postSendDiagnosticsObserved: boolean;
postSendDiagnosticsSequence: number;
fatalEvents: number;
+ encoderInstances: number;
+ config: { codec: string; videoSource: string } | null;
}
class ProbeWebSocket {
@@ -79,6 +89,40 @@ function bridgeProbeContext(): {
}
describe("native one-viewer gate invariants", () => {
+ it("selects the explicit native H.264 source without changing its RTP codec", () => {
+ expect(parseGateMode("native-h264")).toBe("native-h264");
+ expect(parseGateMode("h264")).toBe("h264");
+ expect(parseGateMode("unsupported")).toBe("vp8");
+ });
+
+ it("keeps the fixed native target title and an audible process-tree source", () => {
+ const source = decodeURIComponent(animatedSourceUrl());
+ expect(source).toContain("<title>Native Gate Source</title>");
+ expect(source).toContain("createOscillator()");
+ });
+
+ it("retains the exact native hardware lifecycle without creating a browser encoder", () => {
+ const probe = bridgeProbeContext();
+ const socket = new probe.WebSocket("ws://127.0.0.1/media");
+ socket.send(JSON.stringify({ kind: "config", codec: "h264", videoSource: "native-window-h264",
+ width: 1280, height: 720, fps: 30, bitrate: 3_000_000, encoderInstances: 1 }));
+ socket.emitMessage(JSON.stringify({ kind: "native-video-state", status: {
+ state: "starting", hardwareOnly: true, adapterIndex: 0, adapterName: "GPU",
+ adapterLuid: "0:1", mftIndex: 0, mftName: "Hardware H264", mftClsid: "clsid",
+ } }));
+ socket.emitMessage(JSON.stringify({ kind: "native-video-state", status: {
+ state: "active", hardwareOnly: true, profileLevelId: "42c01f",
+ width: 1280, height: 720, fps: 30,
+ } }));
+ expect(probe.snapshot()).toMatchObject({
+ nativeHardwareStarting: true,
+ nativeHardwareActive: true,
+ nativeAdapterLuid: "0:1",
+ encoderInstances: 0,
+ config: { codec: "h264", videoSource: "native-window-h264" },
+ });
+ });
+
it("rejects cumulative evidence from a rebuilt Viewer PC", () => {
const baseline = captureOneViewerIdentity(viewer, sender);
expect(baseline).not.toBeNull();