Open source Grok Bot alternative, Bring your own agent and subscription: message Claude Code, Codex, Gemini CLI, Cursor, Copilot and 40+ AI agents as bots on your own computer, from iPhone, Mac, Windows, Linux or terminal. Agent-agnostic and free. Also a Muse and Dots alternative. Built in Rust.
Rust acpagent-client-protocolai-agentsai-assistantclaudeclaude-codecodexcoding-agentscursorgeminigemini-cligrok-bot-alternativeiosiphonelinuxmacosmcpopen-sourcerustself-hosted
fix(screen): negotiate Linux capture video (#104)
Sean Fergusondc1794e74638
diff --git a/.github/workflows/screen-linux.yml b/.github/workflows/screen-linux.ymlindex ba36b626..c4c507ae 100644--- a/.github/workflows/screen-linux.yml+++ b/.github/workflows/screen-linux.yml@@ -21,7 +21,7 @@ jobs:timeout-minutes: 10run: |sudo apt-get -o Acquire::Retries=3 -o Acquire::http::Timeout=60 -o Acquire::https::Timeout=60 update- sudo apt-get -o Acquire::Retries=3 -o Acquire::http::Timeout=60 -o Acquire::https::Timeout=60 install -y --no-install-recommends libgstreamer-plugins-base1.0-dev libgstreamer-plugins-bad1.0-dev+ sudo apt-get -o Acquire::Retries=3 -o Acquire::http::Timeout=60 -o Acquire::https::Timeout=60 install -y --no-install-recommends libgstreamer-plugins-base1.0-dev libgstreamer-plugins-bad1.0-dev gstreamer1.0-plugins-base gstreamer1.0-plugins-good gstreamer1.0-plugins-bad gstreamer1.0-plugins-ugly gstreamer1.0-nice gstreamer1.0-pipewire- name: Install Rust (rust-toolchain.toml)run: rustup toolchain install- uses: Swatinem/rust-cache@v2diff --git a/apps/screen-linux/src/main.rs b/apps/screen-linux/src/main.rsindex 0aeec297..5229d5e6 100644--- a/apps/screen-linux/src/main.rs+++ b/apps/screen-linux/src/main.rs@@ -9,6 +9,7 @@ mod a11y;mod input;mod portal;mod stream;+mod video;use anyhow::{Context, Result, anyhow};use serde_json::{Value, json};diff --git a/apps/screen-linux/src/stream.rs b/apps/screen-linux/src/stream.rsindex 0701cd57..21180562 100644--- a/apps/screen-linux/src/stream.rs+++ b/apps/screen-linux/src/stream.rs@@ -2,6 +2,7 @@//! `webrtcbin` (non-trickle, like the macOS helper), plus one-off JPEG stills.use crate::portal::Display;+use crate::video::{h264_format, h264_payload};use crate::{Helper, input};use anyhow::{Context, Result, anyhow, bail};use base64::Engine as _;@@ -12,6 +13,22 @@ use std::sync::{Arc, Mutex};use std::time::Duration;use tokio::sync::{mpsc, oneshot};+/// The PipeWire source for one portal stream. Only properties every supported PipeWire has+/// (Ubuntu 24.04's 1.0 onward) go in the builder: an unknown one panics there.+fn capture_source(fd: OwnedFd, node: u32) -> Result<gst::Element> {+ let src = gst::ElementFactory::make("pipewiresrc")+ .property("fd", fd.into_raw_fd())+ .property("path", node.to_string())+ .property("do-timestamp", true)+ .build()+ .context("GStreamer's PipeWire plugin (gstreamer1.0-pipewire) is missing")?;+ if src.has_property("keepalive-time") {+ // A still screen keeps sending frames now and then, so keyframe requests get answered.+ src.set_property("keepalive-time", 1000i32);+ }+ Ok(src)+}+/// Hardware encoders first; x264 is the always-there fallback.const ENCODERS: &[&str] = &["vah264lpenc",@@ -36,17 +53,12 @@ impl Session {helper: Arc<Helper>,config: &Value,) -> Result<Self> {+ let payload = h264_payload(config["sdp"].as_str().context("missing offer")?)?;let pipeline = gst::Pipeline::new();- let src = gst::ElementFactory::make("pipewiresrc")- .property("fd", fd.into_raw_fd())- .property("path", display.node.to_string())- .property("do-timestamp", true)- .build()- .context("GStreamer's PipeWire plugin (gstreamer1.0-pipewire) is missing")?;- if src.has_property("keepalive-time") {- // A still screen keeps sending frames now and then, so keyframe requests get answered.- src.set_property("keepalive-time", 1000i32);- }+ // Capture and keepalive timestamps use the system clock. pipewiresrc only gained a+ // switch for this (`provide-clock`) in PipeWire 1.6; Ubuntu 24.04 ships 1.0.+ pipeline.use_clock(Some(&gst::SystemClock::obtain()));+ let src = capture_source(fd, display.node)?;let convert = make("videoconvert")?;let crop = make("videocrop")?;let scale = make("videoscale")?;@@ -56,6 +68,7 @@ impl Session {rate.set_property("max-rate", i32::try_from(fps)?);rate.set_property("drop-only", true);let convert2 = make("videoconvert")?;+ let format = h264_format()?;let queue = make("queue")?;queue.set_property_from_str("leaky", "downstream");queue.set_property("max-size-buffers", 2u32);@@ -68,14 +81,14 @@ impl Session {let pay = make("rtph264pay")?;pay.set_property("config-interval", -1i32);pay.set_property_from_str("aggregate-mode", "zero-latency");- pay.set_property("pt", 96u32);+ pay.set_property("pt", u32::try_from(payload)?);let rtp_caps = make("capsfilter")?;rtp_caps.set_property("caps",gst::Caps::builder("application/x-rtp").field("media", "video").field("encoding-name", "H264")- .field("payload", 96i32)+ .field("payload", payload).field("clock-rate", 90000i32).build(),);@@ -99,8 +112,8 @@ impl Session {}}let chain = [- &src, &convert, &crop, &scale, &size, &rate, &convert2, &queue, &encoder, &parse, &pay,- &rtp_caps, &webrtc,+ &src, &convert, &crop, &scale, &size, &rate, &convert2, &format, &queue, &encoder,+ &parse, &pay, &rtp_caps, &webrtc,];pipeline.add_many(chain)?;gst::Element::link_many(chain)?;@@ -176,8 +189,8 @@ impl Session {}});pipeline- .set_state(gst::State::Playing)- .context("starting the video pipeline")?;+ .set_state(gst::State::Ready)+ .context("preparing the video pipeline")?;Ok(Self { pipeline, webrtc })}@@ -194,7 +207,8 @@ impl Session {let (p, done) = promise();self.webrtc.emit_by_name::<()>("create-answer", &[&None::<gst::Structure>, &p]);- let reply = done.await?.ok_or_else(|| anyhow!("no answer"))?;+ let reply = done.await?;+ let reply = reply.ok_or_else(|| anyhow!("no answer"))?;let answer = reply.get::<gst_webrtc::WebRTCSessionDescription>("answer").map_err(|_| anyhow!("no answer"))?;@@ -202,6 +216,10 @@ impl Session {self.webrtc.emit_by_name::<()>("set-local-description", &[&answer, &p]);done.await?;+ // Negotiate the profile before the encoder fixes its output caps.+ self.pipeline+ .set_state(gst::State::Playing)+ .context("starting the video pipeline")?;for _ in 0..200 {if self.webrtc@@ -456,3 +474,7 @@ mod ice_tests {assert!(ice_uri("turn:turn.cloudflare.com:3478", None, None).is_err());}}++#[cfg(test)]+#[path = "video_tests.rs"]+mod video_tests;diff --git a/apps/screen-linux/src/video.rs b/apps/screen-linux/src/video.rsnew file mode 100644index 00000000..e472b61a--- /dev/null+++ b/apps/screen-linux/src/video.rs@@ -0,0 +1,42 @@+//! Linux capture format and offered H.264 payload selection.++use anyhow::{Result, anyhow, bail};++/// The answer must use the phone's payload number, which is not necessarily 96.+pub(super) fn h264_payload(offer: &str) -> Result<i32> {+ let sdp =+ gst_sdp::SDPMessage::parse_buffer(offer.as_bytes()).map_err(|_| anyhow!("bad offer"))?;+ for media in sdp.medias().filter(|m| m.media() == Some("video")) {+ for format in media.formats() {+ // rtph264pay only takes dynamic payload numbers.+ let Ok(payload @ 96..=127) = format.parse::<i32>() else {+ continue;+ };+ let Some(caps) = media.caps_from_media(payload) else {+ continue;+ };+ let Some(codec) = caps.structure(0) else {+ continue;+ };+ if codec.get::<&str>("encoding-name").ok() == Some("H264")+ && codec.get::<i32>("clock-rate").ok() == Some(90_000)+ && codec.get::<&str>("packetization-mode").ok() == Some("1")+ {+ return Ok(payload);+ }+ }+ }+ bail!("the phone did not offer packetized H.264 video")+}++/// WebRTC Baseline requires 8-bit 4:2:0; RGB capture must not select 4:4:4.+pub(super) fn h264_format() -> Result<gst::Element> {+ Ok(gst::ElementFactory::make("capsfilter")+ .property(+ "caps",+ gst::Caps::builder("video/x-raw")+ .field("format", gst::List::new(["I420", "NV12"]))+ .build(),+ )+ .build()?)+}diff --git a/apps/screen-linux/src/video_tests.rs b/apps/screen-linux/src/video_tests.rsnew file mode 100644index 00000000..eb3e3c9b--- /dev/null+++ b/apps/screen-linux/src/video_tests.rs@@ -0,0 +1,109 @@+#![allow(clippy::unwrap_used)]++use super::*;++#[test]+fn video_uses_the_offered_h264_payload_instead_of_a_fixed_number() {+ gst::init().unwrap();+ let offer = "v=0\r\no=- 0 0 IN IP4 127.0.0.1\r\ns=-\r\nt=0 0\r\n\+ m=audio 9 UDP/TLS/RTP/SAVPF 99\r\na=rtpmap:99 H264/90000\r\n\+ a=fmtp:99 packetization-mode=1\r\n\+ m=video 9 UDP/TLS/RTP/SAVPF 96 100 102\r\n\+ a=rtpmap:96 VP8/90000\r\na=rtpmap:100 H264/90000\r\n\+ a=fmtp:100 packetization-mode=0\r\na=rtpmap:102 H264/90000\r\n\+ a=fmtp:102 packetization-mode=1;profile-level-id=42e01f\r\n";+ assert_eq!(h264_payload(offer).unwrap(), 102);+ assert!(h264_payload(&offer.replace("packetization-mode=1", "packetization-mode=0")).is_err());+}++#[tokio::test]+async fn baseline_offer_starts_video_after_the_answer_is_negotiated() {+ gst::init().unwrap();+ let pipeline = gst::parse::launch(+ "videotestsrc is-live=true ! videoconvert ! video/x-raw,format=I420 \+ ! x264enc tune=zerolatency ! h264parse ! rtph264pay pt=102 \+ ! application/x-rtp,media=video,encoding-name=H264,payload=102,clock-rate=90000 \+ ! webrtcbin name=peer",+ )+ .unwrap()+ .downcast::<gst::Pipeline>()+ .unwrap();+ let webrtc = pipeline.by_name("peer").unwrap();+ pipeline.set_state(gst::State::Ready).unwrap();+ let session = Session { pipeline, webrtc };+ let offer = "v=0\r\no=- 0 0 IN IP4 127.0.0.1\r\ns=-\r\nt=0 0\r\n\+ m=video 9 UDP/TLS/RTP/SAVPF 102\r\nc=IN IP4 0.0.0.0\r\n\+ a=mid:0\r\na=recvonly\r\na=rtcp-mux\r\na=setup:actpass\r\n\+ a=ice-ufrag:test\r\na=ice-pwd:testpasswordforthevideooffer\r\n\+ a=fingerprint:sha-256 00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00\r\n\+ a=rtpmap:102 H264/90000\r\n\+ a=fmtp:102 packetization-mode=1;profile-level-id=42e01f\r\n";+ let answer_result =+ tokio::time::timeout(std::time::Duration::from_secs(12), session.answer(offer)).await;+ let answer = answer_result+ .expect("complete SDP must finish gathering")+ .unwrap();+ let sdp = gst_sdp::SDPMessage::parse_buffer(answer.as_bytes()).unwrap();+ let video = sdp.media(0).unwrap();+ assert_ne!(video.port(), 0, "the H.264 video offer must be accepted");+ assert_eq!(video.formats().collect::<Vec<_>>(), ["102"]);+ assert!(video.attributes().any(|a| a.key() == "sendonly"));+ assert_eq!(session.pipeline.current_state(), gst::State::Playing);+ assert_eq!(+ session+ .webrtc+ .property::<gst_webrtc::WebRTCICEGatheringState>("ice-gathering-state"),+ gst_webrtc::WebRTCICEGatheringState::Complete,+ );+}++#[test]+fn capture_encodes_as_eight_bit_420_before_webrtc_negotiates() {+ gst::init().unwrap();+ let pipeline = gst::Pipeline::new();+ let source = gst::ElementFactory::make("videotestsrc")+ .property("num-buffers", 1i32)+ .build()+ .unwrap();+ let rgb = gst::ElementFactory::make("capsfilter")+ .property(+ "caps",+ gst::Caps::builder("video/x-raw")+ .field("format", "RGB")+ .build(),+ )+ .build()+ .unwrap();+ let convert = make("videoconvert").unwrap();+ let format = h264_format().unwrap();+ let encoder = encoder(1000).unwrap();+ let parse = make("h264parse").unwrap();+ let sink = gst_app::AppSink::builder().build();+ let chain = [+ &source,+ &rgb,+ &convert,+ &format,+ &encoder,+ &parse,+ sink.upcast_ref(),+ ];+ pipeline.add_many(chain).unwrap();+ gst::Element::link_many(chain).unwrap();+ pipeline.set_state(gst::State::Playing).unwrap();+ let sample = sink.try_pull_sample(gst::ClockTime::from_seconds(5));+ pipeline.set_state(gst::State::Null).unwrap();+ let sample = sample.expect("RGB capture should produce an H.264 frame");+ let caps = sample.caps().unwrap();+ let h264_caps = caps.structure(0).unwrap();+ assert_eq!(h264_caps.get::<String>("chroma-format").unwrap(), "4:2:0");+ assert_eq!(h264_caps.get::<u32>("bit-depth-luma").unwrap(), 8);+ assert_eq!(h264_caps.get::<u32>("bit-depth-chroma").unwrap(), 8);+}++#[test]+fn capture_source_builds_on_the_installed_pipewire() {+ gst::init().unwrap();+ let fd = std::fs::File::open("/dev/null").unwrap();+ capture_source(fd.into(), 0).unwrap();+}diff --git a/docs/features/remote-screen.md b/docs/features/remote-screen.mdindex 5c1e484f..d77aa49c 100644--- a/docs/features/remote-screen.md+++ b/docs/features/remote-screen.md@@ -33,6 +33,7 @@ computer display without choosing the phone's orientation.- `apps/screen-macos/` is the macOS capture/input helper (Xcode `Screen` target, `CodyncScreen.app`). The desktop app bundles it and registers it through `SMAppService` (`apps/desktop/src/main/screen.ts`); it is responsible for the OS permissions, including those of the computer-use driver it starts.- `apps/screen-linux/` implements the Linux helper using desktop portals and GStreamer, including ICE URL conversion and TURN transport configuration. The host starts it from beside its own executable (then `PATH`). Linux host releases ship `codync-screen` in the same archive (built on Ubuntu 24.04: glibc 2.39+ and GStreamer 1.22+ with the base, good and bad plugins, PipeWire and xdg-desktop-portal at run time); `install.sh`, Homebrew and `codync-host update` put it next to the host.- Helpers communicate locally through `~/.codync/screen.sock`. SDP is non-trickle; input uses the `input` and `input-fast` data channels.+- The Linux helper answers with the packetized H.264 payload the phone offers, feeds the encoder 8-bit 4:2:0 video and timestamps capture on the system clock. It answers before capture starts, so the profile is negotiated before the encoder fixes its output.- Screen access is on by default on macOS and on Linux while a graphical session (Wayland or X11) is running; headless servers stay off. Turning it on or off is remembered. Enabling through `setScreenEnabled` requires a loopback caller. Interactive OS permission prompts must be completed on the computer.- Bots with their computer capability enabled receive the built-in `computer` MCP tools, run by the computer-use driver that Codync Screen starts on macOS ([computer use](computer-use.md)). Their permission policy still applies. An interactive phone can take over; bots may still look.- Each device may hold four sessions; the host allows 32 total, including pending sessions. Credential issuance is limited to 12 requests per minute per account by `TURN_LIMITER` (Cloudflare's per-location rate limiter).