Skip to content

Open source Grok Bot alternative, Bring your own agent and subscription: message Claude Code, Codex, Gemini CLI, Cursor, Copilot and 40+ AI agents as bots on your own computer, from iPhone, Mac, Windows, Linux or terminal. Agent-agnostic and free. Also a Muse and Dots alternative. Built in Rust.

fix(screen): negotiate Linux capture video (#104)

Sean Fergusondc1794e74638
diff --git a/.github/workflows/screen-linux.yml b/.github/workflows/screen-linux.yml
index ba36b626..c4c507ae 100644
--- a/.github/workflows/screen-linux.yml
+++ b/.github/workflows/screen-linux.yml
@@ -21,7 +21,7 @@ jobs:
timeout-minutes: 10
run: |
sudo apt-get -o Acquire::Retries=3 -o Acquire::http::Timeout=60 -o Acquire::https::Timeout=60 update
- sudo apt-get -o Acquire::Retries=3 -o Acquire::http::Timeout=60 -o Acquire::https::Timeout=60 install -y --no-install-recommends libgstreamer-plugins-base1.0-dev libgstreamer-plugins-bad1.0-dev
+ sudo apt-get -o Acquire::Retries=3 -o Acquire::http::Timeout=60 -o Acquire::https::Timeout=60 install -y --no-install-recommends libgstreamer-plugins-base1.0-dev libgstreamer-plugins-bad1.0-dev gstreamer1.0-plugins-base gstreamer1.0-plugins-good gstreamer1.0-plugins-bad gstreamer1.0-plugins-ugly gstreamer1.0-nice gstreamer1.0-pipewire
- name: Install Rust (rust-toolchain.toml)
run: rustup toolchain install
- uses: Swatinem/rust-cache@v2
diff --git a/apps/screen-linux/src/main.rs b/apps/screen-linux/src/main.rs
index 0aeec297..5229d5e6 100644
--- a/apps/screen-linux/src/main.rs
+++ b/apps/screen-linux/src/main.rs
@@ -9,6 +9,7 @@ mod a11y;
mod input;
mod portal;
mod stream;
+mod video;
use anyhow::{Context, Result, anyhow};
use serde_json::{Value, json};
diff --git a/apps/screen-linux/src/stream.rs b/apps/screen-linux/src/stream.rs
index 0701cd57..21180562 100644
--- a/apps/screen-linux/src/stream.rs
+++ b/apps/screen-linux/src/stream.rs
@@ -2,6 +2,7 @@
//! `webrtcbin` (non-trickle, like the macOS helper), plus one-off JPEG stills.
use crate::portal::Display;
+use crate::video::{h264_format, h264_payload};
use crate::{Helper, input};
use anyhow::{Context, Result, anyhow, bail};
use base64::Engine as _;
@@ -12,6 +13,22 @@ use std::sync::{Arc, Mutex};
use std::time::Duration;
use tokio::sync::{mpsc, oneshot};
+/// The PipeWire source for one portal stream. Only properties every supported PipeWire has
+/// (Ubuntu 24.04's 1.0 onward) go in the builder: an unknown one panics there.
+fn capture_source(fd: OwnedFd, node: u32) -> Result<gst::Element> {
+ let src = gst::ElementFactory::make("pipewiresrc")
+ .property("fd", fd.into_raw_fd())
+ .property("path", node.to_string())
+ .property("do-timestamp", true)
+ .build()
+ .context("GStreamer's PipeWire plugin (gstreamer1.0-pipewire) is missing")?;
+ if src.has_property("keepalive-time") {
+ // A still screen keeps sending frames now and then, so keyframe requests get answered.
+ src.set_property("keepalive-time", 1000i32);
+ }
+ Ok(src)
+}
+
/// Hardware encoders first; x264 is the always-there fallback.
const ENCODERS: &[&str] = &[
"vah264lpenc",
@@ -36,17 +53,12 @@ impl Session {
helper: Arc<Helper>,
config: &Value,
) -> Result<Self> {
+ let payload = h264_payload(config["sdp"].as_str().context("missing offer")?)?;
let pipeline = gst::Pipeline::new();
- let src = gst::ElementFactory::make("pipewiresrc")
- .property("fd", fd.into_raw_fd())
- .property("path", display.node.to_string())
- .property("do-timestamp", true)
- .build()
- .context("GStreamer's PipeWire plugin (gstreamer1.0-pipewire) is missing")?;
- if src.has_property("keepalive-time") {
- // A still screen keeps sending frames now and then, so keyframe requests get answered.
- src.set_property("keepalive-time", 1000i32);
- }
+ // Capture and keepalive timestamps use the system clock. pipewiresrc only gained a
+ // switch for this (`provide-clock`) in PipeWire 1.6; Ubuntu 24.04 ships 1.0.
+ pipeline.use_clock(Some(&gst::SystemClock::obtain()));
+ let src = capture_source(fd, display.node)?;
let convert = make("videoconvert")?;
let crop = make("videocrop")?;
let scale = make("videoscale")?;
@@ -56,6 +68,7 @@ impl Session {
rate.set_property("max-rate", i32::try_from(fps)?);
rate.set_property("drop-only", true);
let convert2 = make("videoconvert")?;
+ let format = h264_format()?;
let queue = make("queue")?;
queue.set_property_from_str("leaky", "downstream");
queue.set_property("max-size-buffers", 2u32);
@@ -68,14 +81,14 @@ impl Session {
let pay = make("rtph264pay")?;
pay.set_property("config-interval", -1i32);
pay.set_property_from_str("aggregate-mode", "zero-latency");
- pay.set_property("pt", 96u32);
+ pay.set_property("pt", u32::try_from(payload)?);
let rtp_caps = make("capsfilter")?;
rtp_caps.set_property(
"caps",
gst::Caps::builder("application/x-rtp")
.field("media", "video")
.field("encoding-name", "H264")
- .field("payload", 96i32)
+ .field("payload", payload)
.field("clock-rate", 90000i32)
.build(),
);
@@ -99,8 +112,8 @@ impl Session {
}
}
let chain = [
- &src, &convert, &crop, &scale, &size, &rate, &convert2, &queue, &encoder, &parse, &pay,
- &rtp_caps, &webrtc,
+ &src, &convert, &crop, &scale, &size, &rate, &convert2, &format, &queue, &encoder,
+ &parse, &pay, &rtp_caps, &webrtc,
];
pipeline.add_many(chain)?;
gst::Element::link_many(chain)?;
@@ -176,8 +189,8 @@ impl Session {
}
});
pipeline
- .set_state(gst::State::Playing)
- .context("starting the video pipeline")?;
+ .set_state(gst::State::Ready)
+ .context("preparing the video pipeline")?;
Ok(Self { pipeline, webrtc })
}
@@ -194,7 +207,8 @@ impl Session {
let (p, done) = promise();
self.webrtc
.emit_by_name::<()>("create-answer", &[&None::<gst::Structure>, &p]);
- let reply = done.await?.ok_or_else(|| anyhow!("no answer"))?;
+ let reply = done.await?;
+ let reply = reply.ok_or_else(|| anyhow!("no answer"))?;
let answer = reply
.get::<gst_webrtc::WebRTCSessionDescription>("answer")
.map_err(|_| anyhow!("no answer"))?;
@@ -202,6 +216,10 @@ impl Session {
self.webrtc
.emit_by_name::<()>("set-local-description", &[&answer, &p]);
done.await?;
+ // Negotiate the profile before the encoder fixes its output caps.
+ self.pipeline
+ .set_state(gst::State::Playing)
+ .context("starting the video pipeline")?;
for _ in 0..200 {
if self
.webrtc
@@ -456,3 +474,7 @@ mod ice_tests {
assert!(ice_uri("turn:turn.cloudflare.com:3478", None, None).is_err());
}
}
+
+#[cfg(test)]
+#[path = "video_tests.rs"]
+mod video_tests;
diff --git a/apps/screen-linux/src/video.rs b/apps/screen-linux/src/video.rs
new file mode 100644
index 00000000..e472b61a
--- /dev/null
+++ b/apps/screen-linux/src/video.rs
@@ -0,0 +1,42 @@
+//! Linux capture format and offered H.264 payload selection.
+
+use anyhow::{Result, anyhow, bail};
+
+/// The answer must use the phone's payload number, which is not necessarily 96.
+pub(super) fn h264_payload(offer: &str) -> Result<i32> {
+ let sdp =
+ gst_sdp::SDPMessage::parse_buffer(offer.as_bytes()).map_err(|_| anyhow!("bad offer"))?;
+ for media in sdp.medias().filter(|m| m.media() == Some("video")) {
+ for format in media.formats() {
+ // rtph264pay only takes dynamic payload numbers.
+ let Ok(payload @ 96..=127) = format.parse::<i32>() else {
+ continue;
+ };
+ let Some(caps) = media.caps_from_media(payload) else {
+ continue;
+ };
+ let Some(codec) = caps.structure(0) else {
+ continue;
+ };
+ if codec.get::<&str>("encoding-name").ok() == Some("H264")
+ && codec.get::<i32>("clock-rate").ok() == Some(90_000)
+ && codec.get::<&str>("packetization-mode").ok() == Some("1")
+ {
+ return Ok(payload);
+ }
+ }
+ }
+ bail!("the phone did not offer packetized H.264 video")
+}
+
+/// WebRTC Baseline requires 8-bit 4:2:0; RGB capture must not select 4:4:4.
+pub(super) fn h264_format() -> Result<gst::Element> {
+ Ok(gst::ElementFactory::make("capsfilter")
+ .property(
+ "caps",
+ gst::Caps::builder("video/x-raw")
+ .field("format", gst::List::new(["I420", "NV12"]))
+ .build(),
+ )
+ .build()?)
+}
diff --git a/apps/screen-linux/src/video_tests.rs b/apps/screen-linux/src/video_tests.rs
new file mode 100644
index 00000000..eb3e3c9b
--- /dev/null
+++ b/apps/screen-linux/src/video_tests.rs
@@ -0,0 +1,109 @@
+#![allow(clippy::unwrap_used)]
+
+use super::*;
+
+#[test]
+fn video_uses_the_offered_h264_payload_instead_of_a_fixed_number() {
+ gst::init().unwrap();
+ let offer = "v=0\r\no=- 0 0 IN IP4 127.0.0.1\r\ns=-\r\nt=0 0\r\n\
+ m=audio 9 UDP/TLS/RTP/SAVPF 99\r\na=rtpmap:99 H264/90000\r\n\
+ a=fmtp:99 packetization-mode=1\r\n\
+ m=video 9 UDP/TLS/RTP/SAVPF 96 100 102\r\n\
+ a=rtpmap:96 VP8/90000\r\na=rtpmap:100 H264/90000\r\n\
+ a=fmtp:100 packetization-mode=0\r\na=rtpmap:102 H264/90000\r\n\
+ a=fmtp:102 packetization-mode=1;profile-level-id=42e01f\r\n";
+ assert_eq!(h264_payload(offer).unwrap(), 102);
+ assert!(h264_payload(&offer.replace("packetization-mode=1", "packetization-mode=0")).is_err());
+}
+
+#[tokio::test]
+async fn baseline_offer_starts_video_after_the_answer_is_negotiated() {
+ gst::init().unwrap();
+ let pipeline = gst::parse::launch(
+ "videotestsrc is-live=true ! videoconvert ! video/x-raw,format=I420 \
+ ! x264enc tune=zerolatency ! h264parse ! rtph264pay pt=102 \
+ ! application/x-rtp,media=video,encoding-name=H264,payload=102,clock-rate=90000 \
+ ! webrtcbin name=peer",
+ )
+ .unwrap()
+ .downcast::<gst::Pipeline>()
+ .unwrap();
+ let webrtc = pipeline.by_name("peer").unwrap();
+ pipeline.set_state(gst::State::Ready).unwrap();
+ let session = Session { pipeline, webrtc };
+ let offer = "v=0\r\no=- 0 0 IN IP4 127.0.0.1\r\ns=-\r\nt=0 0\r\n\
+ m=video 9 UDP/TLS/RTP/SAVPF 102\r\nc=IN IP4 0.0.0.0\r\n\
+ a=mid:0\r\na=recvonly\r\na=rtcp-mux\r\na=setup:actpass\r\n\
+ a=ice-ufrag:test\r\na=ice-pwd:testpasswordforthevideooffer\r\n\
+ a=fingerprint:sha-256 00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00\r\n\
+ a=rtpmap:102 H264/90000\r\n\
+ a=fmtp:102 packetization-mode=1;profile-level-id=42e01f\r\n";
+ let answer_result =
+ tokio::time::timeout(std::time::Duration::from_secs(12), session.answer(offer)).await;
+ let answer = answer_result
+ .expect("complete SDP must finish gathering")
+ .unwrap();
+ let sdp = gst_sdp::SDPMessage::parse_buffer(answer.as_bytes()).unwrap();
+ let video = sdp.media(0).unwrap();
+ assert_ne!(video.port(), 0, "the H.264 video offer must be accepted");
+ assert_eq!(video.formats().collect::<Vec<_>>(), ["102"]);
+ assert!(video.attributes().any(|a| a.key() == "sendonly"));
+ assert_eq!(session.pipeline.current_state(), gst::State::Playing);
+ assert_eq!(
+ session
+ .webrtc
+ .property::<gst_webrtc::WebRTCICEGatheringState>("ice-gathering-state"),
+ gst_webrtc::WebRTCICEGatheringState::Complete,
+ );
+}
+
+#[test]
+fn capture_encodes_as_eight_bit_420_before_webrtc_negotiates() {
+ gst::init().unwrap();
+ let pipeline = gst::Pipeline::new();
+ let source = gst::ElementFactory::make("videotestsrc")
+ .property("num-buffers", 1i32)
+ .build()
+ .unwrap();
+ let rgb = gst::ElementFactory::make("capsfilter")
+ .property(
+ "caps",
+ gst::Caps::builder("video/x-raw")
+ .field("format", "RGB")
+ .build(),
+ )
+ .build()
+ .unwrap();
+ let convert = make("videoconvert").unwrap();
+ let format = h264_format().unwrap();
+ let encoder = encoder(1000).unwrap();
+ let parse = make("h264parse").unwrap();
+ let sink = gst_app::AppSink::builder().build();
+ let chain = [
+ &source,
+ &rgb,
+ &convert,
+ &format,
+ &encoder,
+ &parse,
+ sink.upcast_ref(),
+ ];
+ pipeline.add_many(chain).unwrap();
+ gst::Element::link_many(chain).unwrap();
+ pipeline.set_state(gst::State::Playing).unwrap();
+ let sample = sink.try_pull_sample(gst::ClockTime::from_seconds(5));
+ pipeline.set_state(gst::State::Null).unwrap();
+ let sample = sample.expect("RGB capture should produce an H.264 frame");
+ let caps = sample.caps().unwrap();
+ let h264_caps = caps.structure(0).unwrap();
+ assert_eq!(h264_caps.get::<String>("chroma-format").unwrap(), "4:2:0");
+ assert_eq!(h264_caps.get::<u32>("bit-depth-luma").unwrap(), 8);
+ assert_eq!(h264_caps.get::<u32>("bit-depth-chroma").unwrap(), 8);
+}
+
+#[test]
+fn capture_source_builds_on_the_installed_pipewire() {
+ gst::init().unwrap();
+ let fd = std::fs::File::open("/dev/null").unwrap();
+ capture_source(fd.into(), 0).unwrap();
+}
diff --git a/docs/features/remote-screen.md b/docs/features/remote-screen.md
index 5c1e484f..d77aa49c 100644
--- a/docs/features/remote-screen.md
+++ b/docs/features/remote-screen.md
@@ -33,6 +33,7 @@ computer display without choosing the phone's orientation.
- `apps/screen-macos/` is the macOS capture/input helper (Xcode `Screen` target, `CodyncScreen.app`). The desktop app bundles it and registers it through `SMAppService` (`apps/desktop/src/main/screen.ts`); it is responsible for the OS permissions, including those of the computer-use driver it starts.
- `apps/screen-linux/` implements the Linux helper using desktop portals and GStreamer, including ICE URL conversion and TURN transport configuration. The host starts it from beside its own executable (then `PATH`). Linux host releases ship `codync-screen` in the same archive (built on Ubuntu 24.04: glibc 2.39+ and GStreamer 1.22+ with the base, good and bad plugins, PipeWire and xdg-desktop-portal at run time); `install.sh`, Homebrew and `codync-host update` put it next to the host.
- Helpers communicate locally through `~/.codync/screen.sock`. SDP is non-trickle; input uses the `input` and `input-fast` data channels.
+- The Linux helper answers with the packetized H.264 payload the phone offers, feeds the encoder 8-bit 4:2:0 video and timestamps capture on the system clock. It answers before capture starts, so the profile is negotiated before the encoder fixes its output.
- Screen access is on by default on macOS and on Linux while a graphical session (Wayland or X11) is running; headless servers stay off. Turning it on or off is remembered. Enabling through `setScreenEnabled` requires a loopback caller. Interactive OS permission prompts must be completed on the computer.
- Bots with their computer capability enabled receive the built-in `computer` MCP tools, run by the computer-use driver that Codync Screen starts on macOS ([computer use](computer-use.md)). Their permission policy still applies. An interactive phone can take over; bots may still look.
- Each device may hold four sessions; the host allows 32 total, including pending sessions. Credential issuance is limited to 12 requests per minute per account by `TURN_LIMITER` (Cloudflare's per-location rate limiter).