Skip to content

LordOS — a free, fast and secure GNU/Linux distribution based on Debian, with its own desktop environment: display server and Wayland compositor in C, a React UI engine on QuickJS, and private native apps.

Docs: typing lag in Linux apps on the GPU path, the 2-minute check and the key latency trace

Federico Pereiradf6cc2ae77ff
diff --git a/CHANGELOG.md b/CHANGELOG.md
index ccbfd03..f4320a7 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -23,6 +23,13 @@ of the matching [GitHub release](https://github.com/lordbasex/LordOS/releases),
CPU time, threads, context switches, page faults, start time, nice, control group and LordOS
restrictions), open files and ports (with the addresses of each socket), the executable and the
command line, updated every second ([docs/MONITOR.md](docs/MONITOR.md#process-actions-081)).
+- **`tools/diag/keylag.sh`: where does the typing delay come from?** In two minutes it reports the
+ composition mode, the VM and its guest tools and Chromium's complaints, and with
+ `lordos.inputdebug=1` on the kernel line (one boot, root only) it times each stage of a key in
+ ms: read by `lord-server`, delivered to the app, the app's answer, on screen. The trace holds
+ timings only (no key, character or modifier), only for keys sent to Linux apps (never LordOS's
+ own fields, the Terminal, Passwords, login or lock), in a 0600 file in the session's runtime
+ folder, and stops after 120 s ([docs/TROUBLESHOOTING.md](docs/TROUBLESHOOTING.md)).
- **Settings → Sharing → Remote login (SSH).** A new switch, off by default on every install,
turns the standard SSH server on and off for good (`systemctl enable --now` / `disable --now`).
The first time, it installs `openssh-server` through the package engine (the Store's path). The
@@ -78,6 +85,15 @@ of the matching [GitHub release](https://github.com/lordbasex/LordOS/releases),
### Fixed
+- **Typing no longer lags in Chromium and Firefox on GPU composition.** A Linux app's new frame is
+ only a texture on the GPU path, and a display with no other change was not flipped, so a typed
+ character appeared only when something else on the screen moved (up to ~1 s later; Firefox
+ looked frozen), and Chromium logged "buggy presentation feedback". Every GPU display now presents
+ an app's new content in the next frame; a frame shown by a modeset gets its presentation feedback
+ at once; and with nothing drawn (screen off, session away from the monitor) apps still get their
+ frame callbacks once a second. Measured with Chromium: p50/p95 from the key to the screen
+ 949/1866 ms before, 18/25 ms after; also in 0.8.0. Seen on VMware, reproduced on Proxmox
+ ([docs/TROUBLESHOOTING.md](docs/TROUBLESHOOTING.md)).
- **Menus are always above the Dock.** With the Dock on the left, the crown menu opened underneath
it and the Dock took its clicks; select lists inside apps (Passwords' category list) were drawn
under the form fields after them. The desktop now has one stacking order, in layers (Dock < menu
diff --git a/docs/GPU.md b/docs/GPU.md
index 21d66d3..59dcc95 100644
--- a/docs/GPU.md
+++ b/docs/GPU.md
@@ -37,6 +37,12 @@ and its buffers and the apps' dmabufs become framebuffers with `DRM_IOCTL_MODE_A
Diagnostics: `LORD_GPU_DEBUG=1` shows the dmabuf formats, failing imports and the frames sent by
direct scanout. The log says `gpu: GPU composition (renderer…)` or why not.
+**Linux apps' frames (0.8.1 fix).** A Linux app's new frame on the GPU path is only a texture,
+not UI damage. Every GPU display is therefore marked (`Out.wl_new` in `main.c`) and presents it in
+the next frame. Before 0.8.1 nothing was flipped until something else changed, so typing in
+Chromium showed up to a second late. Measurements and the key latency trace are in
+[TROUBLESHOOTING.md](TROUBLESHOOTING.md), "Typing feels slow".
+
**VMware virtual machines** (VMware SVGA, the `vmwgfx` kernel driver). `vmwgfx` is not in the
list of plain drivers, so Mesa is loaded: with the machine's 3D acceleration on, Mesa's `svga`
driver (`vmwgfx_dri.so`, in `libgl1-mesa-dri`, which the ISO ships) is the renderer and lord-server
diff --git a/docs/ISO.md b/docs/ISO.md
index fa1f1a8..0445ecc 100644
--- a/docs/ISO.md
+++ b/docs/ISO.md
@@ -86,6 +86,11 @@ guest's IP); its units have `ConditionVirtualization=vmware` and stay inactive e
part (`open-vm-tools-desktop`, X11) and are not supported yet. GPU or CPU composition on VMware:
see `GPU.md`.
+**Typing lag in Chromium on VMware (0.8.0 and the first 0.8.1 ISOs):** it was a GPU-path
+presentation bug, not `open-vm-tools`, and it is fixed in 0.8.1. See
+[TROUBLESHOOTING.md](TROUBLESHOOTING.md), "Typing feels slow"; `tools/diag/keylag.sh` checks a
+machine in two minutes.
+
Tested with the rebuilt ISO on VM 123 (Proxmox, q35, 4 vCPUs, emulated HD Audio):
| Firmware | Display | RAM | Result |
diff --git a/docs/TROUBLESHOOTING.md b/docs/TROUBLESHOOTING.md
new file mode 100644
index 0000000..b64e564
--- /dev/null
+++ b/docs/TROUBLESHOOTING.md
@@ -0,0 +1,153 @@
+# Troubleshooting
+
+## Typing feels slow
+
+If keys show up late in one app (for example Chromium) but not in LordOS's own apps (Notes, the
+Terminal), check where the time goes before changing anything. `tools/diag/keylag.sh` does it in
+about two minutes.
+
+### The 2-minute check
+
+1. Get the script onto the machine (it is not part of the package):
+
+ ```sh
+ curl -fsSLO https://raw.githubusercontent.com/lordbasex/LordOS/main/tools/diag/keylag.sh
+ ```
+
+2. Run it in the LordOS Terminal: `sh keylag.sh`. It prints:
+ - the version;
+ - GPU or CPU composition;
+ - the virtual machine and its guest tools;
+ - how many times Chromium logged `The server has buggy presentation feedback` in this session.
+
+ On LordOS 0.8.0 and the 0.8.1 pre-release ISOs, that Chromium line on the GPU path is the bug
+ described below. To confirm it without the trace, reboot and add `lordos.gpu=0` to the kernel
+ line for one boot. That forces CPU composition. Then type in Chromium again.
+
+3. On 0.8.1 or later, turn the trace on for one boot. In the boot menu, press `e`, add
+ `lordos.inputdebug=1` at the end of the line that starts with `linux`, and press Ctrl+X. On an
+ installed system with a hidden menu, hold Shift (BIOS) or press Esc (UEFI) while it starts.
+ Log in and run `sh keylag.sh` again. When it asks, click the slow app and type normally for
+ 30 seconds.
+
+The script then prints p50, p95 and the maximum for each stage, in ms after the kernel's evdev
+timestamp:
+
+| Stage | What happened |
+|---|---|
+| `rx` | `lord-server` read the key |
+| `flush` | the key was written to the app's socket (`wl_keyboard.key`) |
+| `commit` | the app sent its next frame (its answer: the glyph) |
+| `shown` | the page flip with that frame finished |
+
+It also reports the share of key presses that arrived less than 15 ms after the previous one.
+Real typing almost never does that. A high share means the keys were held back before the
+guest's kernel and then delivered in clumps.
+
+How to read it:
+
+- **`rx` high:** `lord-server`'s main loop is busy. Report it with the output.
+- **`commit − flush` high:** the app itself is slow.
+- **`shown − commit` high:** frames reach the screen late, a compositor problem. Report it, and use
+ `lordos.gpu=0` until it is fixed.
+- **Everything under ~40 ms but typing still lags:** the delay is before the guest's kernel, in
+ the hypervisor. On VMware, the reported fixes are on the host side (VMware's forums and
+ [open-vm-tools#696](https://github.com/vmware/open-vm-tools/issues/696)):
+ - turn off "Accelerate 3D graphics";
+ - use fewer vCPUs;
+ - add `keyboard.vusb.enable = "TRUE"` to the `.vmx`.
+
+ None of them is a guest bug.
+
+### The trace and your privacy
+
+The trace measures time. It is not a key log:
+
+- **What a line holds:** an opaque sequence number, press or release, and the stage times. No key
+ code, character or modifier is ever written.
+- **Which keys are traced:** only keys handed to a Linux (Wayland) app. Keys for LordOS itself are
+ never traced or counted: the theme, its text fields, the Terminal, the Passwords app, the
+ keyring and admin prompts, the login and lock screens, isolated LordOS apps and protected
+ windows.
+- **Who can turn it on:** only `lord-sessiond` (root) passes `LORD_INPUT_DEBUG=1` to user
+ sessions, and only when the kernel line has `lordos.inputdebug=1` or the service sets
+ `LORD_INPUT_DEBUG=1`. There is no run-time switch, and the login screen never gets it.
+- **Where it goes:** its own file, `$XDG_RUNTIME_DIR/lordos-keylag.log`, created with mode 0600 in
+ the session's tmpfs and gone at logout. It is never written to the session log.
+- **How long it runs:** it stops by itself 120 s after the first traced key, or after 4000 lines.
+ Log out and back in for a new one.
+
+The code is in `server/src/keylag.c`.
+
+### The 0.8.1 fix: Linux apps' frames on the GPU path
+
+The VMware report (Fusion, UEFI, 3D on): typing lagged in Chromium but not in Notes, and Chromium
+logged `buggy presentation feedback` hundreds of times. Firefox looked frozen.
+
+**Cause.** On the GPU path, a Linux app's new frame only replaces a texture. It is not UI damage,
+so `render_frame` skipped every display that had no other damage, and nothing was flipped.
+
+- **What the user saw:** the typed character appeared only when something else on the screen
+ changed: the clock, the cursor, an animation.
+- **What the apps got:** frame callbacks still went out, but presentation feedback piled up
+ waiting for a flip.
+
+The bug was present in 0.8.0 as well (same code). On the CPU path, the window is painted into the
+UI and damages it, so that path was never affected.
+
+**Fix** (`server/src/main.c`):
+
+- A Linux app's new content marks every GPU display (`wl_new`), and those displays present it in
+ the next frame.
+- When a frame is shown by a modeset instead of a page flip, its presentation feedback is sent
+ right away.
+- When nothing is drawn (screen off, the session away from the monitor), apps still get their
+ frame callbacks once a second and their feedback is discarded (`wl_frame_unseen`), so none of
+ them stalls.
+
+**Measured** on Proxmox (VM 123, 4 vCPUs, GPU composition with llvmpipe). The harness typed 80
+keys into a Chromium text area with `tools/diag/keylag-inject.py` and timed them with the trace.
+Times are in ms after the kernel event, p50 / p95:
+
+| Build | Chromium, static page: on screen | Chromium, page animating: on screen | Chromium "buggy presentation feedback" |
+|---|---|---|---|
+| 0.8.0 (46c972a) | 829 / 1840 | 903 / 1876 | 113 lines |
+| main before the fix | 949 / 1866 | 1000 / 1917 | 112 lines |
+| main with the fix | **18 / 25** | **28 / 47** | **0** |
+| main with the fix, CPU composition | 15 / 21 | 24 / 47 | 0 |
+
+Other results:
+
+- **Firefox with the fix** (GPU path): on screen 31 / 39 ms.
+- **The test client** (`tools/diag/keylag-client.c`, a minimal Wayland window that repaints on each
+ key), CPU composition: same in 0.8.0 and main. Idle: on screen 2.5 / 6.6 ms in 0.8.0 and
+ 2.6 / 7.5 ms in main. With Chromium animating: 32 / 47 ms in 0.8.0 and 35 / 59 ms in main.
+- **The key path itself:** reading evdev and handing the key to the app (`rx`, `flush`) stayed
+ under 1 ms at p50 in every build. Nothing on that path changed between 0.8.0 and 0.8.1.
+
+### The harness (developers)
+
+Everything is in `tools/diag/`:
+
+- **`keylag-inject.py`** (root) creates a uinput keyboard and types at known `CLOCK_MONOTONIC`
+ times. `lord-sessiond` lends the new device to the session through inotify.
+- **`keylag-client.c`** prints when `wl_keyboard.key` arrived and when its frame was presented.
+ Build commands are at the top of the file.
+- **`LORD_INPUT_DEBUG=1`** in a drop-in for `lordos.service`
+ (`[Service] Environment=LORD_INPUT_DEBUG=1`) turns the trace on in development VMs.
+- **`LORD_GPU=1`** in the same drop-in forces GPU composition (llvmpipe) on a display without 3D.
+
+### open-vm-tools
+
+The 0.8.1 ISO keeps `open-vm-tools`, which installs only on VMware
+(`ConditionVirtualization=vmware`). It runs `vmtoolsd` with these plugins:
+
+- power operations, guest info, time sync (only if the host enables it), backup quiescing and
+ `deployPkg`;
+- `resolutionKMS`, which only acts when the host asks for a new display layout.
+
+None of these is on the input path. The kernel's VMMouse driver handles the absolute pointer by
+itself, and `vmtoolsd` does not touch it. No upstream report ties `open-vm-tools` to keyboard
+lag: [open-vm-tools#696](https://github.com/vmware/open-vm-tools/issues/696) was closed as a
+VMware Workstation bug. The VMware report above was the GPU presentation bug, and it reproduced
+on Proxmox, where `open-vm-tools` is inactive.