Skip to content

A treemap for finding and removing what fills your disk, for Omarchy. Rust + GPUI.

Rust

scan: stream SSH walks with bounded memory and strict mount boundaries

Tobi Lutke8b55208fdc08
diff --git a/Cargo.lock b/Cargo.lock
index 32f18ec..dde47dd 100644
--- a/Cargo.lock
+++ b/Cargo.lock
@@ -1455,6 +1455,7 @@ name = "disktree-scan"
version = "0.11.0"
dependencies = [
"disktree-core",
+ "rustix",
"serde_json",
"tempfile",
]
diff --git a/README.md b/README.md
index 5dc8df5..bfe21a9 100644
--- a/README.md
+++ b/README.md
@@ -202,13 +202,23 @@ terminal, it can ask for a password or key passphrase there; GUI launches
need key/agent access or a working SSH askpass helper. Connect once with
`ssh dev` to verify access and accept its host key.
-The headless helper reuses the parallel scanner: allocated blocks, hardlink
-deduplication, hidden entries and mount boundaries have the same semantics as
-a local scan. One compressed SSH connection carries a versioned JSON request
-on stdin and compact postorder records on stdout. File contents never cross
-the connection, and paths are never interpolated into a remote shell command.
-The complete tree is retained for offline navigation; very large trees still
-require memory proportional to their entry count.
+The Linux/macOS helper streams a depth-first walk directly to stdout. It
+never constructs a tree or retains a directory's children: memory scales
+with open ancestors (at most 256), not file count. Only pending hardlink
+identities are retained, capped at 65,536; ordinary files need no identity
+storage. A resource limit fails explicitly instead of returning a partial
+tree. Symlinks are not followed and every nested mount is skipped, even a
+bind mount or same-device subvolume. Select those as separate roots.
+
+One compressed SSH connection carries a bounded JSON request on stdin and
+compact postorder records on stdout. The client keeps stdin open so the
+helper exits on disconnect, including while scanning or blocked on output.
+Linux helpers also enforce a 1 GiB address-space ceiling, disable core dumps,
+and stop after five minutes. File contents never cross the connection and
+paths are never interpolated into remote shell commands. Only the desktop
+retains the complete tree for offline navigation; its memory still scales
+with entry count. The native Windows helper currently retains the original
+tree-building scan path.
Remote mode is **export-only**. Mark with Space, open review with `c`, then:
@@ -382,7 +392,8 @@ and removes duplicate hardlinks.
Press `v` (or click **Volumes** beside the disk name in the side panel) to
bring up the volume picker. It lists every candidate volume mounted on the
system together with its free space, ordered fullest first, filtering out
-pseudo-filesystems and duplicate btrfs/APFS mount points. Selecting any
+pseudo-filesystems and snapshots. Linux bind mounts and btrfs subvolumes
+remain separately selectable, even when they share a device. Selecting any
entry resets the scan root directly to that volume.
## The whole disk
@@ -400,15 +411,15 @@ outside `~` (on this machine, seconds instead of a full rescan). The current
view stays on screen until the wider tree lands, which then opens with the
directory you came from selected. Going back down is just navigation.
-A scan stays on one volume, and a volume is the mount *source*, not the
-device number: btrfs gives each subvolume its own `st_dev`, so `/home`,
-`/var/log` and `/var/cache/pacman/pkg` are included, while `/proc`,
-`/sys`, `/run`, tmpfs, `/boot`, other disks, network shares and automount
-points are left out (checked by path, so an automounted NAS is never
-mounted just to be measured). Snapshot subvolumes are left out too: their
-files share blocks with the live ones, and counting them would count the disk
-twice. `-X` crosses into everything: a mark that lands on another volume is
-still removed, but it is not added to the metered volume's free space.
+By default a scan stops at **every nested mount point**, including bind
+mounts and same-device btrfs subvolumes. Boundaries are checked by path,
+before metadata or directory reads, so an automounted NAS is not activated
+just to measure its parent. Choose `/home`, `/var/log`, or another mount as
+its own root rather than implicitly traversing it from `/`.
+
+Local `-X` explicitly crosses mounts; remote streaming scans reject it.
+A local mark on another volume is still removed, but is not added to the
+metered volume's free-space projection.
Without root, some system directories cannot be read; they are counted as
unreadable in the top bar rather than guessed at.
diff --git a/crates/disktree-core/src/lib.rs b/crates/disktree-core/src/lib.rs
index 410143e..b25f711 100644
--- a/crates/disktree-core/src/lib.rs
+++ b/crates/disktree-core/src/lib.rs
@@ -18,6 +18,8 @@ mod macos;
mod mft;
pub mod payload;
pub mod remote;
+#[cfg(unix)]
+mod remote_walk;
pub mod removal;
pub mod scan;
pub mod scan_threads;
diff --git a/crates/disktree-core/src/remote.rs b/crates/disktree-core/src/remote.rs
index b6502ad..ebe3d30 100644
--- a/crates/disktree-core/src/remote.rs
+++ b/crates/disktree-core/src/remote.rs
@@ -11,7 +11,9 @@ use std::sync::{Arc, Mutex, mpsc};
use serde::{Deserialize, Serialize};
-use crate::scan::{ScanOptions, scan};
+use crate::scan::ScanOptions;
+#[cfg(not(unix))]
+use crate::scan::scan;
use crate::space::{SpaceInfo, space_info, volume_root_for};
use crate::tree::{Node, NodeKind, aggregate};
@@ -38,6 +40,16 @@ pub struct Request {
pub options: ScanOptions,
}
+// Old helpers ignore this extra field. Old clients may still half-close
+// stdin; only clients opting in promise to keep it open during the scan.
+#[derive(Serialize, Deserialize)]
+struct SessionRequest {
+ #[serde(flatten)]
+ request: Request,
+ #[serde(default)]
+ watch_disconnect: bool,
+}
+
#[derive(Debug, Serialize, Deserialize)]
pub struct Header {
version: u32,
@@ -59,13 +71,24 @@ pub struct Snapshot {
// Children are already on the receiver's stack; counts preserve empty dirs
// and deduplicated leaves without transmitting derived subtree totals.
#[derive(Debug, Serialize, Deserialize)]
-struct Record(String, u8, u64, u64, i64, bool, usize);
+pub(crate) struct Record(
+ pub String,
+ pub u8,
+ pub u64,
+ pub u64,
+ pub i64,
+ pub bool,
+ pub usize,
+);
fn json_error(error: serde_json::Error) -> io::Error {
io::Error::new(io::ErrorKind::InvalidData, error)
}
-fn send(out: &mut impl Write, value: &impl Serialize) -> io::Result<()> {
+pub(crate) fn send(
+ out: &mut impl Write,
+ value: &impl Serialize,
+) -> io::Result<()> {
serde_json::to_writer(&mut *out, value).map_err(json_error)?;
out.write_all(b"\n")
}
@@ -97,7 +120,18 @@ fn receive<T: serde::de::DeserializeOwned>(
/// The helper has exactly one operation: scan. It never removes files,
/// installs software, invokes sudo, or executes a command from the client.
pub fn serve(input: &mut impl BufRead, out: &mut impl Write) -> io::Result<()> {
- let request: Request = receive(input)?;
+ let (request, _) = read_request(input)?;
+ serve_request(request, out)
+}
+
+/// Read one bounded request, including the opt-in connection lifetime.
+pub fn read_request(input: &mut impl BufRead) -> io::Result<(Request, bool)> {
+ let session: SessionRequest = receive(input)?;
+ Ok((session.request, session.watch_disconnect))
+}
+
+/// Scan a decoded request. The standalone helper supervises its lifetime.
+pub fn serve_request(request: Request, out: &mut impl Write) -> io::Result<()> {
let home = std::env::home_dir();
let root = request
.root
@@ -112,26 +146,30 @@ pub fn serve(input: &mut impl BufRead, out: &mut impl Write) -> io::Result<()> {
if !root.is_dir() {
return Err(io::Error::other("scan root is not a directory"));
}
- let tree = scan(&root, request.options)?;
send(
out,
&Header {
version: VERSION,
disk_root: volume_root_for(&root),
space: space_info(&root).ok(),
- root,
+ root: root.clone(),
home,
platform: std::env::consts::OS.into(),
identity: crate::identity::HostIdentity::current(),
},
)?;
- write_node(out, &tree)?;
+ out.flush()?;
+ #[cfg(unix)]
+ crate::remote_walk::stream(&root, &request.options, out)?;
+ #[cfg(not(unix))]
+ write_node(out, &scan(&root, request.options)?)?;
// An explicit terminator distinguishes a complete scan from a severed SSH
// connection after any otherwise valid prefix of the postorder stream.
send(out, &"done")?;
out.flush()
}
+#[cfg(not(unix))]
fn write_node(out: &mut impl Write, node: &Node) -> io::Result<()> {
for child in &node.children {
write_node(out, child)?;
@@ -280,8 +318,9 @@ fn ssh_error(host: &str, details: &str, fallback: &io::Error) -> io::Error {
}
/// Cancelling a rescan or closing a window kills SSH and closes the pipe.
-/// SSH normally terminates its remote session too; the helper also exits
-/// when a write observes the closed connection.
+///
+/// The helper monitors stdin for EOF even before it has a tree to write;
+/// keep the request pipe open until the entire snapshot has arrived.
#[derive(Debug)]
pub struct RemoteScan {
child: Arc<Mutex<Child>>,
@@ -354,8 +393,13 @@ impl RemoteScan {
&archive, &mut input, &mut stdin,
)?;
}
- send(&mut stdin, &request)?;
- drop(stdin);
+ send(
+ &mut stdin,
+ &SessionRequest {
+ request,
+ watch_disconnect: true,
+ },
+ )?;
read_snapshot(&mut input)
})()
.map(|mut snapshot| {
@@ -364,6 +408,7 @@ impl RemoteScan {
}
snapshot
});
+ drop(stdin);
let mut process = worker.lock().expect("SSH process lock");
if snapshot.is_err() {
let _ = process.kill();
@@ -443,7 +488,7 @@ mod tests {
#[test]
fn ssh_authentication_diagnostics_reach_the_caller() {
let error = failing_process(
- "cat >/dev/null; echo 'Permission denied (publickey,password).' \
+ "read -r request; echo 'Permission denied (publickey,password).' \
>&2; exit 255",
)
.to_string();
@@ -457,7 +502,7 @@ mod tests {
#[test]
fn missing_helper_diagnostics_name_the_server() {
let error = failing_process(
- "cat >/dev/null; echo 'disktree-scan: command not found' \
+ "read -r request; echo 'disktree-scan: command not found' \
>&2; exit 127",
)
.to_string();
@@ -470,7 +515,7 @@ mod tests {
fn banner_and_truncated_output_are_reported_without_a_hang() {
for output in ["hello", "hello\\n"] {
let error =
- failing_process(&format!("cat >/dev/null; printf '{output}'"))
+ failing_process(&format!("read -r request; printf '{output}'"))
.to_string();
assert!(error.contains("SSH dev:"), "{error}");
}
diff --git a/crates/disktree-core/src/remote_walk.rs b/crates/disktree-core/src/remote_walk.rs
new file mode 100644
index 0000000..64c2667
--- /dev/null
+++ b/crates/disktree-core/src/remote_walk.rs
@@ -0,0 +1,253 @@
+//! Bounded-memory SSH traversal. No Node, arena, child vector or sort:
+//! the receiver owns the tree; the helper retains only its open ancestors.
+use std::fs::{self, ReadDir};
+use std::io::{self, Write};
+use std::os::unix::fs::MetadataExt as _;
+use std::path::{Path, PathBuf};
+
+use rustc_hash::{FxHashMap, FxHashSet};
+
+use crate::remote::{Record, send};
+use crate::scan::{ScanOptions, measure, modified_seconds};
+
+// Open directory iterators are a bounded resource too. Fail explicitly,
+// rather than silently truncating an exceptionally deep or hardlinked tree.
+const MAX_ANCESTORS: usize = 256;
+const MAX_PENDING_LINKS: usize = 65_536;
+
+struct Frame {
+ entries: Option<ReadDir>,
+ record: Record,
+}
+
+impl Frame {
+ fn open(path: &Path, name: String) -> Self {
+ let entries = fs::read_dir(path);
+ let failed = entries.is_err();
+ Self {
+ entries: entries.ok(),
+ record: Record(name, 0, 0, 0, 0, failed, 0),
+ }
+ }
+}
+
+pub fn stream(
+ root: &Path,
+ options: &ScanOptions,
+ out: &mut impl Write,
+) -> io::Result<()> {
+ if options.follow_links || !options.one_filesystem {
+ return Err(io::Error::other(
+ "Remote streaming scans never follow symlinks or cross mounts; select that mount as a separate scan root.",
+ ));
+ }
+ // Require a table, not just st_dev: a bind mount can have the same dev.
+ let boundaries =
+ crate::space::mount_boundaries_for(root).ok_or_else(|| {
+ io::Error::other("cannot read remote mount boundaries")
+ })?;
+ let mut excluded: FxHashSet<PathBuf> = boundaries.into_iter().collect();
+ excluded.extend(crate::space::never_scanned(root, root));
+ stream_with_boundaries(root, options, &excluded, out)
+}
+
+fn stream_with_boundaries(
+ root: &Path,
+ options: &ScanOptions,
+ excluded: &FxHashSet<PathBuf>,
+ out: &mut impl Write,
+) -> io::Result<()> {
+ let device = fs::metadata(root)?.dev();
+ let mut frames = vec![Frame::open(root, "root".into())];
+ // Ordinary single-link files need no identity storage. Forget a shared
+ // inode once all its names were met, and bound unmatched identities.
+ let mut links = FxHashMap::<(u64, u64), u64>::default();
+ while let Some(frame) = frames.last_mut() {
+ let entry = match frame.entries.as_mut().and_then(Iterator::next) {
+ None => {
+ let frame = frames.pop().expect("frame checked above");
+ send(out, &frame.record)?;
+ continue;
+ }
+ Some(Err(_)) => {
+ frame.record.5 = true;
+ continue;
+ }
+ Some(Ok(entry)) => entry,
+ };
+ let name = entry.file_name();
+ let Some(name) = name.to_str() else {
+ // A replacement character must never become a deletable path.
+ frame.record.5 = true;
+ continue;
+ };
+ if !options.include_hidden && name.starts_with('.') {
+ continue;
+ }
+ let path = entry.path();
+ // Before metadata or read_dir: do not trigger an automount.
+ if excluded.contains(&path) {
+ continue;
+ }
+ let Ok(meta) = fs::symlink_metadata(&path) else {
+ frame.record.5 = true;
+ continue;
+ };
+ if meta.dev() != device {
+ continue;
+ }
+ if meta.is_dir() {
+ if options.max_depth.is_some_and(|limit| frames.len() > limit) {
+ continue;
+ }
+ if frames.len() >= MAX_ANCESTORS {
+ return Err(io::Error::other(
+ "remote scan nesting limit reached; scan a smaller root",
+ ));
+ }
+ frames.last_mut().expect("parent exists").record.6 += 1;
+ frames.push(Frame::open(&path, name.to_owned()));
+ continue;
+ }
+ let kind = if meta.is_file() {
+ 1
+ } else if meta.is_symlink() {
+ 2
+ } else {
+ 3
+ };
+ let mut bytes = if kind == 3 {
+ 0
+ } else {
+ measure(&meta, options.apparent_size)
+ };
+ if options.dedup_hardlinks && meta.is_file() && meta.nlink() > 1 {
+ let identity = (meta.dev(), meta.ino());
+ if let Some(remaining) = links.get_mut(&identity) {
+ bytes = 0;
+ *remaining -= 1;
+ if *remaining == 0 {
+ links.remove(&identity);
+ }
+ } else {
+ if links.len() >= MAX_PENDING_LINKS {
+ return Err(io::Error::other(
+ "remote hardlink tracking limit reached; scan a smaller root",
+ ));
+ }
+ links.insert(identity, meta.nlink() - 1);
+ }
+ }
+ send(
+ out,
+ &Record(
+ name.to_owned(),
+ kind,
+ bytes,
+ u64::from(meta.is_file()),
+ modified_seconds(&meta),
+ false,
+ 0,
+ ),
+ )?;
+ frames.last_mut().expect("parent exists").record.6 += 1;
+ }
+ Ok(())
+}
+
+#[cfg(test)]
+mod tests {
+ use super::*;
+
+ fn records(
+ root: &Path,
+ options: &ScanOptions,
+ excluded: &[PathBuf],
+ ) -> Vec<Record> {
+ let mut bytes = Vec::new();
+ stream_with_boundaries(
+ root,
+ options,
+ &excluded.iter().cloned().collect(),
+ &mut bytes,
+ )
+ .unwrap();
+ bytes
+ .split(|byte| *byte == b'\n')
+ .filter(|line| !line.is_empty())
+ .map(|line| serde_json::from_slice(line).unwrap())
+ .collect()
+ }
+
+ #[test]
+ fn streaming_accounts_for_hardlinks_symlinks_and_allocated_bytes() {
+ let dir = tempfile::tempdir().unwrap();
+ let data = dir.path().join("data");
+ fs::write(&data, b"hello").unwrap();
+ fs::hard_link(&data, dir.path().join("alias")).unwrap();
+ fs::create_dir(dir.path().join("sub")).unwrap();
+ fs::write(dir.path().join("sub/leaf"), b"leaf").unwrap();
+ std::os::unix::fs::symlink("sub", dir.path().join("link")).unwrap();
+ let rows = records(dir.path(), &ScanOptions::default(), &[]);
+ let actual: u64 =
+ rows.iter().filter(|row| row.1 != 0).map(|row| row.2).sum();
+ let expected =
+ crate::scan::scan(dir.path(), ScanOptions::default()).unwrap();
+ assert_eq!(actual, expected.bytes);
+ assert_eq!(rows.iter().map(|row| row.3).sum::<u64>(), 3);
+ assert_eq!(rows.iter().filter(|row| row.0 == "leaf").count(), 1);
+ assert_eq!(rows.last().unwrap().6, 4);
+ assert_eq!(rows.iter().filter(|row| row.0 == "sub").count(), 1);
+ }
+
+ #[test]
+ fn mounted_children_are_omitted_even_on_the_same_device() {
+ let dir = tempfile::tempdir().unwrap();
+ let mounted = dir.path().join("mounted");
+ fs::create_dir(&mounted).unwrap();
+ fs::write(mounted.join("must-not-read"), b"secret").unwrap();
+ fs::write(dir.path().join("keep"), b"hello").unwrap();
+ let rows = records(dir.path(), &ScanOptions::default(), &[mounted]);
+ assert_eq!(rows.len(), 2);
+ assert_eq!(rows[0].0, "keep");
+ assert_eq!(rows[1].6, 1);
+ }
+
+ #[test]
+ fn hidden_and_depth_filters_do_not_change_postorder_counts() {
+ let dir = tempfile::tempdir().unwrap();
+ fs::write(dir.path().join(".hidden"), b"hidden").unwrap();
+ fs::create_dir(dir.path().join("sub")).unwrap();
+ fs::write(dir.path().join("sub/leaf"), b"leaf").unwrap();
+ fs::write(dir.path().join("visible"), b"visible").unwrap();
+ let rows = records(
+ dir.path(),
+ &ScanOptions {
+ include_hidden: false,
+ max_depth: Some(0),
+ ..ScanOptions::default()
+ },
+ &[],
+ );
+ assert_eq!(rows.len(), 2);
+ assert_eq!(rows[0].0, "visible");
+ assert_eq!(rows[1].6, 1);
+ }
+
+ #[test]
+ fn unsafe_traversal_options_fail_before_walking() {
+ let dir = tempfile::tempdir().unwrap();
+ for options in [
+ ScanOptions {
+ follow_links: true,
+ ..ScanOptions::default()
+ },
+ ScanOptions {
+ one_filesystem: false,
+ ..ScanOptions::default()
+ },
+ ] {
+ assert!(stream(dir.path(), &options, &mut Vec::new()).is_err());
+ }
+ }
+}
diff --git a/crates/disktree-core/src/scan.rs b/crates/disktree-core/src/scan.rs
index 312879b..13202f7 100644
--- a/crates/disktree-core/src/scan.rs
+++ b/crates/disktree-core/src/scan.rs
@@ -49,8 +49,8 @@ pub struct ScanOptions {
/// `~/.cache` is frequently the largest directory in a home directory.
pub include_hidden: bool,
/// Stay on the root's volume: skip other disks, pseudo filesystems,
- /// network shares and automount points, but keep subvolumes of the same
- /// disk. See [`crate::space::foreign_mounts`]. On by default: a disk
+ /// network shares, bind mounts and subvolumes. Select those explicitly
+ /// as another root. See [`crate::space::mount_boundaries`]. A disk
/// tool measures a disk, and its free-space meter only means anything
/// for one volume.
pub one_filesystem: bool,
@@ -375,6 +375,15 @@ impl WalkContext {
{
return Classified::Skipped;
}
+ if self.options.one_filesystem
+ && let Some(foreign) = self.foreign_mounts.get()
+ {
+ // Checked by path before anything reads the directory, so an
+ // automount point is never triggered.
+ if foreign.contains(&path) {
+ return Classified::Skipped;
+ }
+ }
// Read at most once per directory: macOS needs it for the dataless
// flag, and the device check below wherever the mount table cannot
// be read, which on macOS is everywhere.
@@ -397,15 +406,7 @@ impl WalkContext {
tree.name = entry.take_name();
return Classified::Entry(tree);
}
- if self.options.one_filesystem
- && let Some(foreign) = self.foreign_mounts.get()
- {
- // Checked by path before anything reads the directory, so an
- // automount point is never triggered.
- if foreign.contains(&path) {
- return Classified::Skipped;
- }
- } else if self.options.one_filesystem {
+ if self.options.one_filesystem && self.foreign_mounts.get().is_none() {
match (directory(), self.root_device(&path)) {
(Ok(directory), Some(root_device))
if directory.device != root_device =>
@@ -872,7 +873,7 @@ fn scan_on_pool(
let _ = context.never_scanned.set(never.into_iter().collect());
if context.options.one_filesystem {
*lock(&context.root_device) = Some(device_of(&root_meta));
- if let Some(foreign) = crate::space::foreign_mounts_for(canonical) {
+ if let Some(foreign) = crate::space::mount_boundaries_for(canonical) {
let _ = context.foreign_mounts.set(foreign.into_iter().collect());
}
}
@@ -1211,7 +1212,7 @@ fn leaf_node(
/// Last write time in Unix seconds, from the metadata the walk already has:
/// age costs no extra system call.
-fn modified_seconds(meta: &Metadata) -> i64 {
+pub(crate) fn modified_seconds(meta: &Metadata) -> i64 {
meta.modified()
.ok()
.and_then(|time| time.duration_since(std::time::UNIX_EPOCH).ok())
@@ -1220,7 +1221,7 @@ fn modified_seconds(meta: &Metadata) -> i64 {
})
}
-fn measure(meta: &Metadata, apparent_size: bool) -> u64 {
+pub(crate) fn measure(meta: &Metadata, apparent_size: bool) -> u64 {
if apparent_size {
meta.len()
} else {
diff --git a/crates/disktree-core/src/space.rs b/crates/disktree-core/src/space.rs
index f5ca414..e318d87 100644
--- a/crates/disktree-core/src/space.rs
+++ b/crates/disktree-core/src/space.rs
@@ -106,8 +106,7 @@ pub fn device_in(table: &str, path: &Path) -> Option<String> {
.filter_map(|line| {
let mut fields = line.split_whitespace();
let device = fields.next()?;
- // Spaces in mount points are escaped as \040.
- let mount = fields.next()?.replace("\\040", " ");
+ let mount = unescape_octal(fields.next()?);
path.starts_with(&mount)
.then(|| (mount.len(), device.to_string()))
})
@@ -132,8 +131,7 @@ pub fn parse_mounts(table: &str) -> Vec<Mount> {
.filter_map(|line| {
let mut fields = line.split_whitespace();
let source = fields.next()?.to_string();
- // Spaces in mount points are escaped as \040.
- let point = PathBuf::from(fields.next()?.replace("\\040", " "));
+ let point = PathBuf::from(unescape_octal(fields.next()?));
let fstype = fields.next()?.to_string();
let options = fields.next().unwrap_or_default().to_string();
Some(Mount {
@@ -389,6 +387,46 @@ pub fn volume_root_for(path: &Path) -> Option<PathBuf> {
crate::windows::volume_root(path)
}
+/// Every nested mount is a scan boundary, including bind mounts and
+/// same-device subvolumes. Selecting a mount explicitly makes it a root.
+pub fn mount_boundaries(mounts: &[Mount], root: &Path) -> Vec<PathBuf> {
+ mounts
+ .iter()
+ .filter(|mount| mount.point != root && mount.point.starts_with(root))
+ .map(|mount| mount.point.clone())
+ .collect()
+}
+
+/// Boundaries from this machine's own mount namespace.
+pub fn mount_boundaries_for(root: &Path) -> Option<Vec<PathBuf>> {
+ #[cfg(not(any(target_os = "macos", windows)))]
+ {
+ Some(mount_boundaries(&mount_table()?, root))
+ }
+ #[cfg(target_os = "macos")]
+ {
+ let output = std::process::Command::new("mount").output().ok()?;
+ if !output.status.success() {
+ return None;
+ }
+ Some(
+ parse_macos_mounts(&String::from_utf8_lossy(&output.stdout))
+ .into_iter()
+ .filter(|point| point != root && point.starts_with(root))
+ .collect(),
+ )
+ }
+ #[cfg(windows)]
+ {
+ Some(
+ crate::windows::mount_points()
+ .into_iter()
+ .filter(|point| point != root && point.starts_with(root))
+ .collect(),
+ )
+ }
+}
+
/// [`foreign_mounts`] for this machine; `None` when the mount table cannot
/// be read, so the caller can fall back to comparing devices.
#[cfg(not(windows))]
@@ -717,9 +755,8 @@ pub struct Volume {
/// Pseudo filesystems (`/proc`, `/sys`, tmpfs, …), snapshot subvolumes and
/// automount points are left out: switching the scan to one of those would
/// measure the wrong thing, the same reason [`foreign_mounts`] keeps a scan
-/// from entering them. Duplicates from one device mounted twice (a btrfs
-/// subvolume at `/` and `/home`) collapse to the shortest mount point, which
-/// is the top of that disk.
+/// from entering them. Same-device mounts remain individually selectable:
+/// a scan no longer crosses into another mount implicitly.
#[cfg(not(any(target_os = "macos", windows)))]
pub fn volumes() -> Vec<Volume> {
let Ok(table) = std::fs::read_to_string("/proc/self/mounts") else {
@@ -735,15 +772,9 @@ pub fn volumes_in(mounts: &[Mount]) -> Vec<Volume> {
if !is_volume_candidate(mount) {
continue;
}
- // One device mounted twice (a btrfs disk at `/`, `/home`,
- // `/var/log`) is one volume: keep the shortest mount point, which is
- // the top of that disk.
- if let Some(known) = seen.iter_mut().find(|known| {
- known.source == mount.source && known.fstype == mount.fstype
- }) {
- if mount.point.as_os_str().len() < known.point.as_os_str().len() {
- *known = mount;
- }
+ // Nested mounts are not traversed. Even mounts sharing a device
+ // must remain individually selectable; only overmounts collapse.
+ if seen.iter().any(|known| known.point == mount.point) {
continue;
}
seen.push(mount);
@@ -1030,6 +1061,22 @@ portal /run/user/1000/doc fuse.portal rw 0 0
);
}
+ #[test]
+ fn strict_scan_boundaries_include_same_device_and_bind_mounts() {
+ let mounts = parse_mounts(
+ "/dev/root / btrfs rw 0 0\n\
+ /dev/root /home btrfs rw,subvol=@home 0 0\n\
+ /dev/root /bind btrfs rw 0 0\n\
+ /dev/other /external ext4 rw 0 0\n",
+ );
+ let boundaries = mount_boundaries(&mounts, Path::new("/"));
+ assert_eq!(
+ boundaries,
+ ["/home", "/bind", "/external"].map(PathBuf::from)
+ );
+ assert!(mount_boundaries(&mounts, Path::new("/home")).is_empty());
+ }
+
#[test]
fn a_volume_includes_its_subvolumes_and_nothing_else() {
let mounts = parse_mounts(OMARCHY);
@@ -1057,11 +1104,12 @@ portal /run/user/1000/doc fuse.portal rw 0 0
.iter()
.map(|volume| volume.point.clone())
.collect();
- // One btrfs disk (at `/`, collapsing `/home` and `/var/log`), plus
- // the boot disk; tmpfs, autofs, portals and snapshots are not scans.
+ // Same-device subvolumes stay selectable because a scan of the
+ // parent now stops at their mount boundaries.
assert!(points.contains(&PathBuf::from("/")), "{points:?}");
assert!(points.contains(&PathBuf::from("/boot")), "{points:?}");
- assert_eq!(points.len(), 2, "{points:?}");
+ assert!(points.contains(&PathBuf::from("/home")), "{points:?}");
+ assert!(points.contains(&PathBuf::from("/var/log")), "{points:?}");
}
#[test]
diff --git a/crates/disktree-scan/Cargo.toml b/crates/disktree-scan/Cargo.toml
index 90f0c9e..2f6e000 100644
--- a/crates/disktree-scan/Cargo.toml
+++ b/crates/disktree-scan/Cargo.toml
@@ -12,6 +12,9 @@ workspace = true
[dependencies]
disktree-core.workspace = true
+[target.'cfg(target_os = "linux")'.dependencies]
+rustix = { version = "1.1", features = ["process"] }
+
[dev-dependencies]
serde_json = "1"
tempfile = "3"
diff --git a/crates/disktree-scan/src/main.rs b/crates/disktree-scan/src/main.rs
index befc429..0d51f63 100644
--- a/crates/disktree-scan/src/main.rs
+++ b/crates/disktree-scan/src/main.rs
@@ -1,11 +1,48 @@
//! Headless helper: no GPUI dependency and no destructive command surface.
-use std::io::{self, BufWriter};
+use std::io::{self, BufReader, BufWriter, Read as _};
+use std::time::Duration;
fn main() -> io::Result<()> {
let args: Vec<String> = std::env::args().skip(1).collect();
if args == ["--stdio-v1"] {
- disktree_core::remote::serve(
- &mut io::stdin().lock(),
+ limit_resources()?;
+ // A live client is not permission to monopolise the server forever,
+ // including when a filesystem syscall cannot be cancelled.
+ std::thread::Builder::new()
+ .name("scan-deadline".into())
+ .spawn(|| {
+ std::thread::park_timeout(Duration::from_mins(5));
+ eprintln!(
+ "Remote scan exceeded five minutes; scan a smaller path."
+ );
+ std::process::exit(1);
+ })?;
+ let mut input = BufReader::new(io::stdin());
+ let (request, watch_disconnect) =
+ disktree_core::remote::read_request(&mut input)?;
+ if watch_disconnect {
+ std::thread::Builder::new()
+ .name("scan-connection".into())
+ .spawn(move || {
+ let mut byte = [0];
+ loop {
+ match input.read(&mut byte) {
+ Err(error) if error.kind() == io::ErrorKind::Interrupted => {}
+ Ok(0) => {
+ // Exit the process, not just a scan task: a
+ // blocked filesystem must not outlive SSH.
+ std::process::exit(0);
+ }
+ _ => {
+ eprintln!("Remote scan connection closed or sent unexpected input.");
+ std::process::exit(1);
+ }
+ }
+ }
+ })?;
+ }
+ disktree_core::remote::serve_request(
+ request,
&mut BufWriter::new(io::stdout().lock()),
)
} else if args.is_empty() || args == ["--help"] || args == ["-h"] {
@@ -21,3 +58,40 @@ fn main() -> io::Result<()> {
Err(io::Error::other("unknown option; try --help"))
}
}
+
+/// Kernel-enforced, not a sampled RSS check that can miss an allocation
+/// burst. Keep this in the helper: never constrain the desktop process.
+#[cfg(target_os = "linux")]
+fn limit_resources() -> io::Result<()> {
+ use rustix::process::{Resource, Rlimit, getrlimit, setrlimit};
+
+ const MAX_ADDRESS_SPACE: u64 = 1 << 30;
+ let inherited = getrlimit(Resource::As);
+ let maximum = inherited.maximum.unwrap_or(u64::MAX).min(MAX_ADDRESS_SPACE);
+ let current = inherited.current.unwrap_or(u64::MAX).min(maximum);
+ setrlimit(
+ Resource::As,
+ Rlimit {
+ current: Some(current),
+ maximum: Some(maximum),
+ },
+ )?;
+ // An allocation failure must not write a potentially huge core dump.
+ setrlimit(
+ Resource::Core,
+ Rlimit {
+ current: Some(0),
+ maximum: Some(0),
+ },
+ )?;
+ eprintln!(
+ "Remote scanner memory limit: 1 GiB address space; narrow the scan if exhausted."
+ );
+ Ok(())
+}
+
+#[cfg(not(target_os = "linux"))]
+#[allow(clippy::unnecessary_wraps, reason = "same fallible API as Linux")]
+const fn limit_resources() -> io::Result<()> {
+ Ok(())
+}
diff --git a/crates/disktree-scan/tests/stdio.rs b/crates/disktree-scan/tests/stdio.rs
index 19910bc..a5bc79f 100644
--- a/crates/disktree-scan/tests/stdio.rs
+++ b/crates/disktree-scan/tests/stdio.rs
@@ -1,5 +1,5 @@
//! Exercise the shipped executable, not just its protocol functions.
-use std::io::{BufReader, Write as _};
+use std::io::{BufRead as _, BufReader, Write as _};
use std::process::{Command, Stdio};
use disktree_core::remote::{Request, read_snapshot};
@@ -37,3 +37,167 @@ fn executable_returns_a_read_only_snapshot() {
assert_eq!(snapshot.header.root, dir.path().canonicalize().unwrap());
assert_eq!(std::fs::read(file).unwrap(), b"hello");
}
+
+// Every failure path still kills and reaps the child.
+struct Helper(std::process::Child);
+
+impl Drop for Helper {
+ fn drop(&mut self) {
+ let _ = self.0.kill();
+ let _ = self.0.wait();
+ }
+}
+
+fn supervised_helper() -> Helper {
+ Helper(
+ Command::new(env!("CARGO_BIN_EXE_disktree-scan"))
+ .arg("--stdio-v1")
+ .stdin(Stdio::piped())
+ .stdout(Stdio::piped())
+ .stderr(Stdio::piped())
+ .spawn()
+ .unwrap(),
+ )
+}
+
+fn wait_bounded(child: &mut std::process::Child) {
+ let deadline =
+ std::time::Instant::now() + std::time::Duration::from_secs(3);
+ loop {
+ if let Some(status) = child.try_wait().unwrap() {
+ assert!(status.success(), "{status}");
+ return;
+ }
+ assert!(
+ std::time::Instant::now() < deadline,
+ "helper outlived its client"
+ );
+ std::thread::yield_now();
+ }
+}
+
+#[test]
+fn disconnect_stops_a_helper_even_when_its_output_is_blocked() {
+ let dir = tempfile::tempdir().unwrap();
+ // More records than a pipe can hold, without a large or expensive scan.
+ for index in 0..2000 {
+ std::fs::write(dir.path().join(format!("entry-{index:04}")), [])
+ .unwrap();
+ }
+ let mut helper = supervised_helper();
+ let mut input = helper.0.stdin.take().unwrap();
+ let request = serde_json::json!({
+ "root": dir.path(), "disk": false,
+ "options": ScanOptions::default(), "watch_disconnect": true,
+ });
+ serde_json::to_writer(&mut input, &request).unwrap();
+ input.write_all(b"\n").unwrap();
+ let mut output = BufReader::new(helper.0.stdout.take().unwrap());
+ let mut header = String::new();
+ output.read_line(&mut header).unwrap();
+ assert!(header.contains("\"version\""));
+ // Leave stdout open and unread: only the stdin watchdog can stop it.
+ drop(input);
+ wait_bounded(&mut helper.0);
+}
+
+#[test]
+fn a_live_supervised_client_receives_a_complete_snapshot() {
+ let dir = tempfile::tempdir().unwrap();
+ std::fs::write(dir.path().join("data"), b"hello").unwrap();
+ let mut helper = supervised_helper();
+ let mut input = helper.0.stdin.take().unwrap();
+ let request = serde_json::json!({
+ "root": dir.path(), "disk": false,
+ "options": ScanOptions::default(), "watch_disconnect": true,
+ });
+ serde_json::to_writer(&mut input, &request).unwrap();
+ input.write_all(b"\n").unwrap();
+ let snapshot =
+ read_snapshot(&mut BufReader::new(helper.0.stdout.take().unwrap()))
+ .unwrap();
+ assert_eq!(snapshot.tree.files, 1);
+ assert!(helper.0.wait().unwrap().success());
+ drop(input);
+}
+
+#[cfg(target_os = "linux")]
+#[test]
+fn memory_and_core_limits_are_installed_before_reading_a_request() {
+ let mut helper = supervised_helper();
+ let mut notice = String::new();
+ BufReader::new(helper.0.stderr.take().unwrap())
+ .read_line(&mut notice)
+ .unwrap();
+ assert!(notice.contains("memory limit"));
+ let limits =
+ std::fs::read_to_string(format!("/proc/{}/limits", helper.0.id()))
+ .unwrap();
+ let address = limits
+ .lines()
+ .find(|line| line.starts_with("Max address space"))
+ .unwrap();
+ let values: Vec<_> = address.split_whitespace().collect();
+ assert!(values[3].parse::<u64>().unwrap() <= 1 << 30);
+ assert!(values[4].parse::<u64>().unwrap() <= 1 << 30);
+ let core = limits
+ .lines()
+ .find(|line| line.starts_with("Max core file size"))
+ .unwrap();
+ let values: Vec<_> = core.split_whitespace().collect();
+ assert_eq!(values[4], "0");
+ assert_eq!(values[5], "0");
+}
+
+#[cfg(target_os = "linux")]
+#[test]
+#[ignore = "creates 100,000 local empty files to measure helper RSS"]
+fn wide_stream_has_bounded_resident_memory() {
+ let dir = tempfile::tempdir().unwrap();
+ for index in 0..100_000 {
+ std::fs::File::create(dir.path().join(format!("file-{index:06}")))
+ .unwrap();
+ }
+ let mut helper = supervised_helper();
+ let mut input = helper.0.stdin.take().unwrap();
+ serde_json::to_writer(
+ &mut input,
+ &serde_json::json!({
+ "root": dir.path(), "disk": false,
+ "options": ScanOptions::default(), "watch_disconnect": true,
+ }),
+ )
+ .unwrap();
+ input.write_all(b"\n").unwrap();
+ let mut output = BufReader::new(helper.0.stdout.take().unwrap());
+ let mut line = String::new();
+ for sample in 0..2 {
+ for _ in 0..20_000 {
+ line.clear();
+ assert!(output.read_line(&mut line).unwrap() > 0);
+ }
+ // Backpressure keeps the helper alive here; samples do not retain
+ // the received records or run any expensive remote filesystem.
+ let status =
+ std::fs::read_to_string(format!("/proc/{}/status", helper.0.id()))
+ .unwrap();
+ let rss = status
+ .lines()
+ .find(|line| line.starts_with("VmRSS:"))
+ .unwrap();
+ let kib = rss
+ .split_whitespace()
+ .nth(1)
+ .unwrap()
+ .parse::<u64>()
+ .unwrap();
+ eprintln!(
+ "helper RSS after {} records: {kib} KiB",
+ (sample + 1) * 20_000
+ );
+ assert!(kib < 16 * 1024, "{rss}");
+ }
+ std::io::copy(&mut output, &mut std::io::sink()).unwrap();
+ assert!(helper.0.wait().unwrap().success());
+ drop(input);
+}