A modern terminal with a built-in multiplexer: close it and your sessions keep running. Run shells and tons of AI agents across your machines, then pick them up from desktop, browser, TUI and more. Written in Rust with no UI framework: interface drawn straight on the GPU. No Electron, GPUI or webview.
Rust agentagent-orchestrationaiai-agentsclaude-codecodexcoding-agentlinuxmacosmultiplayermultiplexerrustterminalterminal-emulatorterminal-multiplexertmuxtuiweztermwindows
fix(tailnet): stop DERP runners spinning and keep regions up
RoversX5f32be7f60e9
diff --git a/Cargo.toml b/Cargo.tomlindex c7fd4a20..58529562 100644--- a/Cargo.toml+++ b/Cargo.toml@@ -454,8 +454,11 @@ cairo-sys-rs = {path="deps/cairo", version="0.18.0"}# Vendored to bound allocations by the bytes present, not by declared lengths.varbincode = {path="deps/varbincode", version="0.1.0"}# Vendored to list every peer in the tailnet (Device::peers), which 0.6.1-# only offers as lookups by name or address, and so that a forward to an-# actor that is gone is an error rather than a panic.+# only offers as lookups by name or address, so that a forward to an actor+# that is gone is an error rather than a panic, and so that a DERP region's+# client goes with its actor, waits out a failure and moves on to the+# region's next server instead of spinning or giving up (see+# deps/tailscale-rs/PATCHES.txt).tailscale = {path="deps/tailscale-rs/tailscale", version="0.6.1"}ts_runtime = {path="deps/tailscale-rs/ts_runtime", version="0.6.1"}# Vendored so that a connection notices a peer gone without a word, and adiff --git a/deps/tailscale-rs/PATCHES.txt b/deps/tailscale-rs/PATCHES.txtindex be7bec5c..b8473b0b 100644--- a/deps/tailscale-rs/PATCHES.txt+++ b/deps/tailscale-rs/PATCHES.txt@@ -3,8 +3,8 @@ Patches to the vendored tailscale-rs cratesWhat is here-------------The folders here -- tailscale, ts_runtime, ts_control and ts_netstack_smoltcp_core -- are-copies of the tailscale-rs crates of the same names, version 0.6.1 from crates.io, with+The folders here -- tailscale, ts_runtime, ts_control, ts_netstack_smoltcp_core and ts_derp --+are copies of the tailscale-rs crates of the same names, version 0.6.1 from crates.io, withThinkTerm's changes on top. The root Cargo.toml points crates.io's versions at them([patch.crates-io]) and keeps them out of the workspace (exclude). Every other tailscale-rscrate (ts_keys, ts_tunnel, ...) still comes from crates.io, held at the same version by@@ -22,15 +22,15 @@ below:The whole difference from upstream, at any time, against a fresh copy of 0.6.1:- for c in tailscale ts_runtime ts_control ts_netstack_smoltcp_core; do+ for c in tailscale ts_runtime ts_control ts_netstack_smoltcp_core ts_derp; docurl -sL "https://static.crates.io/crates/$c/$c-0.6.1.crate" | tar xz -C /tmpdiff -ru "/tmp/$c-0.6.1" "deps/tailscale-rs/$c"doneUpstream files this copy does not carry (README.md, examples, tests, Cargo.lock,Cargo.toml.orig, .cargo_vcs_info.json) show up as "Only in", and so do the LICENSE and-PATENTS that ts_runtime, ts_control and ts_netstack_smoltcp_core got from the tailscale-crate, as they ship without them. All of that is expected.+PATENTS that ts_runtime, ts_control, ts_netstack_smoltcp_core and ts_derp got from the+tailscale crate, as they ship without them. All of that is expected.The changes-----------@@ -75,35 +75,110 @@ lost peers (ts_netstack_smoltcp_core)retrying forever.Drop when: upstream sets a timeout and a keep-alive on its sockets.-build (all four)+derp dial (ts_derp, ts_runtime)+ A DERP region whose servers cannot be reached, or whose TLS handshake fails, is an error+ for its client; 0.6.1 unwraps the outcome of the dial and panics. A connection gets 15 s+ to come up, as none of its steps has a limit of its own. After a failure the region's+ runner waits and tries again, from 1 s doubling to 30 s, starting from 1 s again once a+ connection held for 10 s. While it waits, what is queued for the region is dropped as it+ comes, and the region becoming home ends the wait. 0.6.1's runner returned on any error,+ leaving the region without a client. A try after a failed one starts with the region's+ next server: the dial goes down a region's servers only up to the first that takes the+ TCP connection, so one that then failed the TLS handshake, or said nothing till the 15 s+ ran out, kept the others from ever being tried.+ Drop when: upstream's DERP client reports a failed dial and its runner retries.++derp frames (ts_derp)+ A frame from a DERP server too short for its type -- a PeerGone without its reason byte,+ as servers from before the reason send, or a packet without its source key -- is passed+ over, and an unknown PeerGone reason only logged; 0.6.1 unwraps the first two, panicking,+ and ends the connection over the third.+ Drop when: upstream's client takes such frames in its stride.++derp reads (ts_derp)+ A DERP connection that has gone 120 s without a frame from the server counts as gone: the+ server sends a keep-alive whenever it has had nothing else to send for a minute, and Go's+ client allows as long. 0.6.1 waited for good, so a connection that died while nothing was+ sent on it -- the machine slept, the network changed -- held its runner, often the home+ region's, and the node got nothing more through it.+ Drop when: upstream's client times its reads out.++derp runner (ts_runtime)+ A region's runner ends with its Uniderp: Uniderp::on_stop stops it, and the runner leaves+ by itself once the Uniderp's end of the home-derp watch is gone. kameo leaves a linked task+ running when a supervised actor is restarted, stops normally or goes down with its+ supervisor, and 0.6.1's runner then spun at full speed on the closed watch for as long as+ the process lived. The runner also ends when the dataplane closes its queue, which 0.6.1+ took for activity and connected over, again and again with no wait. A runner ending while+ its Uniderp lives fails the Uniderp, which is restarted with a new transport; by kameo's+ default the Uniderp stayed up with no runner, and the region had no client.+ A Uniderp that stops leaves its queue to a task that empties it until the dataplane goes:+ routes lead to the transport until they are redone, and the dataplane unwraps each send,+ so a packet coming meanwhile panicked it and it restarted empty. A Uniderp kameo restarts+ asks the bus for the latest DERP latency measurement (RetainedBus's GetRetained::new):+ keeping its actor id, it is not sent what the bus retains, and a restarted home region+ took itself for any other until the DERP map changed.+ Drop when: upstream ties the runner's lifetime to its Uniderp's.++derp regions (ts_runtime)+ A region whose Uniderp failed too often to be restarted is given up on its own, and+ started once more a minute later if the DERP map still has it; by kameo's default+ Multiderp stopped with it, and every other region's client went down too. Given up again+ within 30 minutes of that, it waits twice as long each time, up to 30 minutes, as every+ start takes a transport the dataplane keeps for good; DERP maps that come meanwhile do not+ start it early. A Uniderp whose region left the DERP map stops normally and is not+ restarted (0.6.1 restarted it five times, each taking a transport); should the map have+ the region again by the time it is gone, it is started again at once.+ Multiderp keeps its own record of each region's Uniderp, from its spawn until it is gone+ for good, instead of looking it up in the registry: a Uniderp is there only once started,+ so a map coming meanwhile started a second one for the region, and one gone for good+ keeps its name there, so its region never came back. Multiderp also takes the region's+ transport out of its map itself once the Uniderp is gone for good. Uniderp::on_stop did,+ but kameo tells the supervisor before it runs on_stop: the Uniderp restarted in its place+ could put its transport in first, and out it went.+ Drop when: upstream's Multiderp outlives a region it gave up and starts it again.++build (all five)#![doc = include_str!("../README.md")] became a one-line //! doc: the READMEs are notvendored (the root .gitignore drops *.md). In Cargo.toml, tailscale loses its[[example]], [[test]] and [profile.bench] sections, ts_netstack_smoltcp_core its[dev-dependencies], and ts_runtime gains tokio's "time" feature. Upstream's proptestmodule in ts_runtime/src/peer_tracker/peer_db.rs is switched off, as proptest is not a- dependency here.+ dependency here. ts_derp loses its [[example]] sections, as the examples are not vendored,+ and gains tokio's "time" and "io-util" features.Tests-----The additions above are called from thinkterm-tailnet, so losing one in a merge fails thebuild. The changes in behaviour have tests:- cargo test -p ts_runtime -p ts_control -p ts_netstack_smoltcp_core --lib+ cargo test -p ts_runtime -p ts_control -p ts_netstack_smoltcp_core -p ts_derp --liblost peers socket_impl::tcp::tests (connect and listen)device info node::testssign-in state control_runner::tests (the reason given)no panic env::tests+ derp dial multiderp::uniderp::tests (an unreachable region, becoming home, a+ connection that does not come up, the next server)+ derp reads client::tests in ts_derp (a connection gone quiet, keep-alives)+ derp runner multiderp::uniderp::tests (a runner whose Uniderp is gone, whose queue+ is closed, that ended; a region given up; the transport of a Uniderp+ gone; a Uniderp restarted in the home region)+ derp regions multiderp::tests (the waits); multiderp::uniderp::tests (a region given+ up, waiting out its wait, getting one Uniderp, dropped from the map and+ put back)Not covered by a test: the peer tracker's answer for an empty tailnet, and the Failed statetaking the place of the stop. Signing in to a real tailnet after an upgrade checks the-first; signing in with a wrong auth key checks the second.+first; signing in with a wrong auth key checks the second. Nor are the parts of derp dial+and derp frames that need a working DERP server (the waits starting over after a connection+that held, frames the server sends).Upgrading---------1. Read upstream's CHANGELOG.md for the new version: its security fixes, and anything thattouches the groups above. A group upstream now covers can be dropped instead of merged.-2. For each of the four crates, merge three ways, file by file: the base is upstream's old+2. For each of the five crates, merge three ways, file by file: the base is upstream's oldversion, ours is deps/tailscale-rs/<crate>, theirs is upstream's new version. With bothversions unpacked in /tmp as above:@@ -113,7 +188,7 @@ UpgradingChanges that do not overlap merge by themselves; overlapping ones are left betweenconflict markers. Copy files upstream added; move by hand our changes in a file upstreamremoved or renamed. For Cargo.toml, take upstream's new one and redo the build changes.-3. Bump the version: the four entries under [patch.crates-io] in the root Cargo.toml and+3. Bump the version: the five entries under [patch.crates-io] in the root Cargo.toml andthe two in thinkterm-tailnet/Cargo.toml, then `cargo update` the other tailscale-rscrates to the same version. All tailscale-rs crates move together.4. Build, run the tests above and thinkterm-tailnet's, then sign in to a real tailnet anddiff --git a/deps/tailscale-rs/ts_runtime/src/multiderp/mod.rs b/deps/tailscale-rs/ts_runtime/src/multiderp/mod.rsindex 048f9d5f..3a38a85c 100644--- a/deps/tailscale-rs/ts_runtime/src/multiderp/mod.rs+++ b/deps/tailscale-rs/ts_runtime/src/multiderp/mod.rs@@ -2,17 +2,19 @@ mod uniderp;use std::{collections::HashMap,+ ops::ControlFlow,sync::Arc,time::{Duration, Instant},};use kameo::{- actor::{ActorRef, Spawn, WeakActorRef},+ actor::{ActorId, ActorRef, Spawn, WeakActorRef},error::ActorStopReason,message::{Context, Message},+ supervision::RestartPolicy,};use kameo_actors::scheduler::SetTimeout;-use ts_control::DerpRegion;+use ts_control::{DerpMap, DerpRegion};use ts_derp::RegionId;use ts_packet::PacketMut;use ts_transport::UnderlayTransportId;@@ -37,6 +39,17 @@ pub struct Multiderp {env: Env,debounce_state: DebounceState,peer_state: Arc<PeerDb>,+ // ThinkTerm (derp regions): added.+ /// The Uniderps this actor supervises and their regions, told apart from its other links+ /// in `on_link_died`. A region has a Uniderp from the moment it is spawned until it is gone+ /// for good, restarts included.+ regions: HashMap<ActorId, RegionId>,+ // ThinkTerm (derp regions): added.+ /// The latest DERP map, for starting a region once more. A copy: the update it came in+ /// may carry the whole netmap, which holding the update would keep.+ derp_map: DerpMap,+ // ThinkTerm (derp regions): added.+ given_up: GivenUp,}impl kameo::Actor for Multiderp {@@ -55,6 +68,9 @@ impl kameo::Actor for Multiderp {region_map: Default::default(),debounce_state: Default::default(),peer_state: Default::default(),+ regions: Default::default(),+ derp_map: Default::default(),+ given_up: Default::default(),})}@@ -67,23 +83,89 @@ impl kameo::Actor for Multiderp {Ok(())}++ // ThinkTerm (derp regions): added.+ /// A Uniderp gone for good: kameo calls this for one only when it does not restart it.+ ///+ /// One that failed too often to be restarted is given up on its own, and its region started+ /// once more a while later. By kameo's default, a link dying for any reason but a normal+ /// stop stops this actor as well, and every other region's client with it. One that+ /// stopped normally left the DERP map; should the map have the region again by now, it is+ /// started again at once, as `ensure_region` passed it over while the Uniderp was stopping.+ async fn on_link_died(+ &mut self,+ slf: WeakActorRef<Self>,+ id: ActorId,+ reason: ActorStopReason,+ ) -> Result<ControlFlow<ActorStopReason>, Self::Error> {+ let Some(region_id) = self.regions.remove(&id) else {+ // kameo's default.+ return Ok(match reason {+ ActorStopReason::Normal | ActorStopReason::SupervisorRestart => {+ ControlFlow::Continue(())+ }+ reason => ControlFlow::Break(ActorStopReason::LinkDied {+ id,+ reason: Box::new(reason),+ }),+ });+ };+ let Some(slf) = slf.upgrade() else {+ return Ok(ControlFlow::Continue(()));+ };++ // The region's transport went with its Uniderp. Taken out here rather than by the+ // Uniderp as it stopped: kameo tells the supervisor before it runs the actor's+ // `on_stop`, the Uniderp restarted in its place could put its transport in first, and+ // out it went with the old one's.+ if self.region_map.remove(®ion_id).is_some() {+ self.debounce_map_publish(&slf).await;+ }++ if reason.is_normal() {+ if let Some(region) = self.derp_map.get(®ion_id).cloned() {+ self.ensure_region(&slf, region_id, ®ion).await;+ }+ } else {+ let wait = self.given_up.wait(region_id, Instant::now());+ tracing::warn!(%region_id, %reason, ?wait, "derp region given up for a while");+ let retry = SetTimeout::new(slf.downgrade(), wait, RetryRegion(region_id));+ if let Err(e) = self.env.scheduler.ask(retry).await {+ tracing::warn!(error = ?e, %region_id, "derp region given up for good");+ }+ }+ Ok(ControlFlow::Continue(()))+ }}impl Multiderp {+ // ThinkTerm (derp regions): added.+ /// How long a region given up waits before it is started once more, at first and at most.+ /// Each start takes a transport the dataplane keeps for good, and a child kameo keeps a+ /// record of for as long as this actor lives, so a region that keeps being given up waits+ /// longer each time (see `GivenUp`).+ const REGION_RETRY: Duration = if cfg!(test) {+ Duration::from_secs(1)+ } else {+ Duration::from_secs(60)+ };+ const REGION_RETRY_MAX: Duration = if cfg!(test) {+ Duration::from_secs(30)+ } else {+ Duration::from_secs(30 * 60)+ };+#[tracing::instrument(skip_all, fields(region_id = %id))]async fn ensure_region(&mut self, slf: &ActorRef<Self>, id: RegionId, region: &DerpRegion) {- if self- .env- .lookup_opt::<Uniderp>(Some(Uniderp::name(id)))- .await- .unwrap()- .is_some()- {+ // ThinkTerm (derp regions): was a lookup of the region's Uniderp in the registry, which+ // has it only once it is started: a DERP map coming meanwhile started a second one for+ // the region. Nor does a Uniderp gone for good leave the registry.+ if self.regions.values().any(|region_id| *region_id == id) {return;}tracing::trace!("spawn new uniderp");- Uniderp::supervise(+ let uniderp = Uniderp::supervise(slf,uniderp::Args {env: self.env.clone(),@@ -91,8 +173,14 @@ impl Multiderp {region_id: id,},)+ // ThinkTerm (derp regions): a Uniderp stops normally when its region leaves the DERP+ // map; restarted, as kameo's default policy has it, it found the region gone and+ // stopped again, five times over, each time taking a transport from the dataplane.+ .restart_policy(RestartPolicy::Transient).spawn().await;+ // ThinkTerm (derp regions): see `on_link_died`.+ self.regions.insert(uniderp.id(), id);}/// Trailing-edge debounce for region id -> transport id map updates.@@ -142,13 +230,69 @@ impl Message<Arc<ts_control::StateUpdate>> for Multiderp {let Some(derp_map) = &msg.derp else {return;};+ // ThinkTerm (derp regions): kept for `on_link_died` and `RetryRegion`; a region gone+ // from the map is no longer one given up.+ self.derp_map = derp_map.clone();+ self.given_up.forget_all_but(derp_map);+ let now = Instant::now();for (id, region) in derp_map {+ // ThinkTerm (derp regions): a region given up waits out its wait, whatever maps+ // come meanwhile; `RetryRegion` starts it.+ if self.given_up.waiting(*id, now) {+ continue;+ }self.ensure_region(ctx.actor_ref(), *id, region).await;}}}+// ThinkTerm (derp regions): added.+/// Start a region given up once more, if the DERP map still has it.+struct RetryRegion(RegionId);++impl Message<RetryRegion> for Multiderp {+ type Reply = ();++ async fn handle(&mut self, RetryRegion(id): RetryRegion, ctx: &mut Context<Self, Self::Reply>) {+ let Some(region) = self.derp_map.get(&id).cloned() else {+ return;+ };+ self.ensure_region(ctx.actor_ref(), id, ®ion).await;+ }+}++// ThinkTerm (derp regions): added.+/// The regions given up, each with the wait it was given last and when.+#[derive(Default)]+struct GivenUp(HashMap<RegionId, (Duration, Instant)>);++impl GivenUp {+ /// The wait of region `id`, given up at `now`: twice the one before, up to the longest,+ /// when the region was given up again before it held for the longest wait since it was+ /// started once more; the first otherwise.+ fn wait(&mut self, id: RegionId, now: Instant) -> Duration {+ let wait = match self.0.get(&id) {+ Some(&(wait, at)) if now < at + wait + Multiderp::REGION_RETRY_MAX => {+ (wait * 2).min(Multiderp::REGION_RETRY_MAX)+ }+ _ => Multiderp::REGION_RETRY,+ };+ self.0.insert(id, (wait, now));+ wait+ }++ /// Whether region `id` is still waiting out its wait at `now`.+ fn waiting(&self, id: RegionId, now: Instant) -> bool {+ self.0.get(&id).is_some_and(|&(wait, at)| now < at + wait)+ }++ /// Forget the regions `derp_map` does not have.+ fn forget_all_but(&mut self, derp_map: &DerpMap) {+ self.0.retain(|id, _| derp_map.contains_key(id));+ }+}+#[derive(Clone, Debug, Default)]pub struct DerpTransportMap(pub Arc<HashMap<RegionId, UnderlayTransportId>>);@@ -250,3 +394,68 @@ impl Message<SendDisco> for Multiderp {tracing::trace!(?sent, %nodekey, ?derp_region, "disco send result");}}++// ThinkTerm (derp regions): the tests are ours.+#[cfg(test)]+mod tests {+ use core::num::NonZeroU32;++ use super::*;++ #[test]+ fn a_region_given_up_again_soon_waits_longer() {+ let region = RegionId(NonZeroU32::new(1).unwrap());+ let other = RegionId(NonZeroU32::new(2).unwrap());+ let mut given_up = GivenUp::default();+ let mut now = Instant::now();++ let mut waits = vec![];+ for _ in 0..7 {+ let wait = given_up.wait(region, now);+ assert!(given_up.waiting(region, now));+ assert!(!given_up.waiting(region, now + wait));+ waits.push(wait.as_secs() / Multiderp::REGION_RETRY.as_secs());+ // Started once more after the wait, and given up again soon after.+ now += wait + Duration::from_secs(3);+ }+ assert_eq!(waits, [1, 2, 4, 8, 16, 30, 30]);++ // Another region starts from the first wait.+ assert_eq!(given_up.wait(other, now), Multiderp::REGION_RETRY);++ // Given up after it held for the longest wait: the first wait again.+ now += Multiderp::REGION_RETRY_MAX * 2 + Duration::from_secs(1);+ assert_eq!(given_up.wait(region, now), Multiderp::REGION_RETRY);+ }++ #[test]+ fn a_region_that_left_the_derp_map_is_forgotten_as_given_up() {+ let gone = RegionId(NonZeroU32::new(1).unwrap());+ let kept = RegionId(NonZeroU32::new(2).unwrap());+ let derp_map: DerpMap = [(+ kept,+ DerpRegion {+ info: ts_derp::RegionInfo {+ name: "test".into(),+ code: "t".into(),+ no_measure_no_home: true,+ },+ servers: vec![],+ },+ )]+ .into_iter()+ .collect();+ let mut given_up = GivenUp::default();+ let now = Instant::now();+ for region in [gone, kept] {+ given_up.wait(region, now);+ given_up.wait(region, now);+ }++ given_up.forget_all_but(&derp_map);+ assert!(!given_up.waiting(gone, now));+ assert_eq!(given_up.wait(gone, now), Multiderp::REGION_RETRY);+ assert!(given_up.waiting(kept, now));+ assert_eq!(given_up.wait(kept, now), Multiderp::REGION_RETRY * 4);+ }+}diff --git a/deps/tailscale-rs/ts_runtime/src/multiderp/uniderp.rs b/deps/tailscale-rs/ts_runtime/src/multiderp/uniderp.rsindex 9354414f..61f28522 100644--- a/deps/tailscale-rs/ts_runtime/src/multiderp/uniderp.rs+++ b/deps/tailscale-rs/ts_runtime/src/multiderp/uniderp.rs@@ -1,11 +1,12 @@use std::{+ ops::ControlFlow,sync::{Arc, RwLock},time::{Duration, Instant},};use futures::FutureExt;use kameo::{- actor::{ActorRef, Spawn, WeakActorRef},+ actor::{ActorId, ActorRef, Spawn, WeakActorRef},error::ActorStopReason,message::{Context, Message},};@@ -25,6 +26,7 @@ use crate::{env::Env,multiderp::{Multiderp, SetRegionTransportId},peer_tracker::{PeerDb, PeerState},+ retained_bus::GetRetained,task::ErasedTask,};@@ -50,8 +52,7 @@ pub struct Uniderp {task: ActorRef<ErasedTask>,home_derp_tx: watch::Sender<bool>,-- env: Env,+ // ThinkTerm (derp regions): `env` went, with the telling of the Multiderp in `on_stop`.}impl Uniderp {@@ -113,6 +114,21 @@ impl kameo::Actor for Uniderp {args.env.subscribe::<Arc<PeerState>>(&slf).await?;args.env.subscribe::<DerpLatencyMeasurement>(&slf).await?;+ // ThinkTerm (derp runner): a Uniderp kameo restarts keeps its actor id, and the bus+ // hands what it retains only to ids new to it. The measurement that made the region+ // home came before the restart, and none comes again until the DERP map changes: the+ // restarted home region took itself for any other, its connection idled out, and the+ // node got nothing more through DERP.+ if let Ok(Some(measurement)) = args+ .env+ .bus+ .ask(GetRetained::<DerpLatencyMeasurement>::new())+ .await+ && let Some(home) = measurement.measurement.first()+ {+ home_derp_tx.send_replace(home.id == args.region_id);+ }+args.env.register(Some(Self::name(args.region_id)), &slf).await?;@@ -122,7 +138,6 @@ impl kameo::Actor for Uniderp {transport_id,home_derp_tx,task,- env: args.env,})}@@ -131,15 +146,56 @@ impl kameo::Actor for Uniderp {_: WeakActorRef<Self>,_: ActorStopReason,) -> Result<(), Self::Error> {- self.env- .tell::<Multiderp, _>(- None,- SetRegionTransportId(self.runner_state.region_id, None),- )- .await?;+ // ThinkTerm (derp runner): the runner goes with this actor. Being linked to it is not+ // enough: kameo leaves a linked task running when a supervised actor is restarted, stops+ // normally or goes down with its supervisor. Stopped, not killed: the task tells its+ // link how it ended, which reaches the actor restarted in this one's place (same id,+ // same mailbox), and a kill would stop that one too.+ let _ = self.task.stop_gracefully().await;++ // ThinkTerm (derp runner): the queue's receiving end goes to a task that empties it+ // until the dataplane drops the other end, which it does only along with itself. Routes+ // lead here until the Multiderp has taken the transport out of its map and the routes+ // are redone, and the dataplane unwraps each send: a packet coming meanwhile panicked+ // it, and it restarted empty, carrying nothing more for the node. One parked task per+ // Uniderp stopped, as many as the transports the dataplane keeps anyway.+ let queue = self.runner_state.from_dataplane.clone();+ tokio::spawn(async move {+ let mut queue = queue.lock().await;+ while queue.recv().await.is_some() {}+ });++ // ThinkTerm (derp regions): this told the Multiderp to take the region's transport out+ // of its map, which now does so itself once the Uniderp is gone for good; see its+ // `on_link_died`.Ok(())}++ // ThinkTerm (derp runner): added.+ /// The runner ending while this actor lives means the dataplane closed its queue (see+ /// `Runner::wait_for_activity`): the region has no client, and its transport is of no more+ /// use. Failing has the actor restarted with a new one; by kameo's default, a runner that+ /// ended cleanly left this running without one. Any other link goes by the default,+ /// including a runner of the actor this one was restarted in place of (same mailbox).+ async fn on_link_died(+ &mut self,+ _: WeakActorRef<Self>,+ id: ActorId,+ reason: ActorStopReason,+ ) -> Result<ControlFlow<ActorStopReason>, Self::Error> {+ Ok(match reason {+ ActorStopReason::Normal | ActorStopReason::SupervisorRestart+ if id != self.task.id() =>+ {+ ControlFlow::Continue(())+ }+ reason => ControlFlow::Break(ActorStopReason::LinkDied {+ id,+ reason: Box::new(reason),+ }),+ })+ }}impl Message<Arc<ts_control::StateUpdate>> for Uniderp {@@ -228,48 +284,169 @@ struct Runner {keys: NodeKeyPair,}+// ThinkTerm (derp runner): added.+/// What [`Runner::wait_for_activity`] found to do.+enum Activity {+ /// Connect, sending this packet first if there is one.+ Connect(Option<(DynEndpoint, Vec<PacketMut>)>),+ /// The Uniderp this runner belongs to, or the dataplane queue it serves, is gone, and the+ /// runner goes with it.+ Gone,+}+impl Runner {const INACTIVITY_TIMEOUT: Duration = Duration::from_secs(10);+ // ThinkTerm (derp dial): added.+ /// How long to wait before trying the region again after it failed, at first and at most:+ /// the wait doubles with every failure in a row.+ const RETRY_FIRST: Duration = Duration::from_secs(1);+ const RETRY_MAX: Duration = Duration::from_secs(30);+ /// A connection that held this long worked: when it fails, the waits start from the first+ /// again. Counting only connections that end cleanly, a home region's, which never idles+ /// out, left every reconnect at the longest wait once a few had failed.+ const STABLE: Duration = Duration::from_secs(10);+ /// The longest a connection may take to come up: dial, TLS, upgrade and DERP handshake.+ /// None of them has a limit of its own, and a server that takes the TCP connection and then+ /// says nothing (a captive portal, a stuck proxy) held the runner for good.+ const CONNECT_TIMEOUT: Duration = if cfg!(test) {+ Duration::from_millis(300)+ } else {+ Duration::from_secs(15)+ };#[tracing::instrument(skip_all, fields(region_id = %self.region_id))]async fn run(&mut self) -> Result<(), ts_derp::Error> {+ // ThinkTerm (derp dial): a failure is waited out and the region tried again. This+ // returned it, which left the region without a client for good, or panicked with it.+ let mut retry = Self::RETRY_FIRST;+ // ThinkTerm (derp dial): the server of the region a try starts with (see `connect`).+ let mut first = 0;+ loop {+ // ThinkTerm (derp runner): the runner ends with its Uniderp.+ let Activity::Connect(pending) = self.wait_for_activity().await else {+ return Ok(());+ };+ let connect = self.connect(first, pending);+ let result = match tokio::time::timeout(Self::CONNECT_TIMEOUT, connect).await {+ Ok(Ok(transport)) => {+ let connected = Instant::now();+ let result = self.run_transport(transport).await;+ if connected.elapsed() >= Self::STABLE {+ retry = Self::RETRY_FIRST;+ }+ result+ }+ Ok(Err(e)) => Err(e),+ Err(_) => Err(std::io::Error::new(+ std::io::ErrorKind::TimedOut,+ "the DERP connection did not come up in time",+ )+ .into()),+ };+ match result {+ Ok(()) => retry = Self::RETRY_FIRST,+ Err(e) => {+ tracing::warn!(error = %e, ?retry, "derp region failed, trying again");+ first += 1;+ if !self.wait_out(retry).await {+ return Ok(());+ }+ retry = (retry * 2).min(Self::RETRY_MAX);+ }+ }+ }+ }++ // ThinkTerm (derp dial): added.+ /// Wait `wait` before the region is tried again. What is queued for it meanwhile is dropped+ /// as it comes: stale by then, and sent again by the connections it belongs to; kept, it+ /// would pile up for the whole wait. The wait ends early when the region becomes home,+ /// which is worth a try at once. False when the Uniderp or the queue went away meanwhile.+ async fn wait_out(&mut self, wait: Duration) -> bool {+ let mut from_dataplane = self.from_dataplane.lock().await;+ let deadline = tokio::time::sleep(wait);+ tokio::pin!(deadline);loop {- let pending = self.wait_for_activity().await;- let transport = self.connect(pending).await?;- self.run_transport(transport).await?;+ tokio::select! {+ _ = &mut deadline => return true,++ changed = self.home_derp_rx.changed() => {+ if changed.is_err() {+ return false;+ }+ if *self.home_derp_rx.borrow() {+ return true;+ }+ },++ queued = from_dataplane.recv() => {+ if queued.is_none() {+ return false;+ }+ },+ }}}+ // ThinkTerm (derp runner): says what to do next rather than handing over a packet alone, as+ // the Uniderp, or the queue, may be gone.#[tracing::instrument(skip_all, level = "trace")]- async fn wait_for_activity(&mut self) -> Option<(DynEndpoint, Vec<PacketMut>)> {+ async fn wait_for_activity(&mut self) -> Activity {tracing::trace!("waiting for packet activity or for this to become home derp");let mut from_dataplane = self.from_dataplane.lock().await;+ // ThinkTerm (derp runner): the Uniderp holds the sender of `home_derp_rx`. Once it is+ // gone, `changed` fails at once, every time; taken for a wake-up, that spun this loop at+ // full speed for as long as the process lived. kameo can leave the runner of a Uniderp+ // behind (see `Uniderp::on_stop`), so the runner has to notice by itself as well. A+ // queue the dataplane closed has nothing more for this transport either; taken for+ // activity, it had the runner connect, find it closed and connect again, with no wait.+ if self.home_derp_rx.has_changed().is_err() || from_dataplane.is_closed() {+ return Activity::Gone;+ }while !*self.home_derp_rx.borrow_and_update() {tokio::select! {- _ = self.home_derp_rx.changed() => {+ changed = self.home_derp_rx.changed() => {+ if changed.is_err() {+ return Activity::Gone;+ }tracing::trace!(is_home_derp = *self.home_derp_rx.borrow());},from_net = from_dataplane.recv() => {tracing::trace!("received packet to send");- return from_net;+ // ThinkTerm (derp runner): None is the queue closing (see above).+ return match from_net {+ Some(pending) => Activity::Connect(Some(pending)),+ None => Activity::Gone,+ };}}}- None+ Activity::Connect(None)}+ // ThinkTerm (derp dial): `first` added.#[tracing::instrument(skip_all, level = "trace")]async fn connect(&self,+ first: usize,pending: Option<(DynEndpoint, Vec<PacketMut>)>,) -> Result<impl UnderlayTransport<Error = ts_derp::Error> + 'static, ts_derp::Error> {tracing::trace!("establishing derp connection");- let transport = ts_derp::DefaultClient::connect(&self.region.servers, &self.keys).await?;+ // ThinkTerm (derp dial): the servers from the `first`th on, round to the one before it;+ // `run` moves on by one after every failure. The dial goes down the servers only up to+ // the first that takes the TCP connection: one that then fails the TLS handshake ends+ // the try, and `CONNECT_TIMEOUT` can run out on one that says nothing at all. Starting+ // with the same server every time, the others were never tried.+ let servers = &self.region.servers;+ let first = first.checked_rem(servers.len()).unwrap_or(0);+ // Collected: rustc cannot tell the future holding the chain itself is `Send`.+ let servers: Vec<_> = servers[first..].iter().chain(&servers[..first]).collect();+ let transport = ts_derp::DefaultClient::connect(servers, &self.keys).await?;if let Some((info, pending)) = pending {tracing::trace!("sending queued packet");@@ -330,7 +507,12 @@ impl Runner {}},- _ = self.home_derp_rx.changed() => {+ changed = self.home_derp_rx.changed() => {+ // ThinkTerm (derp runner): the Uniderp is gone; `wait_for_activity` ends the+ // runner next.+ if changed.is_err() {+ return Ok(());+ }tracing::trace!(is_home_derp = *self.home_derp_rx.borrow());},}@@ -344,3 +526,572 @@ async fn option_timeout(duration: Option<Instant>) {None => core::future::pending().await,}}++// ThinkTerm (derp runner, derp dial, derp regions): the tests are ours.+#[cfg(test)]+mod tests {+ use core::num::NonZeroU32;+ use std::{+ collections::HashMap,+ net::Ipv4Addr,+ sync::atomic::{AtomicUsize, Ordering},+ };++ use kameo::Reply;+ use tokio::sync::mpsc;++ use super::*;++ fn server(port: u16) -> ts_derp::ServerConnInfo {+ ts_derp::ServerConnInfo {+ hostname: "localhost".into(),+ ipv4: ts_derp::IpUsage::FixedAddr(Ipv4Addr::LOCALHOST),+ ipv6: ts_derp::IpUsage::Disable,+ tls_validation_config: ts_derp::TlsValidationConfig::CommonName {+ common_name: "localhost".into(),+ },+ https_port: port,+ stun_port: None,+ stun_only: false,+ supports_port_80: false,+ }+ }++ /// A server of a region nobody can reach: a port of this machine nothing listens on.+ async fn unreachable_server() -> ts_derp::ServerConnInfo {+ let listener = tokio::net::TcpListener::bind((Ipv4Addr::LOCALHOST, 0))+ .await+ .unwrap();+ let port = listener.local_addr().unwrap().port();+ drop(listener);+ server(port)+ }++ /// A server that takes every connection and then drops it, or holds it without a word, and+ /// counts them: each is one try of the runner's.+ async fn silent_server(hold: bool) -> (ts_derp::ServerConnInfo, Arc<AtomicUsize>) {+ let listener = tokio::net::TcpListener::bind((Ipv4Addr::LOCALHOST, 0))+ .await+ .unwrap();+ let port = listener.local_addr().unwrap().port();+ let tries = Arc::new(AtomicUsize::new(0));+ let counted = tries.clone();+ tokio::spawn(async move {+ let mut held = vec![];+ while let Ok((connection, _)) = listener.accept().await {+ counted.fetch_add(1, Ordering::SeqCst);+ if hold {+ held.push(connection);+ }+ }+ });+ (server(port), tries)+ }++ fn region(id: u32, server: ts_derp::ServerConnInfo) -> (RegionId, DerpRegion) {+ let region = DerpRegion {+ info: ts_derp::RegionInfo {+ name: format!("test {id}"),+ code: format!("t{id}"),+ no_measure_no_home: true,+ },+ servers: vec![server],+ };+ (RegionId(NonZeroU32::new(id).unwrap()), region)+ }++ /// A runner as `Uniderp::on_start` makes one, with the sender of its home-derp watch (the+ /// Uniderp's) and of its queue (the dataplane's).+ fn runner(server: ts_derp::ServerConnInfo) -> (Runner, watch::Sender<bool>, Tx<ToUnderlay>) {+ let (home_derp_tx, home_derp_rx) = watch::channel(false);+ let (to_dataplane, _) = mpsc::unbounded_channel();+ let (to_underlay, from_dataplane) = mpsc::unbounded_channel();+ let (region_id, region) = region(1, server);+ let runner = Runner {+ region_id,+ region,+ transport_id: UnderlayTransportId(1),+ home_derp_rx,+ to_dataplane,+ from_dataplane: Arc::new(Mutex::new(from_dataplane)),+ peer_db: Arc::new(RwLock::new(None)),+ keys: NodeKeyPair::random(),+ };+ (runner, home_derp_tx, to_underlay)+ }++ /// Whether `tries` reaches `count` within `within`.+ async fn tried(tries: &AtomicUsize, count: usize, within: Duration) -> bool {+ let deadline = Instant::now() + within;+ while Instant::now() < deadline {+ if tries.load(Ordering::SeqCst) >= count {+ return true;+ }+ tokio::time::sleep(Duration::from_millis(10)).await;+ }+ false+ }++ #[tokio::test(flavor = "multi_thread", worker_threads = 2)]+ async fn a_runner_whose_uniderp_is_gone_ends() {+ let (mut runner, home_derp_tx, _queue) = runner(unreachable_server().await);+ drop(home_derp_tx);+ // 0.6.1 spun here for good.+ let ended = tokio::time::timeout(Duration::from_secs(2), runner.run()).await;+ assert!(matches!(ended, Ok(Ok(()))), "{ended:?}");+ }++ #[tokio::test(flavor = "multi_thread", worker_threads = 2)]+ async fn a_waiting_runner_ends_when_its_uniderp_goes() {+ let (mut runner, home_derp_tx, _queue) = runner(unreachable_server().await);+ let run = tokio::spawn(async move { runner.run().await });+ tokio::time::sleep(Duration::from_millis(100)).await;+ drop(home_derp_tx);+ let ended = tokio::time::timeout(Duration::from_secs(2), run).await;+ assert!(matches!(ended, Ok(Ok(Ok(())))), "{ended:?}");+ }++ #[tokio::test(flavor = "multi_thread", worker_threads = 2)]+ async fn a_runner_whose_queue_is_closed_ends() {+ let (mut runner, _home_derp_tx, queue) = runner(unreachable_server().await);+ drop(queue);+ let ended = tokio::time::timeout(Duration::from_secs(2), runner.run()).await;+ assert!(matches!(ended, Ok(Ok(()))), "{ended:?}");+ }++ #[tokio::test(flavor = "multi_thread", worker_threads = 2)]+ async fn an_unreachable_region_is_tried_again_rather_than_panicking() {+ let (mut runner, home_derp_tx, _queue) = runner(unreachable_server().await);+ // The home region: the runner connects at once, and keeps trying.+ home_derp_tx.send(true).unwrap();+ let run = tokio::spawn(async move { runner.run().await });+ // The first try fails at once, as nothing listens on the port (0.6.1 panicked there),+ // and the second comes a second later.+ tokio::time::sleep(Duration::from_millis(1500)).await;+ assert!(!run.is_finished(), "the runner stopped trying");+ drop(home_derp_tx);+ let ended = tokio::time::timeout(Duration::from_secs(3), run).await;+ assert!(matches!(ended, Ok(Ok(Ok(())))), "{ended:?}");+ }++ #[tokio::test(flavor = "multi_thread", worker_threads = 2)]+ async fn a_region_that_becomes_home_is_tried_at_once() {+ let (server, tries) = silent_server(false).await;+ let (mut runner, home_derp_tx, _queue) = runner(server);+ home_derp_tx.send(true).unwrap();+ let run = tokio::spawn(async move { runner.run().await });+ // The first try fails at once (the server drops the connection); the next is due a+ // second later.+ assert!(tried(&tries, 1, Duration::from_millis(500)).await);+ tokio::time::sleep(Duration::from_millis(100)).await;+ // Chosen as home: tried at once rather than at the end of the wait.+ home_derp_tx.send(true).unwrap();+ assert!(+ tried(&tries, 2, Duration::from_millis(400)).await,+ "the runner sat out its wait"+ );+ drop(home_derp_tx);+ let ended = tokio::time::timeout(Duration::from_secs(3), run).await;+ assert!(matches!(ended, Ok(Ok(Ok(())))), "{ended:?}");+ }++ #[tokio::test(flavor = "multi_thread", worker_threads = 2)]+ async fn a_connection_that_does_not_come_up_is_given_up_in_time() {+ let (server, tries) = silent_server(true).await;+ let (mut runner, home_derp_tx, _queue) = runner(server);+ home_derp_tx.send(true).unwrap();+ let run = tokio::spawn(async move { runner.run().await });+ // Given up after CONNECT_TIMEOUT, tried again after the first wait; 0.6.1 waited on+ // the first connection for good.+ assert!(+ tried(&tries, 2, Duration::from_secs(3)).await,+ "the runner hung on its first connection"+ );+ drop(home_derp_tx);+ let ended = tokio::time::timeout(Duration::from_secs(3), run).await;+ assert!(matches!(ended, Ok(Ok(Ok(())))), "{ended:?}");+ }++ /// The runner of a region whose first server fails as `first` does tries the next server+ /// on its next try: the dial goes no further than a server that takes the connection.+ async fn the_next_server_is_tried_after(first: ts_derp::ServerConnInfo) {+ let (next, tries) = silent_server(false).await;+ let (mut runner, home_derp_tx, _queue) = runner(first);+ runner.region.servers.push(next);+ home_derp_tx.send(true).unwrap();+ let run = tokio::spawn(async move { runner.run().await });+ // The first try fails on the first server; the next comes a second later.+ assert!(+ tried(&tries, 1, Duration::from_secs(3)).await,+ "the next server was never tried"+ );+ drop(home_derp_tx);+ let ended = tokio::time::timeout(Duration::from_secs(3), run).await;+ assert!(matches!(ended, Ok(Ok(Ok(())))), "{ended:?}");+ }++ #[tokio::test(flavor = "multi_thread", worker_threads = 2)]+ async fn the_next_server_is_tried_after_one_that_fails_the_handshake() {+ // It drops the connection as soon as it takes it.+ the_next_server_is_tried_after(silent_server(false).await.0).await;+ }++ #[tokio::test(flavor = "multi_thread", worker_threads = 2)]+ async fn the_next_server_is_tried_after_one_that_does_not_come_up() {+ // It holds the connection without a word, until CONNECT_TIMEOUT.+ the_next_server_is_tried_after(silent_server(true).await.0).await;+ }++ #[derive(Reply)]+ struct RunnerTask(ActorRef<ErasedTask>);++ struct GetRunnerTask;++ impl Message<GetRunnerTask> for Uniderp {+ type Reply = RunnerTask;++ async fn handle(+ &mut self,+ _: GetRunnerTask,+ _: &mut Context<Self, Self::Reply>,+ ) -> RunnerTask {+ RunnerTask(self.task.clone())+ }+ }++ /// Fails the Uniderp that handles it, as a failure that is no normal stop would.+ struct Crash;++ impl Message<Crash> for Uniderp {+ type Reply = ();++ async fn handle(&mut self, _: Crash, _: &mut Context<Self, Self::Reply>) {+ panic!("a Uniderp failing on purpose");+ }+ }++ /// The Uniderp of region `id`, as the registry has it.+ async fn uniderp(env: &Env, id: u32) -> ActorRef<Uniderp> {+ let name = Uniderp::name(RegionId(NonZeroU32::new(id).unwrap()));+ for _ in 0..100 {+ if let Ok(uniderp) = env.lookup::<Uniderp>(Some(name.clone())).await {+ return uniderp;+ }+ tokio::time::sleep(Duration::from_millis(20)).await;+ }+ panic!("no Uniderp for region {id}");+ }++ /// The running Uniderp of region `id` and its runner. A restart keeps the mailbox and makes+ /// a new ActorRef, which the new Uniderp registers once started; until then the registry+ /// answers with the one before, which does not take messages.+ async fn running_uniderp(env: &Env, id: u32) -> (ActorRef<Uniderp>, ActorRef<ErasedTask>) {+ for _ in 0..150 {+ let uniderp = uniderp(env, id).await;+ if let Ok(RunnerTask(task)) = uniderp.ask(GetRunnerTask).await {+ return (uniderp, task);+ }+ tokio::time::sleep(Duration::from_millis(20)).await;+ }+ panic!("region {id} has no running Uniderp");+ }++ #[derive(Reply)]+ struct RegionMap(HashMap<RegionId, UnderlayTransportId>);++ struct GetRegionMap;++ impl Message<GetRegionMap> for Multiderp {+ type Reply = RegionMap;++ async fn handle(+ &mut self,+ _: GetRegionMap,+ _: &mut Context<Self, Self::Reply>,+ ) -> RegionMap {+ RegionMap(self.region_map.clone())+ }+ }++ /// The regions the Multiderp has a transport for.+ async fn regions_with_a_transport(multiderp: &ActorRef<Multiderp>) -> Vec<u32> {+ let RegionMap(map) = multiderp.ask(GetRegionMap).await.unwrap();+ let mut regions: Vec<u32> = map.keys().map(|id| id.0.get()).collect();+ regions.sort();+ regions+ }++ /// A DERP map of regions `ids`, each with a server nobody can reach, as control sends it.+ async fn derp_map(ids: &[u32]) -> Arc<ts_control::StateUpdate> {+ let mut derp = ts_control::DerpMap::new();+ for &id in ids {+ let (id, region) = region(id, unreachable_server().await);+ derp.insert(id, region);+ }+ Arc::new(ts_control::StateUpdate {+ derp: Some(derp),+ node: None,+ peer_update: None,+ ping: None,+ packetfilter: None,+ pop_browser_url: None,+ dial_plan: None,+ })+ }++ /// The DERP clients of a node, as its runtime spawns them, with none of its regions yet.+ fn derp_clients() -> (Env, ActorRef<Multiderp>) {+ let env = Env::new(ts_keys::NodeState::generate());+ DataplaneActor::spawn(env.clone());+ let multiderp = Multiderp::spawn(env.clone());+ (env, multiderp)+ }++ #[tokio::test(flavor = "multi_thread", worker_threads = 2)]+ async fn a_region_given_up_takes_down_neither_its_runners_nor_the_other_regions() {+ let (env, multiderp) = derp_clients();+ env.publish(derp_map(&[1, 2]).await).await.unwrap();++ tokio::time::timeout(Duration::from_secs(10), async {+ let (second, _) = running_uniderp(&env, 2).await;++ // Failed once more than kameo restarts it within its window (5 times in 5 seconds).+ // Through the mailbox, which the restarted Uniderp takes over: a kill would go to+ // the one the ActorRef was made for.+ let mut runners = vec![];+ for _ in 0..6 {+ let (first, runner) = running_uniderp(&env, 1).await;+ runners.push(runner);+ first.tell(Crash).await.unwrap();+ }+ tokio::time::sleep(Duration::from_millis(300)).await;++ assert!(+ !uniderp(&env, 1).await.is_alive(),+ "region 1 was not given up"+ );+ assert!(multiderp.is_alive(), "Multiderp went down with region 1");+ assert!(second.is_alive(), "region 2 went down with region 1");+ for runner in &runners {+ assert!(!runner.is_alive(), "a runner outlived its Uniderp");+ }+ // Its transport went with it; the restarts left its last one in place till then.+ assert_eq!(regions_with_a_transport(&multiderp).await, [2]);++ // Started once more a while later, as the DERP map still has it.+ running_uniderp(&env, 1).await;+ assert_eq!(regions_with_a_transport(&multiderp).await, [1, 2]);+ })+ .await+ .expect("timed out");+ }++ /// Has the Multiderp take a region as given up just now, to wait this long.+ struct GivenUpFor(RegionId, Duration);++ impl Message<GivenUpFor> for Multiderp {+ type Reply = ();++ async fn handle(+ &mut self,+ GivenUpFor(id, wait): GivenUpFor,+ _: &mut Context<Self, Self::Reply>,+ ) {+ self.given_up.0.insert(id, (wait, Instant::now()));+ }+ }++ #[tokio::test(flavor = "multi_thread", worker_threads = 2)]+ async fn a_region_given_up_waits_out_its_wait_whatever_maps_come() {+ let (env, multiderp) = derp_clients();+ let given_up = RegionId(NonZeroU32::new(1).unwrap());+ multiderp+ .ask(GivenUpFor(given_up, Duration::from_secs(3600)))+ .await+ .unwrap();+ env.publish(derp_map(&[1, 2]).await).await.unwrap();++ tokio::time::timeout(Duration::from_secs(10), running_uniderp(&env, 2))+ .await+ .expect("timed out");+ tokio::time::sleep(Duration::from_millis(300)).await;++ // Each Uniderp takes a transport of the dataplane as it starts, numbered from 0.+ let (next, _, _) = env+ .ask::<DataplaneActor, _>(None, NewUnderlayTransport, true)+ .await+ .unwrap();+ assert_eq!(+ next.0, 1,+ "the region given up was started before its wait was out"+ );++ // Gone from the map and back: forgotten as given up, and started at once.+ env.publish(derp_map(&[2]).await).await.unwrap();+ env.publish(derp_map(&[1, 2]).await).await.unwrap();+ tokio::time::timeout(Duration::from_secs(10), running_uniderp(&env, 1))+ .await+ .expect("timed out");+ }++ #[tokio::test(flavor = "multi_thread", worker_threads = 2)]+ async fn a_uniderp_whose_runner_ended_is_started_anew() {+ let (env, _multiderp) = derp_clients();+ env.publish(derp_map(&[1]).await).await.unwrap();++ tokio::time::timeout(Duration::from_secs(10), async {+ let (_, ended) = running_uniderp(&env, 1).await;+ // Ended cleanly, as a runner does when the dataplane closes its queue.+ ended.stop_gracefully().await.unwrap();+ loop {+ let (_, runner) = running_uniderp(&env, 1).await;+ if runner.id() != ended.id() && runner.is_alive() {+ break;+ }+ tokio::time::sleep(Duration::from_millis(20)).await;+ }+ })+ .await+ .expect("the region was left without a runner");+ }++ #[tokio::test(flavor = "multi_thread", worker_threads = 2)]+ async fn a_region_gets_one_uniderp_however_many_maps_come_as_it_starts() {+ let (env, multiderp) = derp_clients();+ // Two maps one after the other in the Multiderp's mailbox: the second comes while the+ // Uniderps of the first are starting, which are in the registry only once started.+ // Told rather than published, for the bus may take its time with the second.+ let map = derp_map(&[1, 2]).await;+ multiderp.tell(map.clone()).await.unwrap();+ multiderp.tell(map).await.unwrap();++ tokio::time::timeout(Duration::from_secs(10), async {+ running_uniderp(&env, 1).await;+ running_uniderp(&env, 2).await;+ })+ .await+ .expect("timed out");+ tokio::time::sleep(Duration::from_millis(300)).await;++ // Each Uniderp takes a transport of the dataplane as it starts, numbered from 0.+ let (next, _, _) = env+ .ask::<DataplaneActor, _>(None, NewUnderlayTransport, true)+ .await+ .unwrap();+ assert_eq!(next.0, 2, "a region got more than one Uniderp");+ }++ #[tokio::test(flavor = "multi_thread", worker_threads = 2)]+ async fn a_region_dropped_from_the_map_and_put_back_at_once_is_started_again() {+ let (env, multiderp) = derp_clients();+ env.publish(derp_map(&[1, 2]).await).await.unwrap();+ let without = derp_map(&[2]).await;+ let with = derp_map(&[1, 2]).await;++ tokio::time::timeout(Duration::from_secs(10), async {+ let (dropped, _) = running_uniderp(&env, 1).await;+ running_uniderp(&env, 2).await;+ // Put back while its Uniderp, told by the first to stop, is still stopping.+ env.publish(without).await.unwrap();+ env.publish(with).await.unwrap();+ loop {+ let (uniderp, _) = running_uniderp(&env, 1).await;+ if uniderp.id() != dropped.id() {+ break;+ }+ tokio::time::sleep(Duration::from_millis(20)).await;+ }+ assert!(!dropped.is_alive());+ assert_eq!(regions_with_a_transport(&multiderp).await, [1, 2]);+ })+ .await+ .expect("timed out");+ }++ /// Whether the Uniderp that handles it takes its region for home.+ struct IsHome;++ impl Message<IsHome> for Uniderp {+ type Reply = bool;++ async fn handle(&mut self, _: IsHome, _: &mut Context<Self, Self::Reply>) -> bool {+ *self.home_derp_tx.borrow()+ }+ }++ /// A measurement of the DERP regions with region `id` the fastest: the node's home.+ fn home_in(id: u32) -> DerpLatencyMeasurement {+ DerpLatencyMeasurement {+ measurement: Arc::new(vec![ts_netcheck::RegionResult {+ latency: Duration::from_millis(10),+ id: RegionId(NonZeroU32::new(id).unwrap()),+ latency_map_key: format!("{id}"),+ connected_remote: (Ipv4Addr::LOCALHOST, 443).into(),+ }]),+ }+ }++ #[tokio::test(flavor = "multi_thread", worker_threads = 2)]+ async fn a_uniderp_restarted_knows_its_region_is_home() {+ let (env, _multiderp) = derp_clients();+ env.publish(derp_map(&[1]).await).await.unwrap();++ tokio::time::timeout(Duration::from_secs(10), async {+ let (uniderp, _) = running_uniderp(&env, 1).await;+ env.publish(home_in(1)).await.unwrap();+ while !uniderp.ask(IsHome).await.unwrap() {+ tokio::time::sleep(Duration::from_millis(20)).await;+ }++ // Asked after the failure, through the same mailbox: the restarted one answers.+ uniderp.tell(Crash).await.unwrap();+ let (restarted, _) = running_uniderp(&env, 1).await;+ assert!(+ restarted.ask(IsHome).await.unwrap(),+ "the restarted Uniderp took its region for any other"+ );+ })+ .await+ .expect("timed out");+ }++ #[tokio::test(flavor = "multi_thread", worker_threads = 2)]+ async fn the_transport_of_a_uniderp_gone_still_takes_what_is_sent_to_it() {+ let (env, multiderp) = derp_clients();+ env.publish(derp_map(&[1, 2]).await).await.unwrap();++ tokio::time::timeout(Duration::from_secs(10), async {+ let (gone, _) = running_uniderp(&env, 1).await;+ running_uniderp(&env, 2).await;+ let RegionMap(map) = multiderp.ask(GetRegionMap).await.unwrap();+ let transport = map[&RegionId(NonZeroU32::new(1).unwrap())];++ // Its region leaves the DERP map; whatever it held is let go.+ env.publish(derp_map(&[2]).await).await.unwrap();+ while gone.is_alive() {+ tokio::time::sleep(Duration::from_millis(20)).await;+ }+ tokio::time::sleep(Duration::from_millis(300)).await;++ // As the dataplane sends to it while routes still lead there: it unwraps the send.+ let taken = env+ .ask::<DataplaneActor, _>(+ None,+ crate::dataplane::SendUnderlay {+ underlay_id: transport,+ endpoint: DynEndpoint::derp(NodeKeyPair::random().public),+ bufs: vec![],+ },+ true,+ )+ .await+ .unwrap();+ assert!(+ taken,+ "the transport's queue is closed: a send panics the dataplane"+ );+ })+ .await+ .expect("timed out");+ }+}diff --git a/deps/tailscale-rs/ts_runtime/src/retained_bus.rs b/deps/tailscale-rs/ts_runtime/src/retained_bus.rsindex 68127781..731e697f 100644--- a/deps/tailscale-rs/ts_runtime/src/retained_bus.rs+++ b/deps/tailscale-rs/ts_runtime/src/retained_bus.rs@@ -190,6 +190,14 @@ where#[derive(Debug, Copy, Clone, Default)]pub struct GetRetained<M>(PhantomData<M>);+// ThinkTerm (derp runner): added, as the derived `Default` asks for an `M: Default`.+impl<M> GetRetained<M> {+ /// Ask for the retained value of type `M`.+ pub fn new() -> Self {+ Self(PhantomData)+ }+}+impl<M> Message<GetRetained<M>> for RetainedBuswhereM: Clone + Send + 'static,