25 MB lightweight cross-platform database client for 100+ databases, including MySQL, PostgreSQL, SQLite, Redis, MongoDB, DuckDB, SQL Server, and Dameng. Built-in AI, MCP Server, CLI, desktop and Docker. | 轻量级跨平台数据库管理工具,支持 MySQL、PostgreSQL、SQLite、Redis、MongoDB、达梦等 100+ 数据库,提供桌面端、Docker、CLI、内置 AI 助手和 MCP。
TypeScript aicliclickhousedatabasedatabase-clientdatabase-managementdockerguimcpmongodbmysqlpostgresqlredisrustsql-serversqlitetaurivue
//! WebView2 process-failure recovery coordinator.
//!
//! wry's platform layer detects WebView2 child-process failures and surfaces
//! a structured [`WebView2ProcessFailedInfo`] signal (see
//! `vendor/wry/src/webview2/mod.rs`). This module owns the
//! *application-level* recovery policy: deciding whether a failure deserves a
//! renderer reload, an application restart, or nothing but diagnostics, and
//! executing the chosen action through Tauri APIs.
//!
//! The policy is deliberately a pure, deterministic state machine
//! ([`WebView2Recovery`]) so it can be unit-tested on any platform without a
//! Windows/WebView2 runtime; only [`install`] touches Tauri/wry.
//!
//! ## Kind policy (mirrors Microsoft's process-related-event guidance)
//!
//! - **Browser process exited**: fatal. Every control is closed and the
//! environment is gone; a plain reload is ineffective. DBX restarts the
//! application through the normal close flow so persisted state (open tabs,
//! window geometry, saved SQL) survives and running queries are cancelled
//! by `AppState::shutdown` (see `commands::connection`).
//! - **Main renderer exited**: confirmed renderer death — reload, bounded by
//! the rolling budget below.
//! - **Renderer unresponsive**: the event can be raised repeatedly and the
//! renderer may recover on its own, so DBX does **not** reload on the first
//! event. Only once the event fires `unresponsive_threshold` times inside
//! `unresponsive_window` does recovery trigger (still bounded by the same
//! rolling budget).
//! - **GPU exited**: WebView2 restarts the GPU process on its own; no reload.
//! - **Subframe renderer exited**: recovery is per-frame; reloading the whole
//! SPA would discard unsaved SQL / editor state for the whole window.
//! - **Utility / other**: log only, no destructive recovery.
//!
//! ## Reload budget (rolling, committed only on success)
//!
//! At most `max_reloads_in_window` automatic reloads are allowed inside a
//! rolling `reload_window`. The policy *decides* a reload is allowed without
//! consuming the budget; the budget is **committed only after the reload is
//! confirmed to have succeeded** ([`WebView2Recovery::record_reload_success`]).
//! A failed reload (main-thread scheduling failure, missing main window, or
//! `window.reload()` returning an error) never consumes a budget slot, so a
//! run of reloads that never actually happened cannot silently exhaust the
//! window and permanently disable recovery. A `reload_cooldown` additionally
//! deduplicates the burst of events a crash loop produces.
//!
//! ### Failure / escalation semantics
//!
//! - **Hard failures** — the main-thread callback cannot be scheduled or the
//! `main` webview window no longer exists. A reload cannot succeed in place:
//! these are exactly the "controls need to be recreated" states Microsoft
//! describes, so they escalate to a controlled [`tauri::AppHandle::request_restart`]
//! (which runs `AppState::shutdown` and restores persisted state).
//! - **`window.reload()` error** — the webview is present and the page can be
//! retried, so the first failures are logged and retried on the next
//! `ProcessFailed` event; only after
//! [`RELOAD_EXECUTION_ESCALATION_THRESHOLD`] consecutive reload errors does
//! DBX escalate to a controlled restart rather than looping forever.
//! - A successful reload resets the consecutive-failure counter.
//!
//! ## Session-task cancellation on reload
//!
//! Before reloading, DBX cancels the tasks whose frontend consumer lives in
//! the (now-dying) renderer session via
//! `AppState::cancel_webview_reload_session_tasks` — currently every task
//! registered in `RunningQueries` (SQL execution, counts/explains and exports).
//! Connection pools, tunnels, transaction sessions and daemons are
//! application-scoped and are intentionally preserved across a renderer reload.
//!
//! ## Production visibility
//!
//! Every decision and execution outcome is appended to `webview2-recovery.log`
//! next to `startup.log` via
//! [`crate::startup_recovery::record_runtime_recovery_event`]. That file is
//! not gated by the `debug_logging_enabled` setting (which turns the whole
//! `log` facade off in packaged builds), so fatal / exhausted / restart
//! decisions are always observable on affected Windows devices. The `log`
//! facade and `eprintln!` are used on top in debug builds.
use std::collections::VecDeque;
use std::sync::Mutex;
use std::time::{Duration, Instant};
use wry::{WebView2ProcessFailedInfo, WebView2ProcessFailedKind};
/// Consecutive `window.reload()` execution errors tolerated before DBX
/// escalates a wedged renderer to a controlled application restart.
const RELOAD_EXECUTION_ESCALATION_THRESHOLD: u32 = 3;
/// What the application should do in response to a WebView2 process failure.
#[derive(Clone, Copy, Debug, PartialEq, Eq)]
enum RecoveryAction {
/// Reload the main webview (bounded by the rolling budget).
Reload,
/// Restart the whole application through the normal close flow.
Restart,
/// No destructive action; the event is still logged.
LogOnly,
}
/// Why an event produced its action; used for diagnostics and tests.
#[derive(Clone, Copy, Debug, PartialEq, Eq)]
enum RecoveryDetail {
/// The browser process exited (fatal; every control is closed).
BrowserProcessExited,
/// The main renderer exited; confirmed death -> bounded reload.
RendererProcessExited,
/// Renderer unresponsive but not yet at the threshold; only armed.
RendererUnresponsiveArmed { count: u32, threshold: u32 },
/// Renderer unresponsive and the threshold was reached -> reload.
RendererUnresponsiveTriggered { count: u32, threshold: u32 },
/// The GPU process exited; WebView2 restarts it on its own.
GpuProcessExited,
/// A subframe renderer exited; a whole-SPA reload is not warranted.
FrameRendererProcessExited,
/// A utility or other process exited; no destructive recovery.
OtherProcessExited,
/// A reload was requested inside the cooldown window; skipped.
ReloadThrottled,
/// The rolling reload budget is exhausted; auto-recovery stops.
ReloadBudgetExhausted { reloads: u32, window_secs: u64 },
}
/// Result of the pure budget-eligibility check; never mutates recovery state.
#[derive(Clone, Copy, Debug, PartialEq, Eq)]
enum ReloadEligibility {
/// The reload is allowed; the budget is *not* consumed by this decision.
Allowed,
/// The last successful reload is still inside the cooldown window.
Cooldown,
/// The rolling window is already full of successful reloads.
BudgetExhausted,
}
/// At which stage a reload execution attempt failed.
#[derive(Clone, Copy, Debug, PartialEq, Eq)]
enum ReloadExecError {
/// `AppHandle::run_on_main_thread` returned an error (event loop down).
ScheduleMainThread,
/// The `main` webview window no longer exists (controls are gone).
MainWindowNotFound,
/// `window.reload()` returned an error.
ReloadError,
}
/// What the recovery layer does after a reload execution attempt. Pure policy
/// output; the Tauri glue in [`install`] logs it and performs any escalation.
#[derive(Clone, Copy, Debug, PartialEq, Eq)]
enum ReloadDisposition {
/// Reload succeeded and the rolling recovery budget was committed.
Committed,
/// Reload failed; budget untouched; retried on the next `ProcessFailed`.
RetryNext,
/// Reload cannot recover in place; escalate to a controlled app restart.
EscalateRestart,
}
/// Tunable constants of the recovery policy.
///
/// Every field has a documented policy purpose; they are kept as plain fields
/// (with [`Default`]) so tests can exercise boundary values directly. Durations
/// are measured against the monotonic [`Instant`] clock, so a system wall-clock
/// rollback cannot distort cooldown or budget windows.
#[derive(Clone, Copy, Debug, PartialEq, Eq)]
struct WebView2RecoveryPolicy {
/// Minimum gap between two *successful* automatic reloads. Absorbs the
/// burst of `ProcessFailed` events a crash loop produces so a single
/// failure is not answered with a reload storm.
pub reload_cooldown: Duration,
/// Rolling window for the reload budget: reloads older than this stop
/// counting, so a long healthy period resets the budget and unrelated
/// failures hours apart never permanently disable recovery.
pub reload_window: Duration,
/// Maximum successful automatic reloads allowed inside [`Self::reload_window`].
/// Once the window is full, auto-recovery stops and the app relies on the
/// user (the failure is logged; on Windows the recovery log records it).
pub max_reloads_in_window: u32,
/// Rolling window for renderer-unresponsive events.
pub unresponsive_window: Duration,
/// Renderer-unresponsive events inside [`Self::unresponsive_window`] that
/// trigger a reload. The first event never reloads (the renderer may
/// recover on its own); reaching this count means the renderer is
/// effectively wedged.
pub unresponsive_threshold: u32,
}
impl Default for WebView2RecoveryPolicy {
fn default() -> Self {
Self {
reload_cooldown: Duration::from_secs(30),
reload_window: Duration::from_secs(10 * 60),
max_reloads_in_window: 3,
unresponsive_window: Duration::from_secs(60),
unresponsive_threshold: 3,
}
}
}
/// Deterministic recovery state machine. Owns nothing but counters and
/// monotonic timestamps, so it can be driven with synthetic event streams in
/// tests using `Instant::now()` offsets (no sleeping, no wall clock).
struct WebView2Recovery {
policy: WebView2RecoveryPolicy,
/// [`Instant`] of each *successfully committed* reload inside the rolling
/// window.
reload_timestamps: VecDeque<Instant>,
/// [`Instant`] of the last successful automatic reload; `None` = never.
last_reload_at: Option<Instant>,
/// Start of the current unresponsive counting window.
unresponsive_window_start: Option<Instant>,
/// Renderer-unresponsive events observed inside the current window.
unresponsive_count: u32,
/// Whether a browser-process-exit restart was already requested for this
/// process. Guards against a second event double-restarting.
browser_fatal_fired: bool,
/// Consecutive `window.reload()` execution failures that were not yet
/// followed by a successful reload.
consecutive_reload_execution_failures: u32,
}
impl WebView2Recovery {
fn new(policy: WebView2RecoveryPolicy) -> Self {
Self {
policy,
reload_timestamps: VecDeque::new(),
last_reload_at: None,
unresponsive_window_start: None,
unresponsive_count: 0,
browser_fatal_fired: false,
consecutive_reload_execution_failures: 0,
}
}
/// Decides the recovery action for one `ProcessFailed` event.
///
/// `now` is a monotonic [`Instant`] (injected for tests). A returned
/// `Reload` action is a *decision* only — it does **not** consume the
/// budget; the budget is committed later via [`Self::record_reload_success`]
/// once the reload is confirmed to have succeeded.
fn handle_process_failed(
&mut self,
info: &WebView2ProcessFailedInfo,
now: Instant,
) -> (RecoveryAction, RecoveryDetail) {
match info.kind {
WebView2ProcessFailedKind::Browser => {
if self.browser_fatal_fired {
(RecoveryAction::LogOnly, RecoveryDetail::BrowserProcessExited)
} else {
self.browser_fatal_fired = true;
(RecoveryAction::Restart, RecoveryDetail::BrowserProcessExited)
}
}
WebView2ProcessFailedKind::Renderer => self.renderer_recovery(now, RecoveryDetail::RendererProcessExited),
WebView2ProcessFailedKind::RendererUnresponsive => {
let window = self.policy.unresponsive_window;
let window_expired = self
.unresponsive_window_start
.map(|start| now.checked_duration_since(start).is_some_and(|elapsed| elapsed >= window))
.unwrap_or(true);
if window_expired {
self.unresponsive_window_start = Some(now);
self.unresponsive_count = 0;
}
self.unresponsive_count += 1;
let count = self.unresponsive_count;
if count >= self.policy.unresponsive_threshold {
// Reset so post-reload evaluation starts fresh; the rolling
// budget still bounds a wedged-renderer crash loop.
self.unresponsive_count = 0;
self.renderer_recovery(
now,
RecoveryDetail::RendererUnresponsiveTriggered {
count,
threshold: self.policy.unresponsive_threshold,
},
)
} else {
(
RecoveryAction::LogOnly,
RecoveryDetail::RendererUnresponsiveArmed {
count,
threshold: self.policy.unresponsive_threshold,
},
)
}
}
// WebView2 restarts the GPU process on its own; do not reload the SPA.
WebView2ProcessFailedKind::Gpu => (RecoveryAction::LogOnly, RecoveryDetail::GpuProcessExited),
// A subframe renderer exit only needs the affected frame; reloading the
// whole SPA would discard unsaved SQL / editor state.
WebView2ProcessFailedKind::FrameRenderer => {
(RecoveryAction::LogOnly, RecoveryDetail::FrameRendererProcessExited)
}
_ => (RecoveryAction::LogOnly, RecoveryDetail::OtherProcessExited),
}
}
/// Shared renderer recovery decision: allow the reload only if the pure
/// eligibility check passes, without consuming the budget.
fn renderer_recovery(&mut self, now: Instant, detail: RecoveryDetail) -> (RecoveryAction, RecoveryDetail) {
match self.can_reload(now) {
ReloadEligibility::Allowed => (RecoveryAction::Reload, detail),
ReloadEligibility::Cooldown => (RecoveryAction::LogOnly, RecoveryDetail::ReloadThrottled),
ReloadEligibility::BudgetExhausted => (
RecoveryAction::LogOnly,
RecoveryDetail::ReloadBudgetExhausted {
reloads: self.reload_timestamps.len() as u32,
window_secs: self.policy.reload_window.as_secs(),
},
),
}
}
/// Pure eligibility check. Returns whether a reload may be *attempted* at
/// `now` without mutating any state, so the budget survive a later
/// execution failure.
fn can_reload(&self, now: Instant) -> ReloadEligibility {
if let Some(last) = self.last_reload_at {
if now.checked_duration_since(last).is_none_or(|elapsed| elapsed < self.policy.reload_cooldown) {
return ReloadEligibility::Cooldown;
}
}
let active = self
.reload_timestamps
.iter()
.filter(|ts| now.checked_duration_since(**ts).is_none_or(|elapsed| elapsed < self.policy.reload_window))
.count();
if active >= self.policy.max_reloads_in_window as usize {
return ReloadEligibility::BudgetExhausted;
}
ReloadEligibility::Allowed
}
/// Commits one successful reload into the rolling budget. Only call this
/// after the reload is confirmed to have succeeded (see
/// [`settle_reload_execution`]); a failure must not reach this method.
fn record_reload_success(&mut self, now: Instant) {
// Drop successful reloads that rolled out of the window: a long healthy
// period resets the budget and unrelated failures hours apart never
// permanently disable recovery.
while let Some(&ts) = self.reload_timestamps.front() {
if now.checked_duration_since(ts).is_some_and(|elapsed| elapsed >= self.policy.reload_window) {
self.reload_timestamps.pop_front();
} else {
break;
}
}
self.reload_timestamps.push_back(now);
self.last_reload_at = Some(now);
self.consecutive_reload_execution_failures = 0;
}
/// Records one reload execution failure and reports whether a controlled
/// restart should be escalated to. Successful reloads reset the counter.
fn record_reload_execution_failure(&mut self) -> bool {
self.consecutive_reload_execution_failures += 1;
self.consecutive_reload_execution_failures >= RELOAD_EXECUTION_ESCALATION_THRESHOLD
}
/// `(active_successful_reloads, max_reloads)` for the production-visible
/// execution diagnostics (recorded by the Windows `record_execution_outcome`).
#[cfg(target_os = "windows")]
fn reload_budget_state(&self, now: Instant) -> (u32, u32) {
let active = self
.reload_timestamps
.iter()
.filter(|ts| now.checked_duration_since(**ts).is_none_or(|elapsed| elapsed < self.policy.reload_window))
.count();
(active as u32, self.policy.max_reloads_in_window)
}
}
/// Settles a reload execution attempt against the recovery policy.
///
/// `Ok(())` means the reload was confirmed to have succeeded, so the rolling
/// budget is committed exactly once. A failure is never counted as success:
/// hard failures (event loop down / main window gone) escalate immediately to
/// a controlled restart, while a `window.reload()` error is retried on the
/// next event and escalates only after
/// [`RELOAD_EXECUTION_ESCALATION_THRESHOLD`] consecutive failures.
///
/// This is pure (no Tauri types) so the state transitions are unit-testable;
/// production's `install` is a thin shell around it.
fn settle_reload_execution(
recovery: &Mutex<WebView2Recovery>,
now: Instant,
error: Option<ReloadExecError>,
) -> ReloadDisposition {
let mut recovery = recovery.lock().unwrap_or_else(|poisoned| poisoned.into_inner());
match error {
None => {
recovery.record_reload_success(now);
ReloadDisposition::Committed
}
Some(ReloadExecError::ScheduleMainThread | ReloadExecError::MainWindowNotFound) => {
// The event loop is down or the main control is gone: a reload
// cannot succeed in place. Recreate the webview by restarting the
// application through the normal close flow (persisted state is
// preserved and running tasks are cancelled by `AppState::shutdown`).
ReloadDisposition::EscalateRestart
}
Some(ReloadExecError::ReloadError) => {
if recovery.record_reload_execution_failure() {
ReloadDisposition::EscalateRestart
} else {
ReloadDisposition::RetryNext
}
}
}
}
fn recovery_log_message(action: RecoveryAction, detail: RecoveryDetail, info: &WebView2ProcessFailedInfo) -> String {
format!(
"recovery decision={action:?} detail={detail:?} kind={:?} reason={} exit_code={} process={} frames={}",
info.kind,
info.reason.as_deref().unwrap_or("-"),
info.exit_code.map(|code| code.to_string()).unwrap_or_else(|| "-".to_string()),
info.process_description.as_deref().unwrap_or("-"),
info.affected_frames,
)
}
/// Session-task cancellation diagnostic line (pure; recorded by Windows install).
fn cancel_session_log_message(kind: WebView2ProcessFailedKind, cancelled: Option<usize>) -> String {
match cancelled {
Some(count) => format!(
"recovery decision=Reload kind={kind:?} execution=cancel-session-tasks result=ok cancelled={count}"
),
None => format!(
"recovery decision=Reload kind={kind:?} execution=cancel-session-tasks result=failed error=app_state_missing"
),
}
}
/// Reload execution outcome diagnostic line (pure; recorded by Windows install).
fn reload_execution_log_message(
info: &WebView2ProcessFailedInfo,
error: Option<ReloadExecError>,
disposition: ReloadDisposition,
budget_active: u32,
budget_max: u32,
) -> String {
let stage = match error {
None => "window-reload",
Some(ReloadExecError::ScheduleMainThread) => "schedule-main-thread",
Some(ReloadExecError::MainWindowNotFound) => "find-main-window",
Some(ReloadExecError::ReloadError) => "window-reload",
};
let result = if error.is_none() { "success" } else { "failed" };
let budget_commit = matches!(disposition, ReloadDisposition::Committed);
let escalate = matches!(disposition, ReloadDisposition::EscalateRestart);
let failure = error.map(|error| format!("{error:?}")).unwrap_or_else(|| "-".to_string());
format!(
"recovery decision=Reload kind={:?} execution={stage} result={result} error={} budget_commit={budget_commit} budget={budget_active}/{budget_max} escalate={escalate}",
info.kind,
failure,
)
}
/// Installs the WebView2 process-failure handler for this application.
///
/// Must be called once during Tauri setup on Windows. The callback receives
/// every `ProcessFailed` event from the (single) main webview, applies the
/// recovery policy, records a production-visible log line, and executes the
/// action:
///
/// - `Restart` → [`tauri::AppHandle::request_restart`], which runs the normal
/// close flow (`RunEvent::ExitRequested` → [`AppState::shutdown`] cancels
/// running queries and closes pools/tunnels) before Tauri respawns the
/// process. Persisted state (open tabs incl. unsaved SQL, window geometry,
/// saved SQL library) survives the restart.
/// - `Reload` → cancels the webview-session tasks (`AppState::cancel_webview_reload_session_tasks`),
/// then reloads the `main` webview on the main thread. The rolling budget is
/// committed only after the reload is confirmed to have succeeded.
/// - `LogOnly` → nothing but the log line.
#[cfg(target_os = "windows")]
pub fn install(app: &tauri::AppHandle) {
use std::sync::{Arc, LazyLock};
use wry::WebView2ProcessFailedCallback;
static RECOVERY: LazyLock<Mutex<WebView2Recovery>> =
LazyLock::new(|| Mutex::new(WebView2Recovery::new(WebView2RecoveryPolicy::default())));
let app = app.clone();
let callback: WebView2ProcessFailedCallback = Arc::new(move |info: &WebView2ProcessFailedInfo| {
let now = Instant::now();
let (action, detail) = {
let mut recovery = RECOVERY.lock().unwrap_or_else(|poisoned| poisoned.into_inner());
recovery.handle_process_failed(info, now)
};
let message = recovery_log_message(action, detail, info);
crate::startup_recovery::record_runtime_recovery_event(&message);
log::error!("{message}");
eprintln!("{message}");
match action {
RecoveryAction::Restart => {
// Full close flow: `ExitRequested` runs `AppState::shutdown` (cancels
// running queries, closes pools/tunnels/daemons) before Tauri
// respawns the process, so the restart is not a raw kill.
app.request_restart();
}
RecoveryAction::Reload => {
let recovery: &'static Mutex<WebView2Recovery> = &RECOVERY;
perform_reload_recovery(&app, recovery, now, info);
}
RecoveryAction::LogOnly => {}
}
});
wry::set_webview2_process_failed_callback(Some(callback));
}
/// Executes a renderer recovery reload: cancels the webview-session tasks,
/// schedules the reload on the main thread, and settles the result against the
/// budget (commit-on-success / retry / escalate).
#[cfg(target_os = "windows")]
fn perform_reload_recovery(
app: &tauri::AppHandle,
recovery: &'static Mutex<WebView2Recovery>,
now: Instant,
info: &WebView2ProcessFailedInfo,
) {
use crate::commands::connection::AppState;
use std::sync::Arc;
use tauri::Manager;
// 1. Cancel the webview-session tasks before reloading, so no SQL keeps
// running for a consumer that is about to disappear. Cancellation
// outcome is always logged; a missing AppState is recorded rather than
// silently reloading, but does not abort the reload (reloading is still
// the least-destructive recovery available).
let session_cancel =
app.try_state::<Arc<AppState>>().map(|state| state.inner().cancel_webview_reload_session_tasks());
let cancel_line = cancel_session_log_message(info.kind, session_cancel);
crate::startup_recovery::record_runtime_recovery_event(&cancel_line);
log::info!("{cancel_line}");
// 2. Schedule the actual reload on the main thread. Only the main-thread
// closure can confirm the reload outcome, so budget commit happens there.
let reload_app = app.clone();
let info_for_thread = info.clone();
let schedule = app.run_on_main_thread(move || {
let execute_now = Instant::now();
let Some(window) = reload_app.get_webview_window("main") else {
let error = Some(ReloadExecError::MainWindowNotFound);
let disposition = settle_reload_execution(recovery, execute_now, error);
record_execution_outcome(&info_for_thread, recovery, execute_now, error, disposition);
escalate_if_needed(&reload_app, disposition);
return;
};
let error = match window.reload() {
Ok(()) => None,
Err(reload_error) => {
log::warn!("webview2 recovery: window.reload() failed: {reload_error}");
Some(ReloadExecError::ReloadError)
}
};
let disposition = settle_reload_execution(recovery, execute_now, error);
record_execution_outcome(&info_for_thread, recovery, execute_now, error, disposition);
escalate_if_needed(&reload_app, disposition);
});
if let Err(schedule_error) = schedule {
log::error!("webview2 recovery: failed to schedule main-thread reload: {schedule_error}");
let error = Some(ReloadExecError::ScheduleMainThread);
let disposition = settle_reload_execution(recovery, now, error);
record_execution_outcome(info, recovery, now, error, disposition);
escalate_if_needed(app, disposition);
}
}
/// Logs a reload execution outcome to the production-visible recovery log and
/// the `log` facade.
#[cfg(target_os = "windows")]
fn record_execution_outcome(
info: &WebView2ProcessFailedInfo,
recovery: &Mutex<WebView2Recovery>,
now: Instant,
error: Option<ReloadExecError>,
disposition: ReloadDisposition,
) {
let (budget_active, budget_max) = {
let recovery = recovery.lock().unwrap_or_else(|poisoned| poisoned.into_inner());
recovery.reload_budget_state(now)
};
let line = reload_execution_log_message(info, error, disposition, budget_active, budget_max);
crate::startup_recovery::record_runtime_recovery_event(&line);
log::info!("{line}");
}
/// Performs a controlled application restart when a reload execution has been
/// escalated. Records it first so the escalation is observable on the device.
#[cfg(target_os = "windows")]
fn escalate_if_needed(app: &tauri::AppHandle, disposition: ReloadDisposition) {
if matches!(disposition, ReloadDisposition::EscalateRestart) {
let line = "recovery decision=Reload escalation=restart reason=renderer_reload_cannot_recover_in_place";
crate::startup_recovery::record_runtime_recovery_event(line);
log::error!("{line}");
app.request_restart();
}
}
#[cfg(test)]
mod tests {
use super::*;
fn policy_with(
reload_cooldown: Duration,
reload_window: Duration,
max_reloads: u32,
unresponsive_window: Duration,
unresponsive_threshold: u32,
) -> WebView2RecoveryPolicy {
WebView2RecoveryPolicy {
reload_cooldown,
reload_window,
max_reloads_in_window: max_reloads,
unresponsive_window,
unresponsive_threshold,
}
}
fn info(kind: WebView2ProcessFailedKind) -> WebView2ProcessFailedInfo {
WebView2ProcessFailedInfo { kind, reason: None, exit_code: None, process_description: None, affected_frames: 0 }
}
fn base() -> Instant {
Instant::now()
}
fn at(base: Instant, secs: u64) -> Instant {
base + Duration::from_secs(secs)
}
fn mutex(recovery: WebView2Recovery) -> Mutex<WebView2Recovery> {
Mutex::new(recovery)
}
// ------------------------------------------------------------------
// Kind policy
// ------------------------------------------------------------------
#[test]
fn browser_process_exit_requests_a_restart() {
let mut recovery = WebView2Recovery::new(WebView2RecoveryPolicy::default());
assert_eq!(
recovery.handle_process_failed(&info(WebView2ProcessFailedKind::Browser), base()),
(RecoveryAction::Restart, RecoveryDetail::BrowserProcessExited)
);
}
#[test]
fn browser_process_exit_never_restarts_twice() {
let mut recovery = WebView2Recovery::new(WebView2RecoveryPolicy::default());
let t0 = base();
assert_eq!(
recovery.handle_process_failed(&info(WebView2ProcessFailedKind::Browser), t0).0,
RecoveryAction::Restart
);
assert_eq!(
recovery.handle_process_failed(&info(WebView2ProcessFailedKind::Browser), at(t0, 1)).0,
RecoveryAction::LogOnly
);
}
#[test]
fn renderer_exit_decides_reload_without_consuming_budget() {
let mut recovery = WebView2Recovery::new(WebView2RecoveryPolicy::default());
let t0 = base();
// The decision alone must not consume the budget: after deciding Reload,
// a second immediate decision is still allowed (only a committed success
// enters the cooldown/budget).
assert_eq!(
recovery.handle_process_failed(&info(WebView2ProcessFailedKind::Renderer), t0),
(RecoveryAction::Reload, RecoveryDetail::RendererProcessExited)
);
assert!(recovery.reload_timestamps.is_empty());
assert_eq!(recovery.can_reload(t0), ReloadEligibility::Allowed);
}
#[test]
fn first_unresponsive_event_never_reloads() {
let mut recovery = WebView2Recovery::new(WebView2RecoveryPolicy::default());
let (action, detail) =
recovery.handle_process_failed(&info(WebView2ProcessFailedKind::RendererUnresponsive), base());
assert_eq!(action, RecoveryAction::LogOnly);
assert_eq!(detail, RecoveryDetail::RendererUnresponsiveArmed { count: 1, threshold: 3 });
}
#[test]
fn unresponsive_reloads_only_after_the_threshold() {
let mut recovery = WebView2Recovery::new(WebView2RecoveryPolicy::default());
let t0 = base();
for (offset, expected_count) in [(0u64, 1u32), (20, 2)] {
let (action, detail) =
recovery.handle_process_failed(&info(WebView2ProcessFailedKind::RendererUnresponsive), at(t0, offset));
assert_eq!(action, RecoveryAction::LogOnly);
assert_eq!(detail, RecoveryDetail::RendererUnresponsiveArmed { count: expected_count, threshold: 3 });
}
let (action, detail) =
recovery.handle_process_failed(&info(WebView2ProcessFailedKind::RendererUnresponsive), at(t0, 40));
assert_eq!(action, RecoveryAction::Reload);
assert_eq!(detail, RecoveryDetail::RendererUnresponsiveTriggered { count: 3, threshold: 3 });
let (action, detail) =
recovery.handle_process_failed(&info(WebView2ProcessFailedKind::RendererUnresponsive), at(t0, 50));
assert_eq!(action, RecoveryAction::LogOnly);
assert_eq!(detail, RecoveryDetail::RendererUnresponsiveArmed { count: 1, threshold: 3 });
}
#[test]
fn unresponsive_counter_expires_with_the_window() {
let mut recovery = WebView2Recovery::new(policy_with(
Duration::from_secs(30),
Duration::from_secs(600),
3,
Duration::from_secs(60),
3,
));
let t0 = base();
recovery.handle_process_failed(&info(WebView2ProcessFailedKind::RendererUnresponsive), t0);
recovery.handle_process_failed(&info(WebView2ProcessFailedKind::RendererUnresponsive), at(t0, 20));
let (action, detail) =
recovery.handle_process_failed(&info(WebView2ProcessFailedKind::RendererUnresponsive), at(t0, 120));
assert_eq!(action, RecoveryAction::LogOnly);
assert_eq!(detail, RecoveryDetail::RendererUnresponsiveArmed { count: 1, threshold: 3 });
}
#[test]
fn gpu_exit_never_reloads() {
let mut recovery = WebView2Recovery::new(WebView2RecoveryPolicy::default());
let (action, detail) = recovery.handle_process_failed(&info(WebView2ProcessFailedKind::Gpu), base());
assert_eq!(action, RecoveryAction::LogOnly);
assert_eq!(detail, RecoveryDetail::GpuProcessExited);
}
#[test]
fn frame_renderer_exit_never_reloads_the_whole_webview() {
let mut recovery = WebView2Recovery::new(WebView2RecoveryPolicy::default());
let (action, detail) = recovery.handle_process_failed(&info(WebView2ProcessFailedKind::FrameRenderer), base());
assert_eq!(action, RecoveryAction::LogOnly);
assert_eq!(detail, RecoveryDetail::FrameRendererProcessExited);
}
#[test]
fn utility_and_unknown_exits_only_log() {
let mut recovery = WebView2Recovery::new(WebView2RecoveryPolicy::default());
for kind in [
WebView2ProcessFailedKind::Utility,
WebView2ProcessFailedKind::SandboxHelper,
WebView2ProcessFailedKind::PpapiPlugin,
WebView2ProcessFailedKind::PpapiBroker,
WebView2ProcessFailedKind::Unknown,
] {
let (action, detail) = recovery.handle_process_failed(&info(kind), base());
assert_eq!(action, RecoveryAction::LogOnly, "{kind:?}");
assert_eq!(detail, RecoveryDetail::OtherProcessExited, "{kind:?}");
}
}
// ------------------------------------------------------------------
// Budget commit-on-success semantics
// ------------------------------------------------------------------
#[test]
fn success_commits_budget_and_exhausts_after_three() {
let mut recovery = WebView2Recovery::new(policy_with(
Duration::from_secs(30),
Duration::from_secs(600),
3,
Duration::from_secs(60),
3,
));
let t0 = base();
for offset in [0u64, 60, 120] {
let now = at(t0, offset);
assert_eq!(recovery.can_reload(now), ReloadEligibility::Allowed, "offset {offset}");
recovery.record_reload_success(now);
}
assert_eq!(recovery.can_reload(at(t0, 180)), ReloadEligibility::BudgetExhausted);
}
#[test]
fn failed_reload_does_not_consume_budget() {
let recovery = mutex(WebView2Recovery::new(policy_with(
Duration::from_secs(30),
Duration::from_secs(600),
3,
Duration::from_secs(60),
3,
)));
let t0 = base();
// A reload error is retried, never committed.
assert_eq!(
settle_reload_execution(&recovery, t0, Some(ReloadExecError::ReloadError)),
ReloadDisposition::RetryNext
);
let inner = recovery.lock().unwrap();
assert!(inner.reload_timestamps.is_empty());
assert_eq!(inner.can_reload(t0), ReloadEligibility::Allowed);
}
#[test]
fn two_success_and_one_failure_leaves_one_slot() {
let mut recovery = WebView2Recovery::new(policy_with(
Duration::from_secs(30),
Duration::from_secs(600),
3,
Duration::from_secs(60),
3,
));
let t0 = base();
for offset in [0u64, 60] {
recovery.record_reload_success(at(t0, offset));
}
// A failed reload must not fill the third slot ...
let recovery = mutex(recovery);
assert_eq!(
settle_reload_execution(&recovery, at(t0, 120), Some(ReloadExecError::ReloadError)),
ReloadDisposition::RetryNext
);
// ... so it is still eligible here (the failure consumed no budget).
// Note: each `lock().method()` uses a temporary guard dropped at the
// end of its statement, so no guard is held across a re-lock (which
// would deadlock).
assert_eq!(recovery.lock().unwrap().can_reload(at(t0, 120)), ReloadEligibility::Allowed);
// ... and a subsequent success still fits in the third slot.
recovery.lock().unwrap().record_reload_success(at(t0, 180));
assert_eq!(recovery.lock().unwrap().can_reload(at(t0, 240)), ReloadEligibility::BudgetExhausted);
}
#[test]
fn cooldown_is_based_on_successful_reloads_only() {
let recovery = mutex(WebView2Recovery::new(WebView2RecoveryPolicy::default()));
let t0 = base();
// A failed reload neither commits nor starts a cooldown.
assert_eq!(
settle_reload_execution(&recovery, t0, Some(ReloadExecError::ReloadError)),
ReloadDisposition::RetryNext
);
let mut inner = recovery.lock().unwrap();
assert_eq!(inner.can_reload(t0), ReloadEligibility::Allowed);
assert_eq!(inner.last_reload_at, None);
// A successful reload starts the 30s cooldown.
inner.record_reload_success(t0);
assert_eq!(inner.can_reload(at(t0, 10)), ReloadEligibility::Cooldown);
assert_eq!(inner.can_reload(at(t0, 31)), ReloadEligibility::Allowed);
}
// ------------------------------------------------------------------
// Monotonic time / rolling window
// ------------------------------------------------------------------
#[test]
fn budget_expires_after_the_rolling_window() {
let mut recovery = WebView2Recovery::new(policy_with(
Duration::from_secs(30),
Duration::from_secs(600),
3,
Duration::from_secs(60),
3,
));
let t0 = base();
for offset in [0u64, 60, 120] {
recovery.record_reload_success(at(t0, offset));
}
// Just inside the window the budget is still exhausted...
assert_eq!(recovery.can_reload(at(t0, 179)), ReloadEligibility::BudgetExhausted);
// ... and once the oldest reload falls out of the rolling window one
// slot frees up again (a long healthy period resets the budget).
assert_eq!(recovery.can_reload(at(t0, 600 + 1)), ReloadEligibility::Allowed);
}
#[test]
fn unrelated_failures_hours_apart_do_not_deplete_the_lifetime_budget() {
let recovery = mutex(WebView2Recovery::new(policy_with(
Duration::from_secs(30),
Duration::from_secs(600),
3,
Duration::from_secs(60),
3,
)));
let t0 = base();
// One successful reload every hour: each is checked before the commit
// (well past the 30s cooldown) and always eligible, so the budget is
// never permanently consumed across the WebView lifetime.
for hour in 0..10 {
let now = at(t0, hour * 3600);
assert_eq!(recovery.lock().unwrap().can_reload(now), ReloadEligibility::Allowed, "hour {hour}");
assert_eq!(settle_reload_execution(&recovery, now, None), ReloadDisposition::Committed);
}
}
#[test]
fn unresponsive_threshold_window_now_uses_instant() {
let recovery = mutex(WebView2Recovery::new(policy_with(
Duration::from_secs(30),
Duration::from_secs(600),
3,
Duration::from_secs(60),
3,
)));
let t0 = base();
// Two armed events inside the 60s window...
recovery.lock().unwrap().handle_process_failed(&info(WebView2ProcessFailedKind::RendererUnresponsive), t0);
recovery
.lock()
.unwrap()
.handle_process_failed(&info(WebView2ProcessFailedKind::RendererUnresponsive), at(t0, 20));
// ... the window has not expired, so a third event triggers the reload.
let (action, _) = recovery
.lock()
.unwrap()
.handle_process_failed(&info(WebView2ProcessFailedKind::RendererUnresponsive), at(t0, 40));
assert_eq!(action, RecoveryAction::Reload);
}
#[test]
fn system_clock_does_not_drive_policy_state() {
// Policy timestamps are `Instant` (monotonic). A wall-clock rollback
// cannot shrink a `now.checked_duration_since(last)` elapsed value: the
// policy reader only ever computes `now - earlier` on an `Instant`
// timeline, which is strictly nondecreasing. This is a sanity guard that
// the policy stores `Instant` values (never Unix wall-clock millis).
let recovery = WebView2Recovery::new(WebView2RecoveryPolicy::default());
let _: Option<Instant> = recovery.last_reload_at;
let t0 = base();
assert_eq!(t0.duration_since(t0), Duration::ZERO);
assert!(at(t0, 5).checked_duration_since(t0).is_some());
}
// ------------------------------------------------------------------
// Reload execution settling / escalation
// ------------------------------------------------------------------
#[test]
fn reload_success_commits_exactly_once() {
let recovery = mutex(WebView2Recovery::new(WebView2RecoveryPolicy::default()));
let t0 = base();
assert_eq!(settle_reload_execution(&recovery, t0, None), ReloadDisposition::Committed);
assert_eq!(settle_reload_execution(&recovery, t0, None), ReloadDisposition::Committed);
let inner = recovery.lock().unwrap();
assert_eq!(inner.reload_timestamps.len(), 2);
assert_eq!(inner.consecutive_reload_execution_failures, 0);
}
#[test]
fn reload_error_is_retried_then_escalates_after_threshold() {
let recovery = mutex(WebView2Recovery::new(WebView2RecoveryPolicy::default()));
let t0 = base();
// The first two `window.reload()` errors (failure counter 1, 2) are
// retried on the next `ProcessFailed` event; the third consecutive
// failure reaches `RELOAD_EXECUTION_ESCALATION_THRESHOLD` (3) and
// escalates to a controlled restart.
for offset in [0u64, 10] {
assert_eq!(
settle_reload_execution(&recovery, at(t0, offset), Some(ReloadExecError::ReloadError)),
ReloadDisposition::RetryNext,
"offset {offset} must retry before the threshold"
);
}
assert_eq!(
settle_reload_execution(&recovery, at(t0, 20), Some(ReloadExecError::ReloadError)),
ReloadDisposition::EscalateRestart
);
// The counter keeps climbing, so a further error still escalates.
assert_eq!(
settle_reload_execution(&recovery, at(t0, 30), Some(ReloadExecError::ReloadError)),
ReloadDisposition::EscalateRestart
);
}
#[test]
fn successful_reload_resets_the_failure_counter() {
let recovery = mutex(WebView2Recovery::new(WebView2RecoveryPolicy::default()));
let t0 = base();
assert_eq!(
settle_reload_execution(&recovery, t0, Some(ReloadExecError::ReloadError)),
ReloadDisposition::RetryNext
);
// A success resets the counter, so the escalation threshold restarts.
assert_eq!(settle_reload_execution(&recovery, at(t0, 40), None), ReloadDisposition::Committed);
assert_eq!(
settle_reload_execution(&recovery, at(t0, 50), Some(ReloadExecError::ReloadError)),
ReloadDisposition::RetryNext
);
let inner = recovery.lock().unwrap();
assert_eq!(inner.consecutive_reload_execution_failures, 1);
}
#[test]
fn window_not_found_escalates_to_restart_without_committing() {
let recovery = mutex(WebView2Recovery::new(WebView2RecoveryPolicy::default()));
let t0 = base();
assert_eq!(
settle_reload_execution(&recovery, t0, Some(ReloadExecError::MainWindowNotFound)),
ReloadDisposition::EscalateRestart
);
let inner = recovery.lock().unwrap();
assert!(inner.reload_timestamps.is_empty());
}
#[test]
fn schedule_main_thread_failure_escalates_to_restart() {
let recovery = mutex(WebView2Recovery::new(WebView2RecoveryPolicy::default()));
let t0 = base();
assert_eq!(
settle_reload_execution(&recovery, t0, Some(ReloadExecError::ScheduleMainThread)),
ReloadDisposition::EscalateRestart
);
}
#[test]
fn repeated_hard_failures_never_commit_budget() {
let recovery = mutex(WebView2Recovery::new(WebView2RecoveryPolicy::default()));
let t0 = base();
for offset in [0u64, 5, 10] {
assert_eq!(
settle_reload_execution(&recovery, at(t0, offset), Some(ReloadExecError::MainWindowNotFound)),
ReloadDisposition::EscalateRestart
);
}
let inner = recovery.lock().unwrap();
assert!(inner.reload_timestamps.is_empty());
}
// ------------------------------------------------------------------
// Logging
// ------------------------------------------------------------------
#[test]
fn recovery_log_message_includes_kind_and_action() {
let mut made = info(WebView2ProcessFailedKind::Renderer);
made.reason = Some("crashed".to_string());
made.exit_code = Some(-1);
made.process_description = Some("renderer".to_string());
made.affected_frames = 2;
let message = recovery_log_message(RecoveryAction::Reload, RecoveryDetail::RendererProcessExited, &made);
assert!(message.contains("decision=Reload"));
assert!(message.contains("detail=RendererProcessExited"));
assert!(message.contains("kind=Renderer"));
assert!(message.contains("reason=crashed"));
assert!(message.contains("exit_code=-1"));
assert!(message.contains("frames=2"));
}
#[test]
fn cancel_session_log_reflects_missing_state() {
assert!(cancel_session_log_message(WebView2ProcessFailedKind::Renderer, None).contains("result=failed"));
let ok = cancel_session_log_message(WebView2ProcessFailedKind::Renderer, Some(3));
assert!(ok.contains("result=ok"));
assert!(ok.contains("cancelled=3"));
}
#[test]
fn reload_execution_log_includes_stage_budget_and_escalation() {
let made = info(WebView2ProcessFailedKind::Renderer);
let committed = reload_execution_log_message(&made, None, ReloadDisposition::Committed, 1, 3);
assert!(committed.contains("execution=window-reload"));
assert!(committed.contains("result=success"));
assert!(committed.contains("budget_commit=true"));
assert!(committed.contains("budget=1/3"));
assert!(committed.contains("escalate=false"));
let failed = reload_execution_log_message(
&made,
Some(ReloadExecError::MainWindowNotFound),
ReloadDisposition::EscalateRestart,
0,
3,
);
assert!(failed.contains("execution=find-main-window"));
assert!(failed.contains("result=failed"));
assert!(failed.contains("budget_commit=false"));
assert!(failed.contains("escalate=true"));
assert!(failed.contains("error=MainWindowNotFound"));
}
}