Open source Loom alternative. Beautiful, shareable screen recordings.
Rust appcapcossloommacnextjsnextjs15open-sourceossreactrecordrustscreen-capturescreen-recorderscreenshotsolidjstauritauri-apptypescriptvite
ci(verify): retry a lost GPU device once under proven starvation
Richie McIlroy37d01581be1a
diff --git a/crates/verify/README.md b/crates/verify/README.mdindex 9551eec66..9e9d0a005 100644--- a/crates/verify/README.md+++ b/crates/verify/README.md@@ -132,8 +132,8 @@ blessed failure is a bug that the suite keeps checking.A wake gap of 100 ms or more proves the runner stopped scheduling theprocess. Only wall-clock failures are excused by it: a real-time playbackcase that delivered too few frames becomes `inconclusive`, and a test- killed at its timeout or by wgpu's 6.3 s queue drain is retryable when its- sentinel log shows the gap. `scripts/verify/retry.py` reruns those tests+ killed at its timeout, by wgpu's 6.3 s queue drain or by a lost GPU device is+ retryable when its sentinel log shows the gap. `scripts/verify/retry.py` reruns those testsonce with `CAP_VERIFY_RETRY=1`; a case that is inconclusive again fails as`environment`, and the job passes only if every failure was retryable andevery retry passed. Content, frame identity, timestamps and sync neverdiff --git a/scripts/verify/retry.py b/scripts/verify/retry.pyindex a8018d4a0..918df6c77 100644--- a/scripts/verify/retry.py+++ b/scripts/verify/retry.py@@ -10,8 +10,9 @@ is one that wall-clock timing can cause:judged in cap_verify::report::settle), or- nextest killed it at its timeout and its sentinel log shows such a gap, or- it panicked in wgpu-core's queue drop ("We timed out while waiting on the- last successful submission", a 6.3 s limit) and its sentinel log shows- such a gap.+ last successful submission", a 6.3 s limit) or on a lost GPU device+ ("Device with 'cap-rendering-device' label is invalid") and its sentinel+ log shows such a gap.A test with any FAIL or XPASS case, or that crashed or timed out withoutstarvation evidence, is not retried: it fails the run as before. The retry@@ -31,6 +32,9 @@ import xml.etree.ElementTree as ETSTARVATION_GAP_MS = 100.0QUEUE_DRAIN_PANIC = "We timed out while waiting on the last successful submission"+# The GPU device was lost (WARP gives up on work that takes too long) and the+# next wgpu call on it panics instead of reporting an error.+DEVICE_LOST_PANIC = "Device with 'cap-rendering-device' label is invalid"def records(report_dir):@@ -95,6 +99,10 @@ def main():blocking += 1elif timed_out and longest >= STARVATION_GAP_MS:print(f"{binary}\t{name}\ttimed out; sentinel longest wake gap {longest:.0f} ms")+ elif DEVICE_LOST_PANIC in text and longest >= STARVATION_GAP_MS:+ # Product hazard too (a lost device should be an error, not a+ # panic), excused only under the sentinel's evidence.+ print(f"{binary}\t{name}\twgpu device lost; sentinel longest wake gap {longest:.0f} ms")elif QUEUE_DRAIN_PANIC in text and longest >= STARVATION_GAP_MS:# wgpu-core gives a queue 6.3 s to finish its GPU work when the# last handle drops, then panics. A product lifecycle hazard (the