Skip to content

A fast 2D and 3D game engine — TypeScript SDK, C++/WebAssembly core, visual editor. Ship one project to web, desktop, WeChat MiniGames, playable ads, and native Android / iOS.

ci(editor): measure the disk on the two axes that can still explain ENOSPC

yhh5277877a7c12
diff --git a/.github/workflows/build.yml b/.github/workflows/build.yml
index 0cad57ee5..419e17912 100644
--- a/.github/workflows/build.yml
+++ b/.github/workflows/build.yml
@@ -372,15 +372,28 @@ jobs:
run: |
sudo rm -rf /usr/share/dotnet /usr/share/swift /opt/ghc /usr/local/.ghcup \
/opt/hostedtoolcache/CodeQL || true
- # Four more trees and the image cache, none of them on a path this job
- # takes. It has run out of disk three times in seven runs and passed the
- # other four, so it is not leaking without bound — it sits on the line,
- # and this is the margin. Android stays: the packaging steps below are
- # the one thing here that could want it.
+ # Four more trees and the image cache, kept for being free rather than for
+ # working: the run after this measured the whole job at ~1GB, ending with
+ # 104G of 145G still free, so block exhaustion on / is NOT what killed the
+ # three that died of ENOSPC. Android stays for the packaging steps below.
sudo rm -rf /usr/share/miniconda /usr/local/share/powershell /opt/az || true
sudo rm -rf /usr/local/share/boost || true
sudo docker image prune -af || true
- df -h /
+ df -h / && df -i /
+
+ # ENOSPC killed this job three times in seven runs, and the reports around it
+ # measure ~1GB used with 104G free — so what fills the disk does it INSIDE a
+ # step and gives it back, which no reading at a step boundary can see. Five
+ # seconds apart, kept to one file, and the worst line is printed at the end.
+ - name: Watch the disk, since a spike between readings is invisible
+ run: |
+ nohup bash -c 'while :; do
+ a=$(df -B1G / | awk "NR==2{print \$4}")
+ i=$(df -i / | awk "NR==2{print \$5}")
+ echo "$(date -u +%H:%M:%S) availG=$a inodes=$i" >> /tmp/disk-watch.log
+ sleep 5
+ done' >/dev/null 2>&1 &
+ echo started
# The editor is private; without its key this job fails here rather than
# reporting green about a checkout it never had.
@@ -405,7 +418,7 @@ jobs:
# run too, so the next run answers it either way.
- name: How much is left, after the authoring checks
if: always()
- run: df -h /
+ run: df -h / && df -i /
# The pixel gates whose subject is an editor door — the reference grid, hit
# testing, the editor eye, preview renders. They cannot run on the engine's
@@ -426,9 +439,14 @@ jobs:
- name: Say what is left, and what took it
if: always()
run: |
- df -h /
+ df -h / && df -i /
du -x -h -d 2 /home/runner 2>/dev/null | sort -rh | head -25 || true
du -x -h -d 1 /tmp 2>/dev/null | sort -rh | head -10 || true
+ # The low-water mark the sampler saw, which is the only reading that can
+ # catch what a step gives back before it ends.
+ awk '{gsub(/availG=/,"",$2); if (min=="" || $2+0 < min+0) {min=$2; at=$0}} END{print "lowest free seen: " (at=="" ? "no samples" : at)}' /tmp/disk-watch.log 2>/dev/null || true
+ tail -3 /tmp/disk-watch.log 2>/dev/null || true
+ echo "samples: $(wc -l < /tmp/disk-watch.log 2>/dev/null || echo 0)"
# Both are DEEPER oracles the Android tests skip when absent, so fetching them
# must not be able to cost more than they are worth. Unbounded, this step hung