SeaweedFS is a distributed storage system for object storage (S3), file systems, Iceberg table buckets, and Lance vector buckets, designed to handle billions of files with O(1) disk access and effortless horizontal scaling.
FUSE volume server failover tests
Integration tests for what happens to FUSE mounts when a volume server goes away, comes back, or restarts underneath in-flight IO. They automate the manual matrix reported in discussion #10206: one mount appends to a file while a second mount tails it, and a volume server is stopped, started or restarted mid-stream.
The cluster is 1 master (-defaultReplication=001), 3 volume servers, 1 filer
and 2 mounts, all as local processes. With 001 every chunk has a copy on two of
the three servers, so losing any single server must be invisible to both mounts.
| Reported scenario | Test |
|---|---|
| control: append + tail with nothing failing | TestAppendWithoutChaos |
| read a file while one volume server is down | TestReadWithVolumeServerDown |
| "STOP volumes": append + tail, kill a server mid-stream | TestAppendWhileVolumeServerStops |
| "Start volumes": append + tail with a server down, start it mid-stream | TestAppendWhileVolumeServerStarts |
| "Re-start volumes": append + tail, restart a server mid-stream | TestAppendWhileVolumeServerRestarts |
| part 2: large file copy instead of small appends | TestLargeWriteWhileVolumeServerStops |
Volume servers are dropped with SIGKILL, the closest local equivalent of a Swarm task disappearing from the overlay network: no deregistration, and the address stops answering.
Running
go build -o weed/weed ./weed
WEED_BINARY=$PWD/weed/weed go test -v -count=1 -timeout=30m ./test/fuse_failover/...
Needs FUSE and, on Linux, a working /dev/fuse. Logs from a failed run are
copied to /tmp/seaweedfs-fuse-failover-logs/.