Skip to content

SeaweedFS is a distributed storage system for object storage (S3), file systems, Iceberg table buckets, and Lance vector buckets, designed to handle billions of files with O(1) disk access and effortless horizontal scaling.

1.7 KB
Raw Blame

FUSE volume server failover tests

Integration tests for what happens to FUSE mounts when a volume server goes away, comes back, or restarts underneath in-flight IO. They automate the manual matrix reported in discussion #10206: one mount appends to a file while a second mount tails it, and a volume server is stopped, started or restarted mid-stream.

The cluster is 1 master (-defaultReplication=001), 3 volume servers, 1 filer and 2 mounts, all as local processes. With 001 every chunk has a copy on two of the three servers, so losing any single server must be invisible to both mounts.

Reported scenario Test
control: append + tail with nothing failing TestAppendWithoutChaos
read a file while one volume server is down TestReadWithVolumeServerDown
"STOP volumes": append + tail, kill a server mid-stream TestAppendWhileVolumeServerStops
"Start volumes": append + tail with a server down, start it mid-stream TestAppendWhileVolumeServerStarts
"Re-start volumes": append + tail, restart a server mid-stream TestAppendWhileVolumeServerRestarts
part 2: large file copy instead of small appends TestLargeWriteWhileVolumeServerStops

Volume servers are dropped with SIGKILL, the closest local equivalent of a Swarm task disappearing from the overlay network: no deregistration, and the address stops answering.

Running

go build -o weed/weed ./weed
WEED_BINARY=$PWD/weed/weed go test -v -count=1 -timeout=30m ./test/fuse_failover/...

Needs FUSE and, on Linux, a working /dev/fuse. Logs from a failed run are copied to /tmp/seaweedfs-fuse-failover-logs/.