SeaweedFS is a distributed storage system for object storage (S3), file systems, Iceberg table buckets, and Lance vector buckets, designed to handle billions of files with O(1) disk access and effortless horizontal scaling.
S3 Vectors integration tests — design
Goal
Cover the vector-bucket API surface end-to-end against a real weed server —
the path no unit or handler-level test exercises: HTTP dispatch on
X-Amz-Target / per-path routing, authentication, catalog persistence,
policy enforcement, and (when a vector engine is reachable) the data plane.
Layout
test/s3vectors/
DESIGN.md this file
s3vectors_integration_test.go environment + test cases
Conventions follow test/s3tables/sts_integration: a TestEnvironment
brings up weed server (master + filer + volume + s3) on testutil-allocated
ports against a temp data dir, the test drives the JSON API over real HTTP,
and testing.Short() skips the suite.
What the test covers
Tier 1 — no vector engine required (runs everywhere)
These ops resolve through the catalog and the vector policy layer without an
engine RPC, so they work against a plain weed server:
CreateVectorBucket/GetVectorBucket/ListVectorBuckets/DeleteVectorBucket— happy path,encryptionConfigurationround-trip (sseType-> storedsseAlgorithm-> echoed back assseType), format filtering (an explicit non-Lance bucket is not a vector bucket).PutVectorBucketPolicy/GetVectorBucketPolicy/DeleteVectorBucketPolicy— the caller's document round-trips verbatim;s3vectors:actions are accepted, foreign-service actions/ARNs rejected;NotAction/NotResourcerejected.- Policy enforcement through the gateway: a non-owner
PutVectors/GetVectorsis denied without a grant, anAllowonarn:aws:s3vectors:...:bucket/B/*grants, aDenyvetoes anAllowon the same action, deletion restores owner-only access — the same lifecycle the package-level conformance test covers, but through signed HTTP requests. - Validation boundaries:
maxResultscaps,topKbounds,dataTyperejection,dimensionrange, empty/oversize PutVectors batches (these fail before the engine is touched, so they work engine-less too).
Tier 2 — vector engine required (skipped when absent)
TestVectorDataPlaneLifecycle spawns a real weed-vector against the
environment's cluster, mirroring the cluster the pytest suite builds in
conftest.py: --masters points at master.http.grpc, which is all the
engine needs, since it keeps tables on the filers and volume servers the
masters list and holds no S3 credentials. The binary resolves as
VECTOR_BINARY env → ./weed-vector (the Makefile's output) → PATH; the
test skips when none is found, and waitForEngineDiscovery polls a
data-plane op until the gateway routes to the registered engine instead of
503. The test then walks the full lifecycle: CreateVectorBucket →
CreateIndex → PutVectors → GetVectors → ListVectors → filtered
QueryVectors → DeleteVectors → DeleteIndex → DeleteVectorBucket.
Harness
NewTestEnvironmentmirrorssts_integration: locate theweedbinary (builtweed/weedfirst, then PATH), allocate ports withtestutil(MustFreeMiniPorts, sinceweed serverderives each gRPC port as HTTP+10000), spawnweed server -dir=<tmp> -master.port -volume.port -filer -s3.portagainst a written-s3.configidentity file, disable the unused Iceberg/Lance listeners, wait for the S3 port to serve/status, then run requests. The vectors router rides the S3 port automatically once the gateway can reach a master — no extra flag.- Requests are signed SigV4 with the
s3vectorsservice name — the gateway routes vectors traffic on the service name in the credential scope, so an unsigned ors3-signed request never reaches the vectors router. A smallvectorsClienthelper wraps POST +X-Amz-Target//op+ signing + status-code assertions; identities vary per-request for policy tests. Cleanupkills the process group and removes the temp dir.TestMain-less design: each test builds its own environment so a hung server can't take down the suite;t.Parallel()is deliberately avoided — port allocation is cheap but vector-engine bring-up is not.
Not covered here (by design)
- Auth failure modes (expired creds, bad signatures) — covered by the
sts_integrationsuite and the s3api auth tests; the vectors router shares the sameiam.AuthenticateRequestpath. - Lance/segment internals — the package's
listSegmentVectorsunit tests and the engine's own suite cover them; the integration layer only asserts the API contract.