SeaweedFS is a distributed storage system for object storage (S3), file systems, Iceberg table buckets, and Lance vector buckets, designed to handle billions of files with O(1) disk access and effortless horizontal scaling.
Go blob-storagecloud-drivedistributed-file-systemdistributed-storageerasure-codingfusehadoop-hdfshdfskuberneteslancelancedbobject-storageposixs3s3-storageseaweedfstiered-file-systemvector-search
Integration Example
This shows how to integrate the new policy engine with the existing S3ApiServer.
Minimal Integration
// In s3api_server.go - modify NewS3ApiServerWithStore function
func NewS3ApiServerWithStore(router *mux.Router, option *S3ApiServerOption, explicitStore string) (s3ApiServer *S3ApiServer, err error) {
// ... existing code ...
// Create traditional IAM
iam := NewIdentityAccessManagementWithStore(option, explicitStore)
s3ApiServer = &S3ApiServer{
option: option,
iam: iam, // Keep existing for compatibility
randomClientId: util.RandomInt32(),
filerGuard: security.NewGuard([]string{}, signingKey, expiresAfterSec, readSigningKey, readExpiresAfterSec),
cb: NewCircuitBreaker(option),
credentialManager: iam.credentialManager,
bucketConfigCache: NewBucketConfigCache(5 * time.Minute),
}
// Optional: Wrap with policy-backed IAM for enhanced features
if option.EnablePolicyEngine { // Add this config option
// Option 1: Create and set legacy IAM separately
policyBackedIAM := NewPolicyBackedIAM()
policyBackedIAM.SetLegacyIAM(iam)
// Option 2: Create with legacy IAM in one call (convenience method)
// policyBackedIAM := NewPolicyBackedIAMWithLegacy(iam)
// Load existing identities as policies
if err := policyBackedIAM.LoadIdentityPolicies(); err != nil {
glog.Warningf("Failed to load identity policies: %v", err)
}
// Replace IAM with policy-backed version
s3ApiServer.iam = policyBackedIAM
}
// ... rest of existing code ...
}
Router Integration
// In registerRouter function, replace bucket policy handlers:
// Old handlers (if they exist):
// bucket.Methods(http.MethodGet).HandlerFunc(s3a.GetBucketPolicyHandler).Queries("policy", "")
// bucket.Methods(http.MethodPut).HandlerFunc(s3a.PutBucketPolicyHandler).Queries("policy", "")
// bucket.Methods(http.MethodDelete).HandlerFunc(s3a.DeleteBucketPolicyHandler).Queries("policy", "")
// New handlers with policy engine:
if policyBackedIAM, ok := s3a.iam.(*PolicyBackedIAM); ok {
// Use policy-backed handlers
bucket.Methods(http.MethodGet).HandlerFunc(track(s3a.iam.Auth(s3a.cb.Limit(policyBackedIAM.GetBucketPolicyHandler, ACTION_READ)), "GET")).Queries("policy", "")
bucket.Methods(http.MethodPut).HandlerFunc(track(s3a.iam.Auth(s3a.cb.Limit(policyBackedIAM.PutBucketPolicyHandler, ACTION_WRITE)), "PUT")).Queries("policy", "")
bucket.Methods(http.MethodDelete).HandlerFunc(track(s3a.iam.Auth(s3a.cb.Limit(policyBackedIAM.DeleteBucketPolicyHandler, ACTION_WRITE)), "DELETE")).Queries("policy", "")
} else {
// Use existing/fallback handlers
bucket.Methods(http.MethodGet).HandlerFunc(track(s3a.iam.Auth(s3a.cb.Limit(s3a.GetBucketPolicyHandler, ACTION_READ)), "GET")).Queries("policy", "")
bucket.Methods(http.MethodPut).HandlerFunc(track(s3a.iam.Auth(s3a.cb.Limit(s3a.PutBucketPolicyHandler, ACTION_WRITE)), "PUT")).Queries("policy", "")
bucket.Methods(http.MethodDelete).HandlerFunc(track(s3a.iam.Auth(s3a.cb.Limit(s3a.DeleteBucketPolicyHandler, ACTION_WRITE)), "DELETE")).Queries("policy", "")
}
Configuration Option
Add to S3ApiServerOption:
type S3ApiServerOption struct {
// ... existing fields ...
EnablePolicyEngine bool // Add this field
}
Example Usage
1. Existing Users (No Changes)
Your existing identities.json continues to work:
{
"identities": [
{
"name": "user1",
"credentials": [{"accessKey": "key1", "secretKey": "secret1"}],
"actions": ["Read:bucket1/*", "Write:bucket1/uploads/*"]
}
]
}
2. New Users (Enhanced Policies)
Set bucket policies via S3 API:
# Allow public read
aws s3api put-bucket-policy --bucket my-bucket --policy file://policy.json
# Where policy.json contains:
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": "*",
"Action": "s3:GetObject",
"Resource": "arn:aws:s3:::my-bucket/*"
}
]
}
3. Advanced Conditions
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": "*",
"Action": "s3:GetObject",
"Resource": "arn:aws:s3:::secure-bucket/*",
"Condition": {
"IpAddress": {
"aws:SourceIp": "192.168.1.0/24"
},
"Bool": {
"aws:SecureTransport": "true"
}
}
}
]
}
Migration Strategy
Phase 1: Enable Policy Engine (Opt-in)
- Set
EnablePolicyEngine: truein server options - Existing
identities.jsonautomatically converted to policies - Add bucket policies as needed
Phase 2: Full Policy Management
- Use AWS CLI/SDK for policy management
- Gradually migrate from
identities.jsonto pure IAM policies - Take advantage of advanced conditions and features
Testing
# Test existing functionality
go test -v -run TestCanDo
# Test new policy engine
go test -v -run TestPolicyEngine
# Test integration
go test -v -run TestPolicyBackedIAM
The integration is designed to be:
- Backward compatible - Existing setups work unchanged
- Opt-in - Enable policy engine only when needed
- Gradual - Migrate at your own pace
- AWS compatible - Use standard AWS tools and patterns