S3 Backend¶

The S3 backend supports AWS S3 and S3-compatible storage services including Cloudflare R2, MinIO, Wasabi, DigitalOcean Spaces, and more.

Installation¶

import "github.com/grokify/omnistorage/backend/s3"

Usage¶

AWS S3¶

backend, err := s3.New(s3.Config{
    Bucket: "my-bucket",
    Region: "us-east-1",
})
defer backend.Close()

// Credentials are loaded from environment or IAM role

Cloudflare R2¶

backend, err := s3.New(s3.Config{
    Bucket:   "my-bucket",
    Endpoint: "https://<account_id>.r2.cloudflarestorage.com",
    Region:   "auto",
})

MinIO (Local)¶

backend, err := s3.New(s3.Config{
    Bucket:       "my-bucket",
    Endpoint:     "http://localhost:9000",
    Region:       "us-east-1",
    UsePathStyle: true,
    DisableSSL:   true,
})

From Environment Variables¶

backend, err := s3.New(s3.ConfigFromEnv())

Environment variables:

AWS_REGION or S3_REGION
S3_BUCKET
S3_ENDPOINT (optional, for non-AWS)
AWS_ACCESS_KEY_ID
AWS_SECRET_ACCESS_KEY

Using the Registry¶

import (
    "github.com/grokify/omnistorage"
    _ "github.com/grokify/omnistorage/backend/s3"
)

backend, err := omnistorage.Open("s3", map[string]string{
    "bucket":   "my-bucket",
    "region":   "us-east-1",
    "endpoint": "", // Empty for AWS
})

Configuration¶

Config Struct¶

type Config struct {
    Bucket       string // Bucket name (required)
    Region       string // AWS region (required)
    Endpoint     string // Custom endpoint for R2, MinIO, etc.
    Prefix       string // Key prefix for all operations
    UsePathStyle bool   // Use path-style URLs (for MinIO)
    DisableSSL   bool   // Disable SSL (for local MinIO)
}

Registry Config¶

Key	Description	Required
`bucket`	S3 bucket name	Yes
`region`	AWS region	Yes
`endpoint`	Custom endpoint URL	No
`prefix`	Key prefix	No
`use_path_style`	Use path-style URLs	No
`disable_ssl`	Disable SSL	No

Features¶

The S3 backend implements ExtendedBackend:

Feature	Supported	Notes
Stat	Yes	Uses HeadObject
Copy	Yes	Server-side CopyObject
Move	Yes	Copy + Delete
Mkdir	Yes	Creates empty prefix
Rmdir	Yes	Deletes prefix

Operations¶

Write¶

w, err := backend.NewWriter(ctx, "data/file.json")
if err != nil {
    return err
}
w.Write([]byte(`{"key": "value"}`))
w.Close() // Upload happens on close

Read¶

r, err := backend.NewReader(ctx, "data/file.json")
if err != nil {
    return err
}
defer r.Close()
data, _ := io.ReadAll(r)

List¶

// List all files with prefix
files, err := backend.List(ctx, "data/")
for _, f := range files {
    fmt.Println(f)
}

Extended Operations¶

ext := backend.(*s3.Backend)

// Get object metadata
info, _ := ext.Stat(ctx, "file.txt")
fmt.Printf("Size: %d bytes\n", info.Size())
fmt.Printf("ETag: %s\n", info.Hash(omnistorage.HashMD5))

// Server-side copy (efficient, no download)
ext.Copy(ctx, "source.txt", "dest.txt")

// Server-side move
ext.Move(ctx, "old.txt", "new.txt")

Multipart Uploads¶

Large files are automatically uploaded using multipart uploads via the AWS SDK's upload manager.

Content Types¶

Set content type on upload:

w, _ := backend.NewWriter(ctx, "data.json",
    omnistorage.WithContentType("application/json"))

Error Handling¶

r, err := backend.NewReader(ctx, "missing.txt")
if errors.Is(err, omnistorage.ErrNotFound) {
    log.Println("Object not found")
}

Best Practices¶

Use regions close to your application - Reduces latency
Use server-side copy - Check Features().Copy before copying
Stream large files - Don't load entire files into memory
Close writers - Uploads complete on Close()