S3 Backups

Let users automatically sync their uploaded images to external S3-compatible storage. Each user configures their own backup destination (AWS S3, MinIO, Backblaze, Wasabi, etc.) with encrypted credentials stored securely in KV.

Why It's Needed

R2 is reliable, but some users want redundancy or prefer keeping data in their own S3 buckets. This gives them control over backups without you managing the storage. Users can restore images independently if something goes wrong.

Setup

Enable backups by ensuring queues are configured (backups use the queue system):

ENABLE_QUEUES='true'

The BACKUP_ENCRYPTION_KEY is auto-generated by Terraform to encrypt user S3 credentials. The BACKUP_QUEUE binding is provisioned automatically when queues are enabled.

Background Sync Schedule

Configure when the nightly sync job runs:

BACKUP_SYNC_CRON='0 3 * * *'  # Daily at 3 AM UTC (default)

This cron job queues up to 100 pending images per user for background sync.

How it Works

Users configure their S3 destination via the API. Images are synced in the background after upload and on a nightly schedule.

sequenceDiagram
    participant User
    participant API
    participant KV
    participant Queue as Backup Queue
    participant R2
    participant S3 as User's S3 Bucket

    User->>API: POST /v1/backup/config<br/>(endpoint, bucket, credentials)
    API->>API: Encrypt credentials with<br/>BACKUP_ENCRYPTION_KEY
    API->>KV: Store encrypted credentials
    API->>API: Save config to D1
    API-->>User: Config saved

    Note over User,S3: Later: Image Upload

    User->>API: Upload image
    API->>R2: Store original
    API->>Queue: Enqueue backup job<br/>(image_id, owner)
    API-->>User: Upload complete

    Note over Queue,S3: Background Processing

    Queue->>API: Process backup message
    API->>KV: Fetch encrypted credentials
    API->>API: Decrypt credentials
    API->>R2: Fetch image data
    API->>S3: Upload to user's bucket
    S3-->>API: Success
    API->>API: Update backup_status: synced

    Note over API,S3: Nightly Cron (3 AM UTC)

    API->>API: Find users with enabled backups
    API->>API: Find pending images (max 100/user)
    API->>Queue: Batch enqueue pending images

User Configuration

Users configure backups via API endpoints:

Create or update config:

POST /v1/backup/config
{
  "enabled": true,
  "s3_endpoint": "https://s3.amazonaws.com",
  "s3_region": "us-east-1",
  "s3_bucket": "my-pixflare-backups",
  "s3_path_prefix": "images",
  "s3_access_key_id": "AKIAIOSFODNN7EXAMPLE",
  "s3_secret_access_key": "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY"
}

Test connection:

POST /v1/backup/test

Get stats:

GET /v1/backup/stats
# Returns: total_images, pending, syncing, synced, failed counts

Backup Status

Each image has a backup_status field:

• pending: Waiting to be synced (default after upload)
• syncing: Currently uploading to S3
• synced: Successfully backed up
• failed: Upload failed (error in backup_error field)

Failed images are retried on the next nightly sync.

S3 Providers

Any S3-compatible provider works. Common examples:

Provider	Endpoint Example
AWS S3	https://s3.amazonaws.com or https://s3.{region}.amazonaws.com
Backblaze B2	https://s3.{region}.backblazeb2.com
Wasabi	https://s3.{region}.wasabisys.com
MinIO (self-hosted)	https://minio.yourdomain.com
DigitalOcean Spaces	https://{region}.digitaloceanspaces.com

Region and endpoint format vary by provider. Consult their documentation for the correct values.

Security

• Credentials encryption: S3 access keys encrypted with BACKUP_ENCRYPTION_KEY (AES-256) before storing in KV
• Per-user isolation: Each user can only access their own backup config
• No credential exposure: Secret keys never returned in API responses (only access key ID shown)
• Connection testing: Users can test credentials before enabling sync

Disabling

To disable backups globally, set ENABLE_QUEUES='false'. Users can disable their individual backups by setting enabled: false in their config or deleting their backup configuration via DELETE /v1/backup/config.