S3 / R2 destinations

A destination with "type": "s3" delivers directly to a bucket your tenant owns. DataEgress supports real AWS S3 and any S3-compatible provider (Cloudflare R2, MinIO, etc.) via an optional endpoint override.

Creating an S3 destination

await dataEgress.createDestination({
  tenantId: "acme",
  id: "acme-s3",
  type: "s3",
  s3: {
    bucket: "acme-data-lake",
    region: "us-east-1",     // use "auto" for Cloudflare R2
    accessKeyId: "...",
    secretAccessKey: "...",
    prefix: "dataegress/",    // optional
    endpoint: undefined,     // set this for R2 (see below) — omit for real AWS S3
  },
});

Credentials are encrypted at rest (AES-256-GCM) and never echoed back by the API — GET/POST responses always mask secretAccessKey.

Using Cloudflare R2 instead of AWS S3

R2 speaks the S3 API. Set region: "auto" and endpoint to your account's R2 endpoint:

s3: {
  bucket: "acme-data-lake",
  region: "auto",
  endpoint: "https://<account_id>.r2.cloudflarestorage.com",
  accessKeyId: "...",
  secretAccessKey: "...",
}

Required IAM permissions

The access key you register needs exactly this, scoped to the bucket (and prefix, if you're using one) — nothing broader:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": ["s3:PutObject", "s3:AbortMultipartUpload", "s3:ListMultipartUploadParts"],
      "Resource": "arn:aws:s3:::acme-data-lake/dataegress/*"
    }
  ]
}

s3:PutObject writes the file. s3:AbortMultipartUpload and s3:ListMultipartUploadParts are needed because large exports upload in parts (see Multipart uploads below) — without them, a failed or retried upload can't clean up or resume a part it already started. DataEgress never lists, deletes, or reads back objects other than to generate a short-lived signed URL for your own dashboard preview of what it just wrote, so nothing beyond these three actions is ever needed. See troubleshooting.md if a real upload fails despite this.

Multipart uploads

Files are uploaded via @aws-sdk/lib-storage's Upload class, which automatically switches to multipart above a configured part size (8MB here). This is a deliberate correctness fix, not a nice-to-have: a naive single-PUT upload silently fails past S3's 5GB single-object limit — see GitLab #430277 for a real-world example of exactly this bug shipping to production elsewhere.

Verifying against a real bucket

The fastest way to check a bucket and credentials actually work is to create the destination via the SDK or dashboard (as above) and run one real export against it:

const run = await dataEgress.runExport({
  tenantId: "acme",
  dataset: "orders", // or any dataset you've already registered
  destination: "acme-s3",
  format: "csv",
});

Poll dataEgress.getExport(run.id) until status is "completed" or "failed". A failure's error field names the actual problem (bad credentials, wrong region, bucket doesn't exist, etc.) rather than a generic message. Use a dedicated test bucket (or a disposable prefix within one) for this — a successful run writes a real object at <prefix>exports/<tenantId>/<dataset>/<run-id>.<format> (the same key a retry overwrites rather than duplicates).

Not supported yet

See limitations.md — no GCS, no Azure Blob, no SFTP, no warehouse-native destinations (Snowflake/BigQuery/Redshift/Databricks) in V1. GCS is the most likely near-term addition since it shares most of the S3 destination's code path.