S3 / R2 destinations

A destination with "type": "s3" delivers directly to a bucket your tenant owns. DataEgress supports real AWS S3 and any S3-compatible provider (Cloudflare R2, MinIO, etc.) via an optional endpoint override.

Creating an S3 destination

await dataEgress.createDestination({
  tenantId: "acme",
  id: "acme-s3",
  type: "s3",
  s3: {
    bucket: "acme-data-lake",
    region: "us-east-1",     // use "auto" for Cloudflare R2
    accessKeyId: "...",
    secretAccessKey: "...",
    prefix: "dataegress/",    // optional
    endpoint: undefined,     // set this for R2 (see below) — omit for real AWS S3
  },
});

Credentials are encrypted at rest (AES-256-GCM) and never echoed back by the API — GET/POST responses always mask secretAccessKey.

Using Cloudflare R2 instead of AWS S3

R2 speaks the S3 API. Set region: "auto" and endpoint to your account's R2 endpoint:

s3: {
  bucket: "acme-data-lake",
  region: "auto",
  endpoint: "https://<account_id>.r2.cloudflarestorage.com",
  accessKeyId: "...",
  secretAccessKey: "...",
}

Multipart uploads

Files are uploaded via @aws-sdk/lib-storage's Upload class, which automatically switches to multipart above a configured part size (8MB here). This is a deliberate correctness fix, not a nice-to-have: a naive single-PUT upload silently fails past S3's 5GB single-object limit — see GitLab #430277 for a real-world example of exactly this bug shipping to production elsewhere.

Verifying against a real bucket

pnpm verify:s3 (run from apps/web) exercises the full destination path — create destination, run export, poll to completion — against a bucket you control. It never prints secret values, only non-sensitive config (bucket, region, prefix) and the resulting manifest.

Add these to apps/web/.env.local (never commit this file — it's gitignored):

DATARELAY_VERIFY_S3_BUCKET=your-test-bucket
DATARELAY_VERIFY_S3_REGION=us-east-1        # "auto" for R2
DATARELAY_VERIFY_S3_ACCESS_KEY_ID=...
DATARELAY_VERIFY_S3_SECRET_ACCESS_KEY=...
DATARELAY_VERIFY_S3_PREFIX=dataegress-test/  # optional
DATARELAY_VERIFY_S3_ENDPOINT=               # only for R2/S3-compatible; omit for AWS S3

Then, with the dev server and Inngest dev server both running:

pnpm verify:s3

Use a dedicated test bucket (or a disposable prefix within one) — the script writes a real object at exports/acme/events/<run-id>.csv.

Not supported yet

See limitations.md — no GCS, no Azure Blob, no SFTP, no warehouse-native destinations (Snowflake/BigQuery/Redshift/Databricks) in V1. GCS is the most likely near-term addition since it shares most of the S3 destination's code path.