Scheduled S3 exports

Problem: "Every night at 2 AM, send all of a tenant's data to their own S3 bucket." Doing this yourself means a cron trigger, a query that doesn't time out or blow up memory on large tables, multipart upload handling, and somewhere to see whether last night's run actually succeeded.

When to use DataEgress for this

• You already have (or can write) a query that returns a tenant's rows, and you need it delivered on a recurring schedule to storage the tenant owns.
• You need retry-without-duplicate-delivery: if the 2 AM run fails, the fix shouldn't be "let's hope the 2:15 retry didn't send it twice."
• You want a manifest (row count, checksum) the receiving side can verify against, not just "trust that the file arrived intact."

When NOT to use DataEgress for this

• The schedule needs sub-minute precision. DataEgress's dispatcher checks for due schedules once a minute (see architecture.md).
• You need the destination to be a data warehouse directly (Snowflake, BigQuery). Land it in S3 and load it yourself for now — see limitations.md.

Architecture

cron dispatcher (every minute)
  -> finds schedules whose cron fired since lastRunAt
  -> creates an export_run + sends the same event a manual export sends
  -> processExportRun: paginate your fetchPage -> write file -> upload to
     the tenant's destination -> write manifest

Full implementation

// 1. Define the dataset once
import { defineDataset } from "@dataegress/sdk";
import { db } from "@/lib/db";

const transactions = defineDataset({
  id: "transactions",
  fetchPage: async ({ tenantId, cursor, limit }) => {
    const rows = await db.transaction.findMany({
      where: { tenantId, id: { gt: cursor ? Number(cursor) : 0 } },
      orderBy: { id: "asc" },
      take: limit,
    });
    return { rows, nextCursor: rows.length < limit ? null : String(rows.at(-1)!.id) };
  },
});
export const POST = transactions.handler(); // mount at e.g. /api/dataegress/transactions

// 2. Register the dataset, create the tenant's destination, and schedule it
import { DataEgress } from "@dataegress/sdk";

const dataEgress = new DataEgress({ baseUrl: process.env.DATARELAY_URL! });

await fetch(`${process.env.DATARELAY_URL}/api/datasets`, {
  method: "POST",
  headers: { "content-type": "application/json" },
  body: JSON.stringify({ id: "transactions", fetchUrl: "https://your-app.com/api/dataegress/transactions" }),
});

await dataEgress.createDestination({
  tenantId: "acme",
  id: "acme-s3",
  type: "s3",
  s3: { bucket: "acme-data-lake", region: "us-east-1", accessKeyId: "...", secretAccessKey: "..." },
});

await dataEgress.createSchedule({
  tenantId: "acme",
  dataset: "transactions",
  destination: "acme-s3",
  format: "csv",
  cron: "0 2 * * *", // every day at 02:00 UTC
});

Common errors

• destination_not_found — you scheduled before creating the destination. Create it first.
• A run completes but the tenant says the file never arrived — check the run's manifest.files[0].path; it includes any prefix you configured on the destination. Confirm the tenant is looking in the right sub-path.
• Schedule never fires — confirm enabled: true (PATCH /api/schedules/:id { "enabled": true }) and that the cron string parses; createSchedule validates it up front and returns invalid_cron with the specific problem if not.

Working example

See examples/scheduled-s3.