DataEgressGet started
← All docs

Quickstart

Get a real export running in under 10 minutes: register a dataset, create a destination, run an export, and get back a manifest with a verifiable checksum.

1. Install

npm install @dataegress/sdk

2. Define a dataset

A dataset is a fetchPage function — your own query, with your own tenant isolation. DataEgress never connects to your database directly (see architecture.md).

// app/api/dataegress/transactions/route.ts  (Next.js App Router example)
import { defineDataset } from "@dataegress/sdk";
import { db } from "@/lib/db";

const transactionsDataset = defineDataset({
  id: "transactions",
  fetchPage: async ({ tenantId, cursor, limit }) => {
    const rows = await db.transaction.findMany({
      where: { tenantId, id: { gt: cursor ? Number(cursor) : 0 } },
      orderBy: { id: "asc" },
      take: limit,
    });
    const nextCursor = rows.length < limit ? null : String(rows[rows.length - 1].id);
    return { rows, nextCursor };
  },
});

export const POST = transactionsDataset.handler();

Deploy this route. Its URL (e.g. https://your-app.com/api/dataegress/transactions) is what you register as the dataset's fetchUrl.

3. Register the dataset with DataEgress

curl -X POST http://localhost:3001/api/datasets \
  -H 'content-type: application/json' \
  -d '{"id": "transactions", "fetchUrl": "https://your-app.com/api/dataegress/transactions"}'

4. Create a destination

curl -X POST http://localhost:3001/api/destinations \
  -H 'content-type: application/json' \
  -d '{
    "tenantId": "acme",
    "type": "s3",
    "s3": {
      "bucket": "acme-data-lake",
      "region": "us-east-1",
      "accessKeyId": "...",
      "secretAccessKey": "..."
    }
  }'

Use "type": "signed_url" instead if you just want a downloadable link and don't have a bucket yet — no s3 object needed.

5. Run an export

curl -X POST http://localhost:3001/api/exports \
  -H 'content-type: application/json' \
  -d '{"tenantId": "acme", "dataset": "transactions", "destination": "acme-s3", "format": "parquet"}'
# => { "id": "exp_...", "status": "pending", ... }

Poll it:

curl http://localhost:3001/api/exports/exp_...

When status is "completed", the response includes rowCount, checksum, downloadUrl, and a full manifest (see manifest shape).

6. Schedule it

curl -X POST http://localhost:3001/api/schedules \
  -H 'content-type: application/json' \
  -d '{
    "tenantId": "acme",
    "dataset": "transactions",
    "destination": "acme-s3",
    "format": "parquet",
    "cron": "0 2 * * *"
  }'

Cron is a standard 5-field expression, evaluated in UTC. DataEgress checks for due schedules every minute and runs them through the exact same path as a manual export — no separate "scheduled execution" logic to keep in sync.

Demo: failure + retry

The seeded demo dataset (events, tenant acme) has two chaos tenants for exercising the failure path without touching real infrastructure:

  • acme-flaky — fails once mid-export, then self-heals on Inngest's automatic retry. No user action needed.
  • acme-stuck — fails on every automatic retry until you call POST /api/exports/:id/retry (or click Retry in the dashboard), which unlocks it. Demonstrates the manual-retry path specifically.

Run either from the dashboard home page (/) or via:

curl -X POST http://localhost:3001/api/exports \
  -H 'content-type: application/json' \
  -d '{"tenantId": "acme-stuck", "dataset": "events", "destination": "demo-download", "format": "csv"}'

Where to go next