DataEgress
Log inSign up
← All docs

Quickstart

Go from nothing to a validated integration and a real export — entirely on your laptop, against your own in-memory data, no database or S3 bucket required — in well under 10 minutes.

Step Time
Sign up + create an API key ~1 min
Copy the example, run the local server ~2 min
Open a public tunnel ~1 min
Register the dataset + Validate fetchPage ~2 min
Add a destination + run your first export ~2 min

These are checkpoints, not guarantees — your mileage will vary with typing speed and how fast npm install runs. The order matters more than the timing: configure, then validate, then export. Validating first means you find out your integration is broken from one fast, unbilled check, not from a failed real export.

1. Sign up and create an API key

Create a workspace, then open API Keys in the dashboard and create one. The full key is shown once — copy it now.

# .env
DATAEGRESS_API_KEY=dte_live_...

2. Copy the example and run it locally

mkdir dataegress-quickstart && cd dataegress-quickstart
npm install @dataegress/sdk

Save this as server.ts — it's the complete, real DataEgress fetch-page protocol over an in-memory table. No fragments to fill in.

// server.ts
import { createServer } from "node:http";
import { defineDataset } from "@dataegress/sdk";

const PORT = 3999;

// 23 rows for tenant "acme" -- swap this (and the fetchPage query below) for
// a real database call once this works end to end.
const ORDERS = Array.from({ length: 23 }, (_, i) => ({
  id: i + 1,
  tenantId: "acme",
  total: Math.round((12 + i * 4.25) * 100) / 100,
  createdAt: new Date(Date.now() - (23 - i) * 3_600_000).toISOString(),
}));

const ordersDataset = defineDataset({
  id: "orders",
  fetchPage: async ({ tenantId, cursor, limit }) => {
    const tenantRows = ORDERS.filter((o) => o.tenantId === tenantId);
    const offset = cursor ? Number(cursor) : 0;
    const page = tenantRows.slice(offset, offset + limit);
    const nextOffset = offset + page.length;
    return {
      rows: page,
      nextCursor: nextOffset < tenantRows.length ? String(nextOffset) : null,
    };
  },
});

const handler = ordersDataset.handler();

createServer(async (req, res) => {
  if (req.url !== "/fetch-orders" || req.method !== "POST") {
    res.writeHead(404).end();
    return;
  }
  const chunks: Buffer[] = [];
  for await (const chunk of req) chunks.push(chunk);
  const request = new Request(`http://localhost:${PORT}/fetch-orders`, {
    method: "POST",
    headers: { "content-type": "application/json" },
    body: Buffer.concat(chunks),
  });
  const response = await handler(request);
  res.writeHead(response.status, { "content-type": "application/json" });
  res.end(await response.text());
}).listen(PORT, () => {
  console.log(`orders dataset listening on http://localhost:${PORT}/fetch-orders`);
});

Run it:

npx tsx server.ts
# or, Node 22+: node --experimental-strip-types server.ts

orders dataset listening on http://localhost:3999/fetch-orders

Sanity-check it yourself before involving DataEgress at all — this is the exact request DataEgress will send:

curl -X POST http://localhost:3999/fetch-orders \
  -H 'content-type: application/json' \
  -d '{"tenantId":"acme","cursor":null,"limit":10}'

{"rows":[{"id":1,"tenantId":"acme","total":12, ...}, ...10 rows total...],"nextCursor":"10"}

nextCursor: "10" means there's more data — pass it back as cursor to get rows 11–20, and so on until nextCursor comes back null. limit is always whatever DataEgress sends (5 during Validate, larger during a real export) — this example honors it exactly rather than hardcoding a page size, same as a real implementation should.

3. Expose it publicly with a tunnel

DataEgress runs on infrastructure that isn't your laptop. It cannot call localhost or a private IP (127.0.0.1, 10.x, 192.168.x, etc.) — this is a deliberate SSRF protection, not a bug or a missing feature. Your fetchUrl has to be a public HTTPS/HTTP address, so for local development you need a tunnel in front of your local server.

Cloudflare Tunnel — no account required, works in one command:

brew install cloudflared        # macOS; see cloudflared's own docs for Linux/Windows
cloudflared tunnel --url http://localhost:3999

This prints a public URL that changes every run, shaped like https://random-words-here.trycloudflare.com. Copy it — you'll use https://random-words-here.trycloudflare.com/fetch-orders as the fetchUrl below. (These "quick tunnels" are meant for testing, not production traffic — fine for this quickstart, not what you'd point a real customer's schedule at.)

Alternative: ngrok (ngrok http 3999) works the same way but requires a free account and an auth token first.

4. Register the dataset

import { DataEgress } from "@dataegress/sdk";

const dataEgress = new DataEgress({ apiKey: process.env.DATAEGRESS_API_KEY });

await dataEgress.registerDataset({
  id: "orders",
  fetchUrl: "https://random-words-here.trycloudflare.com/fetch-orders",
});

Or register it from the dashboard instead of code — paste the same tunnel URL into fetchUrl.

5. Validate fetchPage

Open the dataset in the dashboard and find Validate fetchPage. This makes one real, unbilled call to your fetchUrl with the same request shape a real export sends — no export runs, nothing uploads, no delivery quota is used.

  1. Enter a tenantId — acme, matching the data in the example above. This field is never inferred or defaulted silently; use the same tenant a real export for this dataset would.
  2. Click Validate fetchPage.

Expected result:

Connection valid — received 5 rows in ~400 ms (more pages available).

  • rowsReceived — how many rows your fetchUrl returned for this one call. Validate always asks for a small page (5 rows), regardless of your dataset's real size, so this checks the contract, not your data volume.
  • hasNextCursor ("more pages available") — whether your response included a non-null nextCursor. true here just means your dataset has more than 5 rows for this tenant, which is expected and fine — it's not an error or a warning.

If it fails instead, the dashboard shows a structured reason (title, likely cause, how to fix) rather than a raw error — see Troubleshooting below for the most common ones, or the full troubleshooting guide if yours isn't one of them.

6. Add a destination and run your first export

No bucket yet? Use signed_url — zero setup, DataEgress hosts the file and gives you a download link.

await dataEgress.createDestination({
  tenantId: "acme",
  id: "acme-download",
  type: "signed_url",
});

const run = await dataEgress.runExport({
  tenantId: "acme",
  dataset: "orders",
  destination: "acme-download",
  format: "csv",
});

let result = run;
while (result.status === "pending" || result.status === "running") {
  await new Promise((r) => setTimeout(r, 1000));
  result = await dataEgress.getExport(run.id);
}

console.log(result.status === "completed" ? `Done: ${result.rowCount} rows -> ${result.downloadUrl}` : result.error);

Or trigger it from the dashboard instead. Once status is "completed", the response includes rowCount, checksum, downloadUrl, and a full manifest (row count and checksum per file, so a downstream consumer can verify delivery independently).

Have a real bucket already? Point the destination at type: "s3" instead — see S3 & R2 destinations.

If the run fails instead of completing, the run detail page explains what happened and how to fix it the same way Validate does — see the troubleshooting guide for the full list, including S3/R2 delivery failures.

Raw HTTP alternative (protocol reference)

The SDK above is the recommended path. If you'd rather not add a dependency, or you're implementing fetchPage in a language the SDK doesn't cover, this is the exact contract defineDataset(...).handler() implements — build it by hand against this instead.

DataEgress sends, one call per page, until you return nextCursor: null:

POST /your-fetch-url HTTP/1.1
content-type: application/json

{"tenantId": "acme", "cursor": null, "limit": 5000}

Your endpoint must respond 200 OK with:

{
  "rows": [{ "...": "your columns" }],
  "nextCursor": "opaque-string-or-null"
}
  • rows — always an array, even when empty (an empty array + nextCursor: null is a valid "no data for this tenant" response).
  • nextCursor — an opaque string to echo back as cursor on the next call, or null when this was the last page. DataEgress never parses or interprets it — it's yours to shape (an offset, a row id, a composite key, anything your own query can resume from).
  • cursor in the request is null on the first page.
  • limit is DataEgress's choice, not yours — always honor whatever value is sent rather than hardcoding a page size (5 during Validate; larger for a real export).

Troubleshooting

The dashboard's Validate result already shows one of these for a failed check — this table is the same information, for quick reference. See the troubleshooting guide for the complete list, plus S3/R2 delivery failures and retry behavior.

Symptom Cause Fix
"This URL isn't reachable from DataEgress" / blocked URL fetchUrl is localhost, 127.0.0.1, or another private/internal address Register the public tunnel URL (step 3), not the local one
Validate returns 404 Your tunnel or route path doesn't match what you registered Confirm the tunnel is running and fetchUrl ends in the exact path your handler is mounted at (/fetch-orders above)
Validate returns 401/403 Your endpoint requires its own auth header Remove auth from this route — DataEgress never sends one, the unguessable URL is the only credential. See Security
Timeout / unreachable Local server not running, tunnel dropped, or a firewall is blocking it Confirm npx tsx server.ts is still running and the tunnel process is still up; re-copy the tunnel URL if it restarted (it changes every run)
"Response shape doesn't match the fetchPage contract" (invalid shape) Response isn't exactly { rows: Array, nextCursor: string | null } Return that shape exactly — defineDataset(...).handler() already does this for you if you're on the SDK

Scheduling

Once a manual export works, automate it:

await dataEgress.createSchedule({
  tenantId: "acme",
  dataset: "orders",
  destination: "acme-download",
  format: "csv",
  cron: "0 2 * * *", // every day at 02:00 UTC
});

Cron is a standard 5-field expression, evaluated in UTC. DataEgress checks for due schedules periodically (execution may be delayed by up to ~30 minutes) and runs them through the exact same path as a manual export — no separate "scheduled execution" logic to keep in sync.

Demo: failure + retry

The seeded demo dataset (events, tenant acme) has two chaos tenants for exercising the failure path without touching real infrastructure — useful if you're exploring the product rather than integrating it yet:

  • acme-flaky — fails once mid-export, then self-heals on Inngest's automatic retry. No user action needed.
  • acme-stuck — fails on every automatic retry until you call POST /api/exports/:id/retry (or click Retry in the dashboard), which unlocks it. Demonstrates the manual-retry path specifically.

Run either from the dashboard home page (/) or via:

curl -X POST http://localhost:3001/api/exports \
  -H 'content-type: application/json' \
  -d '{"tenantId": "acme-stuck", "dataset": "events", "destination": "demo-download", "format": "csv"}'

(That last example targets a local DataEgress dev instance, not the hosted one — it's for exploring this repo, not part of the integration flow above.)

Where to go next

Quickstart — DataEgress