Architecture

The core decision: customer-provided callback, not a database connection

DataEgress never connects to your production database. You give it a fetchPage(tenantId, cursor, limit) -> { rows, nextCursor } function, exposed over HTTP (defineDataset(...).handler()). DataEgress calls that endpoint repeatedly to paginate through your data.

This is "Model A" in the design space we evaluated (see below for the others and why we rejected them for V1). It means:

• Your query, your tenant isolation, your column selection. DataEgress can't leak data across tenants because it never sees more than what your fetchPage returns for the tenantId it asked about.
• No database credentials to store, rotate, or leak. The only credentials DataEgress holds are for the destination (a bucket), not the source.
• Lower trust bar for a security review. A vendor that never touches your primary datastore is a much easier "yes" than one asking for a read replica or WAL access.
• The trade-off: DataEgress only knows what your fetchPage chooses to return. It cannot discover deleted rows on its own, and full re-exports re-run your query from the start — see limitations.md.

Models we considered and rejected for V1

Model	Why not (yet)
B — Direct DB connection	Requires production credentials; every customer becomes a security review before their first export.
C — Read replica	Same trust/ops burden as B, just one hop removed.
D — CDC / logical replication (WAL)	Correct for true incremental sync, but drags in an entire category of infrastructure (Kafka-class operational complexity) that turns this into a CDC company, not an export-delivery one. Deliberately out of scope — see limitations.md.
E — Customer-hosted worker (BYOC)	Real answer for large enterprise deals eventually, but too much onboarding friction for a V1 self-serve flow.

Model A (with an optional path to E later, once a customer's security team requires it) is what's implemented.

Request flow

1. You call POST /api/exports { tenantId, dataset, destination, format }
2. DataEgress creates an export_runs row (status=pending) and sends an
   Inngest event.
3. Inngest's processExportRun function:
   a. mark-running
   b. generate-file:  loop POST <your fetchUrl> with { tenantId, cursor, limit }
                       until nextCursor is null, streaming rows into a
                       CSV or Parquet writer as pages arrive (never holding
                       the whole dataset in memory as JS objects — see
                       docs/limitations.md for the Parquet-specific caveat).
   c. deliver-to-destination:  upload to the destination's bucket
                       (multipart-safe for large files) or DataEgress's own
                       storage for signed_url destinations.
   d. finalize:        write the manifest (checksum, row count, schema
                       version), mark the run completed.
4. GET /api/exports/:id returns the run's status and, once completed, the
   full manifest + download URL.

Retrying (automatic, via Inngest, or manual, via POST /api/exports/:id/retry) re-runs generate-file from scratch — it does not resume mid-pagination. Delivery uses a deterministic destination key (exports/{tenant}/{dataset}/{runId}.{format}), so a retry overwrites the same object rather than creating a duplicate.

Why Inngest instead of a hand-rolled queue

Background job orchestration (retries, backoff, step memoization, cron scheduling) is exactly the kind of infrastructure this product's own thesis says you shouldn't build yourself. DataEgress is built on Inngest rather than a custom queue.

Metadata storage

Run/dataset/destination/schedule metadata lives in Postgres via drizzle-orm. Locally, this runs on PGlite (real Postgres compiled to WASM, in-process, zero setup) — the schema is plain drizzle-orm/pg-core, so pointing it at a hosted Postgres instance in production is a one-line driver swap (drizzle-orm/node-postgres + a connection string), not a rewrite.

Known rough edge: don't run next build and next dev against the same PGlite data directory (.data/dataegress) at the same time — Next's build workers can race for the file lock. Delete .data/dataegress if you hit a RuntimeError: Aborted() after a build.