Architecture

The core decision: customer-provided callback, not a database connection

DataEgress never connects to your production database. You give it a fetchPage(tenantId, cursor, limit) -> { rows, nextCursor } function, exposed over HTTP (defineDataset(...).handler()). DataEgress calls that endpoint repeatedly to paginate through your data.

This is the callback-based model described below, chosen over the alternatives in the table that follows. It means:

• Your query, your tenant isolation, your column selection. DataEgress can't leak data across tenants because it never sees more than what your fetchPage returns for the tenantId it asked about.
• No database credentials to store, rotate, or leak. The only credentials DataEgress holds are for the destination (a bucket), not the source.
• Lower trust bar for a security review. A vendor that never touches your primary datastore is a much easier "yes" than one asking for a read replica or WAL access.
• The trade-off: DataEgress only knows what your fetchPage chooses to return. It cannot discover deleted rows on its own, and full re-exports re-run your query from the start — see limitations.md.

Alternatives and why we don't do them

Approach	Why not (yet)
Direct DB connection	Requires production credentials; every customer becomes a security review before their first export.
Read replica	Same trust/ops burden as a direct connection, just one hop removed.
CDC / logical replication (WAL)	Correct for true incremental sync, but drags in an entire category of infrastructure (Kafka-class operational complexity) that turns this into a CDC company, not an export-delivery one. Deliberately out of scope — see limitations.md.
Customer-hosted worker (BYOC)	A real option for large enterprise deals whose security team requires it eventually, but too much onboarding friction to require of every customer up front.

The callback model is what's implemented today, with BYOC available as an upgrade path for customers whose security requirements call for it.

Request flow

1. You call POST /api/exports { tenantId, dataset, destination, format }
2. DataEgress creates an export_runs row (status=pending) and sends an
   Inngest event.
3. Inngest's processExportRun function:
   a. mark-running
   b. generate-file:  loop POST <your fetchUrl> with { tenantId, cursor, limit }
                       until nextCursor is null, streaming rows into a
                       CSV or Parquet writer as pages arrive (never holding
                       the whole dataset in memory as JS objects — see
                       docs/limitations.md for the Parquet-specific caveat).
   c. deliver-to-destination:  upload to the destination's bucket
                       (multipart-safe for large files) or DataEgress's own
                       storage for signed_url destinations.
   d. finalize:        write the manifest (checksum, row count, schema
                       version), mark the run completed.
4. GET /api/exports/:id returns the run's status and, once completed, the
   full manifest + download URL.

Retrying (automatic, via Inngest, or manual, via POST /api/exports/:id/retry) re-runs generate-file from scratch — it does not resume mid-pagination. Delivery uses a deterministic destination key (exports/{tenant}/{dataset}/{runId}.{format}), so a retry overwrites the same object rather than creating a duplicate.

Why Inngest instead of a hand-rolled queue

Background job orchestration (retries, backoff, step memoization, cron scheduling) is exactly the kind of infrastructure this product's own thesis says you shouldn't build yourself. DataEgress is built on Inngest rather than a custom queue.

Tenant model

Two independent levels of multi-tenancy, easy to conflate:

• Organization — a DataEgress account/workspace (created automatically at signup, one per user for now — no invitations/teams/multi-org switching yet). Owns datasets, destinations, schedules and export_runs. Every dashboard API route requires a valid session and scopes all reads/writes to session.organization.id server-side — never just by hiding UI.
• tenantId — an organization's own end-customer identifier (e.g. one of Acme's enterprise customers). Arbitrary string, chosen by the organization, orthogonal to organization id.

datasets and destinations use a composite (organization_id, id) primary key because their id is human-chosen (e.g. "events") and therefore only unique per organization — two organizations can each have a dataset called "events" without colliding. schedules and export_runs keep a single-column id since those are always randomly generated.

The public /demo page needs no login. It's pinned to a single fixed organization that every unauthenticated request is scoped to — an anonymous caller can only ever reach that demo organization's data, never a real workspace's, even by guessing an id.

Metadata storage

Run/dataset/destination/schedule metadata lives in Postgres via drizzle-orm. Locally, this runs on PGlite (real Postgres compiled to WASM, in-process, zero setup) — the schema is plain drizzle-orm/pg-core, so pointing it at a hosted Postgres instance in production is a one-line driver swap (drizzle-orm/node-postgres + a connection string), not a rewrite.

Billing and entitlements

Configuration survives, execution respects current entitlements. Downgrading (including all the way to Sandbox) never deletes or auto-disables a dataset, destination, or schedule. What changes is whether those resources are still allowed to actually run — enforced at execution time, not just at creation time, because a resource created under a higher plan can outlive the subscription that created it:

• Scheduling: the cron dispatcher re-checks entitlements for every due schedule before creating an export run. A blocked occurrence is counted separately in the dispatch summary — never as a failed run (no run is ever created for it).
• Customer-owned S3/R2: the delivery step re-checks the org's entitlement immediately before the upload call, every time — a destination created while the feature was available keeps existing, but stops actually delivering the moment it isn't.
• No backfill. A schedule blocked for days does not fire a burst of missed runs on resubscribe — the dispatcher advances lastRunAt on every due tick, dispatched or skipped alike, so it always resumes at the next real occurrence, never a retroactive catch-up.
• In-flight runs are never interrupted. These checks run once, as a precondition immediately before their one side effect (creating a run; uploading to a customer bucket) — never mid-transfer, and never on any other step. A run already dispatched under paid entitlements completes normally even if the org loses entitlements while it's executing.

Usage periods are billing-aware. Included-delivery quotas are evaluated against the org's real billing cadence, not always a calendar month —

State	Usage period
Sandbox / internal trial / non-entitled (unpaid, canceled, ...)	Calendar month, UTC
Paid, monthly interval	The cached Stripe subscription period, exactly
Paid, annual interval	A monthly window anchored to billingAnchorDay (billed yearly, usage still resets monthly)
Public demo	Unmetered, unconditionally

The annual anchor window clamps independently per month (day 31 → Feb 28/29 → back to 31 in March, never drifting), preserves the real anchor's time-of-day, and uses [start, end) semantics throughout. A paid subscription with a missing or inconsistent period cache fails loudly rather than silently falling back to calendar-month accounting — wrong billing-period accounting is a worse failure mode than a visible error.