Reliable SaaS data exports (retries, monitoring, manifests)

Problem: "Build enterprise data exports with retries and monitoring" — the ask that shows up once a customer has been burned by a silent partial export before. The bar isn't "can it export data," it's "can I trust that what arrived is complete and correct, and what happens when it fails."

When to use DataEgress for this

• You need the answer to "did last night's export actually work?" to be something other than "check if the file exists and hope."
• You need retries that don't produce two deliveries of the same data if something failed halfway through.
• Your buyer will ask for row counts and checksums, not just a file.

When NOT to use DataEgress for this

• You need exactly-once semantics on the source side too (i.e. you need to guarantee no row was double-counted across two overlapping runs due to concurrent writes on your own database). DataEgress guarantees delivery is idempotent; it doesn't guarantee your own fetchPage is read-consistent under concurrent writes — that's on your query.

Architecture: what "reliable" actually means here

1. Idempotent delivery. Every export writes to a deterministic key (exports/{tenant}/{dataset}/{runId}.{format}). A retry — automatic (via Inngest, on transient failure) or manual (POST /api/exports/:id/retry) — overwrites that same object. It never creates a second file for the same run.

2. A verifiable manifest, generated on every successful run:

   {
     "export_id": "exp_9f2a1c",
     "status": "completed",
     "rows": 1000000,
     "schema_version": "1",
     "checksum": "sha256:...",
     "generated_at": "...",
     "completed_at": "...",
     "files": [{ "path": "...", "rows": 1000000, "bytes": 95563648, "checksum": "sha256:..." }]
   }
   

   The checksum is computed by streaming the actual written file through SHA-256 — verified in our own testing to match byte-for-byte on the downloaded file, not just asserted.

3. Honest failure states. A failed run's error, errorStage (e.g. "generating_file", "delivering"), and rowsProcessedAtFailure are all populated — enough to answer "what failed, at what point, and how much had already been processed" without reading logs.

Full implementation: handling a failed run

const run = await dataEgress.getExport(runId);

if (run.status === "failed") {
  console.log(`Failed at ${run.errorStage}: ${run.error}`);
  console.log(`Processed ${run.rowsProcessedAtFailure} rows before failing.`);

  // Fix whatever caused it (expired credentials, a bug in fetchPage, etc.), then:
  await dataEgress.retryExport(runId);
}

Common errors

• Treating status: "running" for a long time as stuck — check rowsProcessedAtFailure/timestamps rather than assuming; large exports legitimately take longer (though 1M rows completes in single-digit seconds in our testing).
• Retrying a run that isn't failed — POST /api/exports/:id/retry returns 409 not_retryable with a clear message if the run is pending, running, or already completed.

Working example

See examples/nextjs for a full dashboard-adjacent integration including polling a run to completion and handling failure.