DataEgressGet started
← All docs

Large Parquet exports

Problem: a customer's data team wants Parquet, not CSV — smaller files, typed columns, and it loads straight into DuckDB/Spark/pandas without a schema-guessing pass. Writing Parquet correctly (row groups, compression, matching schema across pages) from application code is more work than CSV.

When to use DataEgress for this

  • Your buyer is a data engineer, not just an end user clicking "export." Ours research found this repeatedly: Parquet support is the signal a serious data-consuming customer looks for before trusting an export pipeline.
  • You want compression for free — in our own benchmark, the same 1M-row dataset was 95.6MB as CSV and 15.1MB as Parquet (~6.3x smaller), same data, verified byte-for-byte on read-back.
  • You need a schema version alongside the data, not just column names.

When NOT to use DataEgress for this

  • Your consumer explicitly wants CSV (e.g. opening in Excel, or a legacy loader that doesn't support Parquet). Use format: "csv".
  • You need Parquet with more than a few tens of millions of rows in a single file — V1's writer holds the encoded output in memory until the file closes (see limitations.md); it's proven fine at 1M rows but isn't a fully streaming-to-disk writer yet.

Architecture

Each page fetched from your fetchPage becomes one Parquet row group, encoded immediately rather than accumulated as raw JS objects across the whole run. See architecture.md for the full request flow; the only difference from CSV is which writer generate-file uses.

Full implementation

Identical to any other export — just set format: "parquet":

const run = await dataEgress.runExport({
  tenantId: "acme",
  dataset: "transactions",
  destination: "acme-s3",
  format: "parquet",
});

const result = await dataEgress.getExport(run.id);
// result.manifest.files[0] => { path, rows, bytes, checksum }

Column types are inferred from the first row's JS types (number + Number.isInteger → INT32, other numbers → DOUBLE, everything else → STRING, with objects JSON.stringify'd). If you need explicit typed columns beyond that inference, that's not exposed yet — file an issue.

Common errors

  • Mixed types in the same column across pages (e.g. id is a number on page 1 but a string on page 400) will produce an inconsistent schema. Normalize types in your own fetchPage before returning rows.
  • Very wide rows with large properties-style JSON blobs inflate the in-memory buffer faster than row count alone suggests — see the memory caveat in limitations.md.

Working example

See examples/parquet-export.