Limitations

Read this before deciding whether DataEgress fits your use case — being explicit about what it doesn't do is more useful than a vague feature list.

By design (not on the roadmap without a reason to revisit)

• No CDC / logical replication / WAL reading. DataEgress does not, and is not planning to, watch your database for changes. See architecture.md for why.
• No direct database connection. Ever. Model A (customer callback) is the only supported integration pattern.
• No automatic change detection. DataEgress doesn't know a row changed unless your fetchPage tells it, by returning that row again. Incremental exports are supported by convention (pass a cursor like updatedAt and filter in your own query), not by DataEgress inferring anything.
• No reverse ETL. DataEgress delivers datasets to storage destinations (S3, signed URL). It does not sync data into Salesforce, HubSpot, or any other SaaS.

Out of scope for now

• No GCS, Azure Blob, or SFTP destinations. Only s3 (AWS S3 or S3-compatible, e.g. R2) and signed_url exist today.
• No warehouse-native destinations — Snowflake, BigQuery, Redshift, Databricks. Land the file in S3/R2 and load it yourself for now.
• No true CDC-free incremental delivery UI. The API accepts cursor conceptually (your fetchPage receives one), but there's no dashboard affordance yet for "only export what changed since the last run" beyond what you implement in your own query.
• No file retention policy. Generated files aren't automatically deleted.
• No RBAC beyond destination-level tenant scoping. There's no user accounts / permissions system.
• No billing or usage metering. Pricing in the business plan isn't wired into the product yet.
• No MCP server yet — the REST API is stable enough that adding one should be quick; see AGENTS.md for the planned tool shapes.

Real technical trade-offs worth knowing about

• Retries re-run from the start, not from where they failed. A retry (automatic or manual) re-executes the full generate-file step — re-fetching every page from your fetchUrl — rather than resuming from the row count where it stopped. Delivery is still safe (same destination key, so a retry overwrites rather than duplicates), but a retry after 800K of 1M rows re-fetches all 1M.
• Parquet export accumulates encoded bytes in memory until the file is closed. Each page's rows are encoded into Parquet binary form immediately (never held as raw JS row objects beyond one page), but the encoded output itself grows in memory across the whole run rather than streaming to disk row-group by row-group. Verified fine through 1M rows; large enough datasets (likely tens of millions of rows) would need a truly streaming writer or a chunked multi-file output — not yet built.
• CSV export does truly stream to disk (near-constant memory regardless of row count) — this asymmetry with Parquet is a known gap, not a design goal.
• No enforced HTTPS on fetchUrl. DataEgress will happily call an http:// endpoint in dev; nothing currently blocks that in production configs either.
• PGlite (local Postgres) doesn't like concurrent processes. Don't run next build and next dev against the same .data/dataegress directory at the same time — see architecture.md.
• Single-region, single-process job execution locally. Fine for the volumes tested (1M rows in ~3-4s for CSV, ~5s for Parquet); not load-tested beyond that.

When NOT to use DataEgress

• You need real-time/streaming replication, not periodic exports — look at CDC tools instead.
• You need to sync data into Salesforce/HubSpot/etc. — that's reverse ETL, a different problem.
• You need warehouse-native delivery (Snowflake Data Sharing, BigQuery Data Transfer) today, not "land a file in S3 and load it yourself."
• Your compliance requirements need a vetted subprocessor with SOC 2 and a DPA today — see security.md.