DataEgress
Log inSign up
← All docs

Troubleshooting

When a dataset validation or an export run fails, the dashboard already shows what happened, the likely cause, and how to fix it — this page is the same information laid out by symptom, for when you want more detail or a link to share.

Every entry below corresponds to a specific, stable failure code shown in the dashboard (in a failed run's "Technical details", or a Validate result) — the codes are consistent between Validate fetchPage (a pre-flight check) and a real export run: the same underlying problem is always diagnosed the same way in both places.

fetchPage / your dataset callback

These all mean DataEgress could not get a usable response from your dataset's fetchUrl. See the quickstart for the full fetchPage contract.

blocked-url

Symptom: "This URL isn't reachable from DataEgress" / "fetchUrl is not a public address."

Likely cause: The registered fetchUrl is localhost, 127.0.0.1, a private IP range (10.x, 192.168.x, 172.16-31.x), or a cloud metadata address. DataEgress refuses these on every call, not just at registration — this is a deliberate SSRF protection, not a bug.

Fix: Register a public HTTPS/HTTP address. For local development, expose your server with a tunnel (see the quickstart) and register that public URL instead.

Verify: curl the public URL yourself from a machine that isn't your own network — if it works there, DataEgress should reach it too.

unreachable

Symptom: "Could not reach your fetchUrl."

Likely cause: The host is down, the URL has a typo, DNS doesn't resolve, or a firewall — or a dropped local-development tunnel — is refusing the connection.

Fix: Confirm the URL is correct and the server behind it (and tunnel, if any) is still running and publicly reachable. A tunnel URL changes every time you restart it — re-register the current one.

Verify: curl -X POST <fetchUrl> -d '{"tenantId":"...","cursor":null,"limit":5}' from a different machine/network than the one running the server.

timeout

Symptom: "Your fetchUrl did not respond in time."

Likely cause: Your endpoint is slow, hanging, or never sends a response for this specific request shape.

Fix: Confirm your handler always returns a response for a POST with a JSON body, and that nothing in front of it (a proxy, a load balancer) is silently dropping the connection instead of responding.

Verify: Time your own curl against the endpoint with the exact request DataEgress sends (see the quickstart's raw HTTP reference).

http-401-403

Symptom: "Your fetchUrl rejected the request as unauthorized."

Likely cause: Your handler expects an authorization header or session cookie. DataEgress never sends one — the unguessable fetchUrl itself is the only credential it uses.

Fix: Remove any auth requirement from this specific route. If you need this endpoint to also serve authenticated traffic, mount the DataEgress handler at a separate, dedicated path instead.

http-404

Symptom: "Your fetchUrl returned 404."

Likely cause: The route isn't mounted at that exact path, it only accepts a different HTTP method, or — very commonly during local development — a tunnel URL from an earlier session is still registered.

Fix: Confirm defineDataset(...).handler() is mounted at exactly the registered fetchUrl and is deployed; for local dev, confirm the tunnel URL you registered is the one currently running.

http-429

Symptom: "Your fetchUrl rate-limited DataEgress."

Likely cause: A rate limit on your server (or a proxy/WAF in front of it) is throttling DataEgress's requests. This is expected to happen if the limit is tuned for human traffic — DataEgress calls fetchUrl repeatedly, by design, to paginate through a dataset.

Fix: Raise the limit for this route, or exclude DataEgress's calls from it (there's no special header to key off — the route itself is the signal).

http-5xx

Symptom: "Your fetchUrl returned a server error," after DataEgress already retried.

Likely cause: An unhandled exception in your fetchPage implementation, or a downstream dependency (your database, an internal API) failing.

Fix: Check your own server logs for these requests — DataEgress can see that your endpoint failed, not why.

http-client-error

Symptom: "Your fetchUrl rejected the request," a 4xx other than 401/403/404/429.

Likely cause: The request DataEgress sends doesn't match what your handler expects.

Fix: Confirm your handler accepts POST with a JSON body of { tenantId, cursor, limit } and doesn't require anything else (query params, a specific content type beyond application/json, etc.).

invalid-json

Symptom: "Your fetchUrl did not return valid JSON."

Likely cause: Your handler returned something other than JSON — an HTML error page, plain text, or an empty body — usually because an uncaught exception produced a framework's default error page instead of your handler's own response.

Fix: Return a JSON body shaped like { "rows": [...], "nextCursor": null | "..." }. defineDataset(...).handler() from @dataegress/sdk already guarantees this for every response it sends, including its own error responses.

invalid-shape

Symptom: "Your fetchUrl's response doesn't match the fetchPage contract."

Likely cause: rows isn't an array, or nextCursor isn't a string or null.

Fix: Return { rows, nextCursor } exactly — rows is always an array (empty on the last page), nextCursor is a string when there's more data or null when done.

S3 / R2 destinations

These apply to a type: "s3" destination pointed at your own bucket — see S3 & R2 destinations for setup. A signed_url destination (DataEgress-hosted download link) can't hit any of these; there's no customer bucket involved.

See S3 & R2 destinations for the exact, minimum IAM policy the access key you register needs.

s3-access-denied

Symptom: "Access denied writing to your S3/R2 bucket."

Likely cause: The IAM user/role behind the registered access key doesn't have s3:PutObject (or the required multipart permissions) on this bucket or prefix.

Fix: Attach a policy granting the permissions listed above, scoped to this bucket (and prefix, if applicable).

s3-no-such-bucket

Symptom: "The destination bucket doesn't exist."

Likely cause: A typo in the bucket name, or the bucket belongs to a different AWS account than the credentials you registered.

Fix: Double-check the bucket name in the AWS console under the same account the access key belongs to.

s3-invalid-credentials

Symptom: "Your S3/R2 credentials were rejected."

Likely cause: The access key ID doesn't exist (deleted or never valid), or the secret access key doesn't match it.

Fix: Recreate the destination with a fresh access key ID and secret access key pair.

s3-region-mismatch

Symptom: "The bucket's region doesn't match."

Likely cause: The region set on this destination isn't the bucket's actual AWS region.

Fix: Check the bucket's region in the AWS console (Properties tab) and set region to match exactly.

s3-endpoint-unreachable

Symptom: "Could not reach the S3/R2 endpoint."

Likely cause: A custom endpoint (used for R2 — see S3 & R2 destinations) has a typo, or there was a transient network issue reaching your cloud provider.

Fix: Double-check the endpoint value if you set one. This class of failure is often transient — retrying the run can succeed with no other change.

s3-config-invalid

Symptom: "This destination is missing required configuration."

Likely cause: bucket, region, accessKeyId, or secretAccessKey wasn't set when the destination was created.

Fix: Recreate the destination with every required field set.

s3-unknown

Symptom: Delivery failed with a provider error DataEgress doesn't specifically recognize.

Fix: The "Technical details" on the failed run include your provider's exact error — check it against AWS's S3 error code reference. If it's still unclear, contact support with the run's id.

Retries

  • Automatic: a dataset-callback failure (network error, timeout, or a 5xx response) is retried automatically up to 3 times, with a short delay between attempts, before the run is marked failed. You'll see "Retrying dataset callback…" on the run while this is happening.
  • A 4xx response (401/403/404/429/etc.) is never auto-retried — it's a problem with the request or endpoint itself that an identical retry can't fix.
  • Manual retry (the Retry button on a failed run) always re-runs the export from the start — it re-paginates the whole dataset rather than resuming mid-stream. Delivery is idempotent either way: retrying never produces a duplicate file, even if a partial upload had already succeeded.
  • Whether retrying is likely to help depends on the failure: a transient network error or timeout may well succeed on retry with no other change; a configuration problem (bad credentials, a blocked URL, an invalid response shape) will fail again identically until you fix the underlying cause — the dashboard tells you which kind you're looking at.

Runs that stop making progress

stuck-runs

Symptom: A run stayed in "running" far longer than any legitimate export should take, and was eventually force-stopped.

Likely cause: An internal hang unrelated to your fetchUrl or destination configuration — a defense-in-depth safety net catches this independently of the retry logic above.

Fix: Retry the run. If this happens repeatedly for the same dataset, contact support with the run's id.

Internal errors

internal-error

Symptom: "Something went wrong generating this export," with no more specific classification.

Fix: Retry the run — this is usually transient rather than something in your own configuration. If it keeps happening, contact support with the run's id and the technical details shown on it.