POST /task, POST /map, POST /job and POST /crawl accept an Idempotency-Key header. If the connection drops before you get the response, resend the same request with the same key: DataFuel attaches you to the task that is already running, or returns its stored result, instead of creating and charging a new one.

How to use it

Pick a key that is unique per logical request on your account, 1 to 255 characters. A stable id from your own system works well: an order number, a page id plus a date, a hash of the URL.

curl https://scraping-api.datafuel.ai/api/v1/crawl \
  --request POST \
  --header "Content-Type: application/json" \
  --header "X-API-Key: df_key_your_key_here" \
  --header "Idempotency-Key: crawl-docs-2026-09-18" \
  --data '{ "attributes": { "url": "https://example.com/docs", "max_pages": 50 } }'

Send the header on every write. There is no cost to it, and it turns every network timeout into a safe retry.

What the key is bound to

The key is bound to the request body. Sending the same key with a different type, different attributes or different proxy settings is rejected with 422 IDEMPOTENCY_KEY_REUSED. Use a new key for a new request.

Synchronous tasks that are still running

If you retry a POST /task while the original is still running, the retry waits for it and returns the same result. You never get two fetches for one key. A 202 TASK_STILL_PROCESSING answer carries the task id, so you can also poll GET /task/{id} instead of resending.

In the rare case that a create collides with an existing task or job and is not a replay of your key, the API answers 409 TASK_ALREADY_EXISTS or 409 JOB_ALREADY_EXISTS and charges nothing. Retry the request.

Retrying a failure

A stored failure replays on the same key. If a task failed because of a timeout or a proxy error and you want a fresh attempt, send the request again with a new key. See Read the result envelope for how to tell a retryable failure from a blocked page.