Map
Synchronous URL discovery of a site
/map Map a website (synchronous)
POST /mapDiscovers the URLs of a site without crawling it. The API reads the sitemaps
advertised by robots.txt (following sitemap indexes) and the links on the
target page, then returns one deduplicated, same-site list.
A map is a regular task of type: map: the body is the POST /task body
(type may be omitted), it shows up in GET /task/{task_id} and accepts the
same Idempotency-Key header as POST /task.
Pricing: 1 credit per call, multiplied by the proxy plan's request multiplier,
regardless of how many links come back. The charged amount is returned as credits.
Large sites are capped by limit, by the number of sitemap documents read and by a
wall-clock budget; when any of these hits, truncated is true and the links
collected so far are returned. To go deeper, raise limit, narrow with search,
or map one child sitemap at a time via sitemap using the sitemaps list of a
previous result.
Idempotency-Keystringtypestringproxy_typestring400 INVALID_PROXY_TYPE.proxy_countrystringproxy_citystringproxy_statestringproxy_asnstringproxy_session_id DEPRECATEDstringattributes.proxy_session_id instead.proxy_ttl DEPRECATEDintegerattributes.proxy_ttl instead.attributes REQUIREDobjectPOST /task with type: map.curl https://scraping-api.datafuel.ai/api/v1/map \ --request POST \ --header 'Content-Type: application/json' \ --header 'X-API-Key: df_key_your_key_here' \ --data '{ "attributes": { "url": "https://example.com" } }'
Map completed
{ "id": "123e4567-e89b-12d3-a456-426614174000", "status": "completed", "credits_used": 1, "result": { "data": { "url": "https://example.com/", "links": [ { "url": "https://example.com/blog/", "source": "sitemap", "lastmod": "2026-08-01" }, { "url": "https://example.com/blog/hello", "source": "page", "title": "Hello" } ], "total": 2, "truncated": false, "sitemaps": [ "https://example.com/sitemap.xml" ], "credits": 1, "page_status_code": 200 } } }
Not ready yet. Not an error and not a result: poll GET /task/{id} with the id from
the body, or resend the POST with the same Idempotency-Key.
{ "id": "123e4567-e89b-12d3-a456-426614174000", "code": "TASK_STILL_PROCESSING", "message": "The task is still being processed. Please try again." }
Malformed request or invalid attributes
{ "code": "INVALID_ATTRIBUTES", "message": "Invalid attributes for selected task type" }
Missing or invalid API key
{ "code": "UNAUTHORIZED", "message": "You are not authorized to perform this action" }
Not enough credits
{ "code": "INSUFFICIENT_CREDITS", "message": "You do not have enough credits to perform this action" }
The create collided with an existing task (TASK_ALREADY_EXISTS) or job (JOB_ALREADY_EXISTS)
and is not an idempotent replay. Nothing was charged; retry the request.
{ "code": "TASK_ALREADY_EXISTS", "message": "A task with this id already exists; retry the request" }
Idempotency-Key was already used for a different request (other type, attributes or proxy)
{ "code": "IDEMPOTENCY_KEY_REUSED", "message": "Idempotency-Key was already used for a different request" }
Your concurrency limit is reached. Nothing was created or charged.
{ "code": "CONCURRENCY_LIMIT_REACHED", "message": "You have reached your concurrency limit" }
The task type or the LLM engine is switched off. Nothing was created or charged.
{ "code": "ENGINE_UNAVAILABLE", "message": "This engine is switched off at the moment: temporarily unavailable" }