Scraping APIhttps://scraping-api.datafuel.ai/api/v1
Help centerOpenAPI spec

Map

Synchronous URL discovery of a site

POST/map

Map a website (synchronous)

Discovers the URLs of a site without crawling it. The API reads the sitemaps advertised by robots.txt (following sitemap indexes) and the links on the target page, then returns one deduplicated, same-site list.

A map is a regular task of type: map: the body is the POST /task body (type may be omitted), it shows up in GET /task/{task_id} and accepts the same Idempotency-Key header as POST /task.

Pricing: 1 credit per call, multiplied by the proxy plan's request multiplier, regardless of how many links come back. The charged amount is returned as credits.

Large sites are capped by limit, by the number of sitemap documents read and by a wall-clock budget; when any of these hits, truncated is true and the links collected so far are returned. To go deeper, raise limit, narrow with search, or map one child sitemap at a time via sitemap using the sitemaps list of a previous result.

Headers
Idempotency-Key
string
Request body
type
string
map
proxy_type
string
Proxy plan, matched case-insensitively. Your account default when omitted; an unknown plan answers 400 INVALID_PROXY_TYPE.
Basic Premium
proxy_country
string
proxy_city
string
proxy_state
string
proxy_asn
string
proxy_session_id DEPRECATED
string
Accepted but ignored at this level. Set attributes.proxy_session_id instead.
proxy_ttl DEPRECATED
integer
Accepted but ignored at this level. Set attributes.proxy_ttl instead.
attributes REQUIRED
object
Attributes payload for POST /task with type: map.
Request
cURL
curl https://scraping-api.datafuel.ai/api/v1/map \
  --request POST \
  --header 'Content-Type: application/json' \
  --header 'X-API-Key: df_key_your_key_here' \
  --data '{
    "attributes": {
      "url": "https://example.com"
    }
  }'
Response

Map completed

{
  "id": "123e4567-e89b-12d3-a456-426614174000",
  "status": "completed",
  "credits_used": 1,
  "result": {
    "data": {
      "url": "https://example.com/",
      "links": [
        {
          "url": "https://example.com/blog/",
          "source": "sitemap",
          "lastmod": "2026-08-01"
        },
        {
          "url": "https://example.com/blog/hello",
          "source": "page",
          "title": "Hello"
        }
      ],
      "total": 2,
      "truncated": false,
      "sitemaps": [
        "https://example.com/sitemap.xml"
      ],
      "credits": 1,
      "page_status_code": 200
    }
  }
}

Not ready yet. Not an error and not a result: poll GET /task/{id} with the id from the body, or resend the POST with the same Idempotency-Key.

{
  "id": "123e4567-e89b-12d3-a456-426614174000",
  "code": "TASK_STILL_PROCESSING",
  "message": "The task is still being processed. Please try again."
}

Malformed request or invalid attributes

{
  "code": "INVALID_ATTRIBUTES",
  "message": "Invalid attributes for selected task type"
}

Missing or invalid API key

{
  "code": "UNAUTHORIZED",
  "message": "You are not authorized to perform this action"
}

Not enough credits

{
  "code": "INSUFFICIENT_CREDITS",
  "message": "You do not have enough credits to perform this action"
}

The create collided with an existing task (TASK_ALREADY_EXISTS) or job (JOB_ALREADY_EXISTS) and is not an idempotent replay. Nothing was charged; retry the request.

{
  "code": "TASK_ALREADY_EXISTS",
  "message": "A task with this id already exists; retry the request"
}

Idempotency-Key was already used for a different request (other type, attributes or proxy)

{
  "code": "IDEMPOTENCY_KEY_REUSED",
  "message": "Idempotency-Key was already used for a different request"
}

Your concurrency limit is reached. Nothing was created or charged.

{
  "code": "CONCURRENCY_LIMIT_REACHED",
  "message": "You have reached your concurrency limit"
}

The task type or the LLM engine is switched off. Nothing was created or charged.

{
  "code": "ENGINE_UNAVAILABLE",
  "message": "This engine is switched off at the moment: temporarily unavailable"
}