Map

Synchronous URL discovery of a site

POST/map

Map a website (synchronous)

Discovers the URLs of a site without crawling it. The API reads the sitemaps advertised by robots.txt (following sitemap indexes) and the links on the target page, then returns one deduplicated, same-site list.

A map is a regular task of type: map: the body is the POST /task body (type may be omitted), it shows up in GET /task/{task_id} and accepts the same Idempotency-Key header as POST /task.

Pricing: 1 credit per call, multiplied by the proxy plan's request multiplier, regardless of how many links come back. The charged amount is returned as credits.

Large sites are capped by limit, by the number of sitemap documents read and by a wall-clock budget; when any of these hits, truncated is true and the links collected so far are returned. To go deeper, raise limit, narrow with search, or map one child sitemap at a time via sitemap using the sitemaps list of a previous result.

Headers

Idempotency-Key

string

Request body

type

string

map

proxy_type

string

Proxy plan, matched case-insensitively. Your account default when omitted; an unknown plan answers 400 INVALID_PROXY_TYPE.

BasicPremium

proxy_country

string

proxy_city

string

proxy_state

string

proxy_asn

string

proxy_session_idDEPRECATED

string

Accepted but ignored at this level. Set attributes.proxy_session_id instead.

proxy_ttlDEPRECATED

integer

Accepted but ignored at this level. Set attributes.proxy_ttl instead.

attributesREQUIRED

object

Attributes payload for POST /task with type: map.

Request

cURL

curl https://scraping-api.datafuel.ai/api/v1/map \
  --request POST \
  --header 'Content-Type: application/json' \
  --header 'X-API-Key: df_key_your_key_here' \
  --data '{
    "attributes": {
      "url": "https://example.com"
    }
  }'

Response

{
  "id": "123e4567-e89b-12d3-a456-426614174000",
  "status": "completed",
  "credits_used": 1,
  "result": {
    "data": {
      "url": "https://example.com/",
      "links": [
        {
          "url": "https://example.com/blog/",
          "source": "sitemap",
          "lastmod": "2026-08-01"
        },
        {
          "url": "https://example.com/blog/hello",
          "source": "page",
          "title": "Hello"
        }
      ],
      "total": 2,
      "truncated": false,
      "sitemaps": [
        "https://example.com/sitemap.xml"
      ],
      "credits": 1,
      "page_status_code": 200
    }
  }
}

{
  "id": "123e4567-e89b-12d3-a456-426614174000",
  "code": "TASK_STILL_PROCESSING",
  "message": "The task is still being processed. Please try again."
}

{
  "code": "INVALID_ATTRIBUTES",
  "message": "Invalid attributes for selected task type"
}

{
  "code": "UNAUTHORIZED",
  "message": "You are not authorized to perform this action"
}

{
  "code": "INSUFFICIENT_CREDITS",
  "message": "You do not have enough credits to perform this action"
}

{
  "code": "TASK_ALREADY_EXISTS",
  "message": "A task with this id already exists; retry the request"
}

{
  "code": "IDEMPOTENCY_KEY_REUSED",
  "message": "Idempotency-Key was already used for a different request"
}

{
  "code": "CONCURRENCY_LIMIT_REACHED",
  "message": "You have reached your concurrency limit"
}

{
  "code": "ENGINE_UNAVAILABLE",
  "message": "This engine is switched off at the moment: temporarily unavailable"
}