Crawl a site | DataFuel Docs

Follow links from a start URL, scrape every page, and keep the cost under control with limits and path filters.

POST /crawl fetches a start URL, follows the links it finds breadth-first, and scrapes every page with the unlocker options you pass. It returns a job id at once and grows its own set of pages while it runs.

Start a crawl

Shell

curl https://scraping-api.datafuel.ai/api/v1/crawl \
  --request POST \
  --header "Content-Type: application/json" \
  --header "X-API-Key: df_key_your_key_here" \
  --header "Idempotency-Key: docs-crawl-2026-09-18" \
  --data '{
    "proxy_type": "Basic",
    "attributes": {
      "url": "https://example.com/docs",
      "max_pages": 100,
      "max_depth": 3,
      "include_paths": ["^/docs/"],
      "exclude_paths": ["\\.pdf$", "\\?page="],
      "result_format": "markdown",
      "main_content_only": true
    }
  }'

The answer is 202 Accepted:

JSON

{ "job_id": "d3f8ae31-aed3-4b39-a72f-efaa895533ff" }

Limits and filters

Attribute	Default	Notes
`max_pages`	100	Hard stop, cap 10 000. Pages are billed as they are queued, so this bounds the cost.
`max_depth`	3	Start URL is depth 0. Cap 10.
`include_paths`	none	RE2 patterns on `path?query`. When set, only matching URLs are followed. Up to 20.
`exclude_paths`	none	RE2 patterns on `path?query`. Matching URLs are never followed. Exclude wins. Up to 20.
`include_subdomains`	false	Also follow links on subdomains.
`allow_backward_links`	false	When false, only URLs under the start URL’s path are followed, so a crawl of `/docs` never wanders into `/blog`.
`concurrency`	5	Pages of this crawl in flight at once, also bounded by your account limit.

All unlocker options apply per page: result_format, extract_selector, main_content_only, headers. With js_rendering: true every page is rendered in a browser and links are taken from the rendered DOM, so client-side rendered sites crawl correctly. result_use_ai is rejected on crawl. Sitemaps, robots.txt and webhooks are not used yet.

Poll and read results

Code

GET /crawl/{job_id}                      status, page counters, stop_reason
GET /crawl/{job_id}/results?limit=100    pages + next_cursor

Poll every two seconds while status is pending or processing; it ends as completed, completed_with_errors, failed or cancelled. Results can be read while the crawl runs; pages not yet fetched appear as status stubs. Pass next_cursor back as cursor until the response has no next_cursor.

stop_reason says why the crawl ended: max_pages, max_depth_exhausted, insufficient_credits or cancelled. The same id also works with GET /job/{id}.

Cancel a crawl

Code

POST /crawl/{job_id}/cancel

No more links are followed. Pages that were queued but not fetched yet are refunded straight away; pages already being fetched finish and are charged as usual. The crawl ends with status cancelled, and the response says how many pages and credits were refunded. Cancelling a finished crawl returns 409.

Cost

Each page is a regular unlocker task charged when it is queued: 1 credit on Basic, 10 on Premium, 5 and 20 with js_rendering. Failed pages, blocked ones included, are refunded, and the links on a blocked page are never followed, so a blocked start URL is a free zero-page crawl.

total_cost on the crawl is the sum charged at queue time and does not subtract refunds. Sum credits_used over the results for the net figure.

Keep it cheap

Run Map the URLs of a site first. One credit tells you how many pages exist.
Set include_paths to the section you need and max_pages to what you can afford.
Leave js_rendering off unless the map came back empty.
If you can list the URLs, use a job instead. See Scrape a list of URLs.