POST /v2/scrape takes {"url": "https://example.com", "responseOptions": {"format": "html", "onlyMainContent": false}, "timeout": 10000}; timeout is 1000 to 20000 ms.

json
{"success": true, "data": {"html": "<!doctype html>...", "metadata": {
  "sourceURL": "https://example.com", "url": "https://example.com", "statusCode": 200, "contentType": "text/html; charset=utf-8"}}}
  • metadata.url is the URL after redirects, sourceURL the one sent. Redirects carry the cookies the site sets. Without request headers, redirects are followed to any site; with them, see Request headers.
  • A page that failed is still a 200 document, with no html since its body is not stored. metadata.error says why and metadata.errorCode is one of http_status, non_public_address, port_not_allowed, invalid_url, dns, timeout, tls, connection, too_many_redirects, cross_site_redirect, proxy_error (see Proxy), read_body, page_too_large, unsupported_content_type, storage, attempts_exhausted and request_options_unreadable (the server could not decrypt the headers, e.g. after its encryption key changed). statusCode and contentType are absent when no response arrived.
  • A page not scraped within timeout answers 202 with {"success": true, "id": "...", "url": "..."}: the scrape keeps running, and url, also sent as Location, is its status, GET /v2/scrape/{id}. An anonymous caller polls it too, without a key, by the scrape id.

Status

GET /v2/scrape/{id} answers:

json
{"success": true, "status": "completed", "createdAt": "...", "expiresAt": "...", "completedAt": "...",
 "data": {"html": "<!doctype html>...", "metadata": {...}}}
  • status is scraping until the page has its result, then completed; completedAt marks it and data is the document POST /v2/scrape would have answered.
  • A read that carries data counts against the scrape’s 10 result reads an hour; polling before that does not.
  • A scrape lives 1 day, then answers 404 SCRAPE_NOT_FOUND, as does an unknown id, another caller’s scrape or a batch scrape id. Scrapes cannot be cancelled.
  • Every page is capped at 10 MiB after decoding; a larger one fails with page_too_large.