# Scrape (/docs/scrape)

Scrape one URL and get its document in the response.

`POST /v2/scrape` takes
`{"url": "https://example.com", "responseOptions": {"format": "html", "onlyMainContent": false}, "timeout": 10000}`;
`timeout` is 1000 to 20000 ms.

```json
{"success": true, "data": {"html": "<!doctype html>...", "metadata": {
  "sourceURL": "https://example.com", "url": "https://example.com", "statusCode": 200, "contentType": "text/html; charset=utf-8"}}}
```

- `metadata.url` is the URL after redirects, `sourceURL` the one sent. Redirects carry the cookies the site sets.
  Without request headers, redirects are followed to any site; with them, see [Request headers](/docs/request-headers).
- A page that failed is still a 200 document, with no `html` since its body is not stored.
  `metadata.error` says why and `metadata.errorCode` is one of `http_status`, `non_public_address`,
  `port_not_allowed`, `invalid_url`, `dns`, `timeout`, `tls`, `connection`, `too_many_redirects`,
  `cross_site_redirect`, `proxy_error` (see [Proxy](/docs/proxy)), `read_body`, `page_too_large`,
  `unsupported_content_type`, `storage`, `attempts_exhausted` and `request_options_unreadable` (the server could not
  decrypt the headers, e.g. after its encryption key changed). `statusCode` and `contentType` are absent when no
  response arrived.
- A page not scraped within `timeout` answers 202 with `{"success": true, "id": "...", "url": "..."}`: the scrape
  keeps running, and `url`, also sent as `Location`, is its status, `GET /v2/scrape/{id}`. An anonymous caller polls
  it too, without a key, by the scrape id.

## Status

`GET /v2/scrape/{id}` answers:

```json
{"success": true, "status": "completed", "createdAt": "...", "expiresAt": "...", "completedAt": "...",
 "data": {"html": "<!doctype html>...", "metadata": {...}}}
```

- `status` is `scraping` until the page has its result, then `completed`; `completedAt` marks it and `data` is the
  document `POST /v2/scrape` would have answered.
- A read that carries `data` counts against the scrape's 10 result reads an hour; polling before that does not.
- A scrape lives 1 day, then answers 404 `SCRAPE_NOT_FOUND`, as does an unknown id, another caller's scrape or a
  batch scrape id. Scrapes cannot be [cancelled](/docs/cancel).
- Every page is capped at 10 MiB after decoding; a larger one fails with `page_too_large`.