# Batch scrape (/docs/batch-scrape)

Scrape many URLs in one request and download their documents as one file.

`POST /v2/batch/scrape` takes the URLs and the options every document of the batch scrape gets:

```json
{"urls": ["https://example.com", "not a url"], "responseOptions": {"format": "html", "onlyMainContent": false},
 "ignoreInvalidURLs": true,
 "webhook": {"url": "https://receiver.example/hook", "headers": {"X-Token": "t"}, "metadata": {"run": "7"},
             "events": ["start", "finish"], "secret": "at-least-16-characters"}}
```

and answers:

```json
{"success": true, "id": "<id>", "url": "https://api.elisebot.dev/v2/batch/scrape/<id>", "invalidURLs": ["not a url"]}
```

- A URL without a scheme, such as `example.com`, is taken as https.
- Invalid URLs are dropped and listed; with `ignoreInvalidURLs: false` they are a 422, and so is a batch scrape with
  no valid URL left.
- The POST always answers at once, whatever the URL count; the documents come in one package once the batch scrape
  ends.
- `webhook` is described under [Webhooks](/docs/webhooks).

## Status

`GET /v2/batch/scrape/{id}`, with the organization's API key, answers:

```json
{"success": true, "status": "completed", "total": 3, "completed": 3, "createdAt": "...", "expiresAt": "...",
 "completedAt": "...", "package": {"status": "ready", "bytes": 4120,
 "url": "https://api.elisebot.dev/v2/batch/scrape/<id>/download?token=<token>"}}
```

- `status` is `scraping`, `completed` or `cancelled`. `completed` counts the URLs with a result, failed and cancelled
  ones included, so a finished batch scrape has `completed == total`; `completedAt` marks it.
- `package.status` is `pending` until the batch scrape ends and its results are packaged, then `ready`, with `url` and
  its size in `bytes`. It is `failed` when packaging gave up; the batch scrape's credits are then refunded.
- `webhookStatus` and `webhookError` give each chosen webhook event's state (`pending`, `delivered`, `failed`) and
  failure.
- A batch scrape and its content live 1 day, then answer 404 `BATCH_SCRAPE_NOT_FOUND`, as does another organization's
  batch scrape or a [scrape](/docs/scrape#status) id.

## Download

`GET` on `package.url` answers 302 to a presigned URL, valid for 5 minutes, that serves the file as
`<id>.ndjson.gz`. The token in the URL is the capability, so no API key is needed:

```sh
curl -L -o results.ndjson.gz '<package.url>'
gunzip -c results.ndjson.gz | jq -r .metadata.url
```

- The file is gzipped NDJSON: one document per line, in URL order, failed and cancelled URLs included. Each line has
  the shape [`POST /v2/scrape`](/docs/scrape) returns as `data`.
- A package that is not ready yet is a 409 `PACKAGE_NOT_READY`, and a failed one a 409 `PACKAGE_FAILED`. An unknown
  batch scrape or a wrong token is a 404 `BATCH_SCRAPE_NOT_FOUND`.
- Each batch scrape has 10 result reads an hour, past which the answer is 429 with `Retry-After`. Every download
  counts, a 409 included; status reads do not.