POST /v2/batch/scrape takes the URLs and the options every document of the batch scrape gets:

json
{"urls": ["https://example.com", "not a url"], "responseOptions": {"format": "html", "onlyMainContent": false},
 "ignoreInvalidURLs": true,
 "webhook": {"url": "https://receiver.example/hook", "headers": {"X-Token": "t"}, "metadata": {"run": "7"},
             "events": ["start", "finish"], "secret": "at-least-16-characters"}}

and answers:

json
{"success": true, "id": "<id>", "url": "https://api.elisebot.dev/v2/batch/scrape/<id>", "invalidURLs": ["not a url"]}
  • A URL without a scheme, such as example.com, is taken as https.
  • Invalid URLs are dropped and listed; with ignoreInvalidURLs: false they are a 422, and so is a batch scrape with no valid URL left.
  • The POST always answers at once, whatever the URL count; the documents come in one package once the batch scrape ends.
  • webhook is described under Webhooks.

Status

GET /v2/batch/scrape/{id}, with the organization’s API key, answers:

json
{"success": true, "status": "completed", "total": 3, "completed": 3, "createdAt": "...", "expiresAt": "...",
 "completedAt": "...", "package": {"status": "ready", "bytes": 4120,
 "url": "https://api.elisebot.dev/v2/batch/scrape/<id>/download?token=<token>"}}
  • status is scraping, completed or cancelled. completed counts the URLs with a result, failed and cancelled ones included, so a finished batch scrape has completed == total; completedAt marks it.
  • package.status is pending until the batch scrape ends and its results are packaged, then ready, with url and its size in bytes. It is failed when packaging gave up; the batch scrape’s credits are then refunded.
  • webhookStatus and webhookError give each chosen webhook event’s state (pending, delivered, failed) and failure.
  • A batch scrape and its content live 1 day, then answer 404 BATCH_SCRAPE_NOT_FOUND, as does another organization’s batch scrape or a scrape id.

Download

GET on package.url answers 302 to a presigned URL, valid for 5 minutes, that serves the file as <id>.ndjson.gz. The token in the URL is the capability, so no API key is needed:

sh
curl -L -o results.ndjson.gz '<package.url>'
gunzip -c results.ndjson.gz | jq -r .metadata.url
  • The file is gzipped NDJSON: one document per line, in URL order, failed and cancelled URLs included. Each line has the shape POST /v2/scrape returns as data.
  • A package that is not ready yet is a 409 PACKAGE_NOT_READY, and a failed one a 409 PACKAGE_FAILED. An unknown batch scrape or a wrong token is a 404 BATCH_SCRAPE_NOT_FOUND.
  • Each batch scrape has 10 result reads an hour, past which the answer is 429 with Retry-After. Every download counts, a 409 included; status reads do not.