• A batch scrape’s webhook is an object whose url must be public, on port 80 or 443.
  • events picks from start, sent when the batch scrape is accepted, and finish; both by default. start always arrives first.
  • Each event POSTs {"success": true, "type": "batch_scrape.start", "id": "<id>", "metadata": {...}}, with the given metadata and headers. No event carries content.
  • finish is sent once the batch scrape’s results are packaged, and carries the package’s downloadUrl, the status package.url. When packaging failed, finish is still sent, without downloadUrl.
    • Host, Content-Type, Content-Length, Transfer-Encoding, Connection, User-Agent and X-Webhook-Signature cannot be set as headers.
  • Delivery is at-least-once, retried for about 21 hours, and redirects are not followed. Receivers dedupe on id and type.
  • With a secret (16 to 256 printable ASCII characters), each event carries X-Webhook-Signature: sha256=<hex HMAC-SHA256 of the raw body, keyed by the secret>. Receivers compute it over the raw bytes before parsing and compare in constant time.
  • The secret and the headers are stored encrypted (AES-256-GCM under the server’s encryption key), and so is requestOptions; Postgres and its backups never hold them in plaintext. metadata stays plain, since every event echoes it.