POST/v2/batch/scrape
Requires an API key.
Parameters
Idempotency-Keyheader, stringA unique key, such as a UUID, to safely retry the request. Valid for 24 hours.
at most 255 characters, pattern ^[\x21-\x7e]+$
Request body
application/json, required, type BatchScrapeRequest
ignoreInvalidURLsbooleanSkip invalid URLs instead of rejecting the request.
Default
truerequestOptionsRequestOptionsOptions for the requests sent to the URLs.
1 field inside requestOptions
headersarray of RequestHeader or nullHeaders to send, in this order and exactly as written. Omit to use the defaults. Redirect hops also carry the cookies the site sets. With headers, redirects to another site are not followed.
2 fields inside headers
namestringrequiredThe header name.
at least 1 characters
valuestringrequiredThe header value.
responseOptionsResponseOptionsOptions for the content returned for each URL.
2 fields inside responseOptions
formatstringThe format to return the page in.
One of
html,markdownDefault
htmlonlyMainContentbooleanReturn only the page's main content, without navigation, headers and footers.
Default
false
urlsarray of string or nullrequiredThe URLs to scrape.
at least 1 items
webhookWebhookGet notified as the batch scrape progresses.
5 fields inside webhook
eventsarray of string or nullEvents to send. Defaults to all.
One of
start,finishat least 1 items
headersmap of stringHeaders to include with each event.
metadatamap of stringCustom data included with each event.
secretstringA secret to sign each event with, sent in the X-Webhook-Signature header.
urlstringrequiredThe endpoint that receives events. Failed deliveries are retried.
at least 1 characters, at most 2048 characters
Responses
OK
Headers
Idempotent-Replayedstringtrue when the response replays an earlier request with the same key.
One of
true
Type BatchScrapeResponse
idstringrequiredThe batch scrape ID.
invalidURLsarray of stringrequiredURLs skipped as invalid.
successbooleanrequiredAlways true.
One of
trueurlstringrequiredThe URL to check the batch scrape status.
MALFORMED_REQUEST: The request body is missing or is not valid JSON.
Type MalformedRequestResponse
codestringrequiredA machine-readable error code.
One of
MALFORMED_REQUESTdetailsarray of ValidationErrorDetail or nullThe invalid fields.
3 fields inside details
locationstringWhere the error occurred, e.g. body.urls[3].
messagestringWhat went wrong.
valueanyThe rejected value.
errorstringrequiredA human-readable message.
successbooleanrequiredAlways false.
One of
false
UNAUTHORIZED: The API key is invalid or revoked; API_KEY_REQUIRED: This endpoint needs an API key; INVALID_ACCESS_TOKEN: The OAuth access token is unknown, expired or revoked, or was sent to an endpoint other than /mcp, the only one it works on.
Type UnauthorizedResponse
codestringrequiredA machine-readable error code.
One of
UNAUTHORIZED,API_KEY_REQUIRED,INVALID_ACCESS_TOKENdetailsarray of ValidationErrorDetail or nullThe invalid fields.
3 fields inside details
locationstringWhere the error occurred, e.g. body.urls[3].
messagestringWhat went wrong.
valueanyThe rejected value.
errorstringrequiredA human-readable message.
successbooleanrequiredAlways false.
One of
false
INSUFFICIENT_CREDITS: The credit balance does not cover the request's URLs.
Type PaymentRequiredResponse
codestringrequiredA machine-readable error code.
One of
INSUFFICIENT_CREDITSdetailsarray of ValidationErrorDetail or nullThe invalid fields.
3 fields inside details
locationstringWhere the error occurred, e.g. body.urls[3].
messagestringWhat went wrong.
valueanyThe rejected value.
errorstringrequiredA human-readable message.
successbooleanrequiredAlways false.
One of
false
REQUEST_TIMEOUT: The request body took too long to arrive.
Type RequestTimeoutResponse
codestringrequiredA machine-readable error code.
One of
REQUEST_TIMEOUTdetailsarray of ValidationErrorDetail or nullThe invalid fields.
3 fields inside details
locationstringWhere the error occurred, e.g. body.urls[3].
messagestringWhat went wrong.
valueanyThe rejected value.
errorstringrequiredA human-readable message.
successbooleanrequiredAlways false.
One of
false
PAYLOAD_TOO_LARGE: The request body is too large.
Type PayloadTooLargeResponse
codestringrequiredA machine-readable error code.
One of
PAYLOAD_TOO_LARGEdetailsarray of ValidationErrorDetail or nullThe invalid fields.
3 fields inside details
locationstringWhere the error occurred, e.g. body.urls[3].
messagestringWhat went wrong.
valueanyThe rejected value.
errorstringrequiredA human-readable message.
successbooleanrequiredAlways false.
One of
false
UNSUPPORTED_MEDIA_TYPE: The request body must be JSON.
Type UnsupportedMediaTypeResponse
codestringrequiredA machine-readable error code.
One of
UNSUPPORTED_MEDIA_TYPEdetailsarray of ValidationErrorDetail or nullThe invalid fields.
3 fields inside details
locationstringWhere the error occurred, e.g. body.urls[3].
messagestringWhat went wrong.
valueanyThe rejected value.
errorstringrequiredA human-readable message.
successbooleanrequiredAlways false.
One of
false
VALIDATION_FAILED: One or more fields are invalid; TOO_MANY_URLS: The request has more URLs than your rate limit allows; IDEMPOTENCY_CONFLICT: The Idempotency-Key was already used with a different request.
Type ValidationFailedResponse
codestringrequiredA machine-readable error code.
One of
VALIDATION_FAILED,TOO_MANY_URLS,IDEMPOTENCY_CONFLICTdetailsarray of ValidationErrorDetail or nullThe invalid fields.
3 fields inside details
locationstringWhere the error occurred, e.g. body.urls[3].
messagestringWhat went wrong.
valueanyThe rejected value.
errorstringrequiredA human-readable message.
successbooleanrequiredAlways false.
One of
false
RATE_LIMITED: Rate limit exceeded, retry after Retry-After seconds.
Headers
Retry-AfterintegerSeconds to wait before retrying.
minimum 1
Type TooManyRequestsResponse
codestringrequiredA machine-readable error code.
One of
RATE_LIMITEDdetailsarray of ValidationErrorDetail or nullThe invalid fields.
3 fields inside details
locationstringWhere the error occurred, e.g. body.urls[3].
messagestringWhat went wrong.
valueanyThe rejected value.
errorstringrequiredA human-readable message.
successbooleanrequiredAlways false.
One of
false
INTERNAL_ERROR: Something went wrong on our side, try again.
Type InternalErrorResponse
codestringrequiredA machine-readable error code.
One of
INTERNAL_ERRORdetailsarray of ValidationErrorDetail or nullThe invalid fields.
3 fields inside details
locationstringWhere the error occurred, e.g. body.urls[3].
messagestringWhat went wrong.
valueanyThe rejected value.
errorstringrequiredA human-readable message.
successbooleanrequiredAlways false.
One of
false