POST/v2/scrape
An API key is optional; without one the caller is anonymous.
Request body
application/json, required, type ScrapeRequest
requestOptionsRequestOptionsOptions for the request sent to the URL.
1 field inside requestOptions
headersarray of RequestHeader or nullHeaders to send, in this order and exactly as written. Omit to use the defaults. Redirect hops also carry the cookies the site sets. With headers, redirects to another site are not followed.
2 fields inside headers
namestringrequiredThe header name.
at least 1 characters
valuestringrequiredThe header value.
responseOptionsResponseOptionsOptions for the content returned for the URL.
2 fields inside responseOptions
formatstringThe format to return the page in.
One of
html,markdownDefault
htmlonlyMainContentbooleanReturn only the page's main content, without navigation, headers and footers.
Default
false
timeoutinteger (int64)Maximum time to wait, in milliseconds.
Default
10000minimum 1000, maximum 20000
urlstringrequiredThe URL to scrape.
at least 1 characters
Responses
OK
Type ScrapeResponse
dataDocumentrequired3 fields inside data
htmlstringThe page HTML, converted to UTF-8; only its main content with responseOptions.onlyMainContent.
markdownstringThe page as Markdown.
metadataDocumentMetadatarequired6 fields inside metadata
contentTypestringThe response content type.
errorstringWhy the scrape failed.
errorCodestringA machine-readable error code.
sourceURLstringrequiredThe requested URL.
statusCodeinteger (int64)The HTTP status code.
urlstringrequiredThe final URL after redirects.
successbooleanrequiredAlways true.
One of
true
The page was not scraped within timeout. The scrape keeps running: poll url, also sent as Location, for the document.
Headers
LocationstringThe scrape status URL, the same as url.
Type ScrapeAcceptedResponse
idstringrequiredThe scrape ID.
successbooleanrequiredAlways true.
One of
trueurlstringrequiredThe URL to check the scrape status.
MALFORMED_REQUEST: The request body is missing or is not valid JSON.
Type MalformedRequestResponse
codestringrequiredA machine-readable error code.
One of
MALFORMED_REQUESTdetailsarray of ValidationErrorDetail or nullThe invalid fields.
3 fields inside details
locationstringWhere the error occurred, e.g. body.urls[3].
messagestringWhat went wrong.
valueanyThe rejected value.
errorstringrequiredA human-readable message.
successbooleanrequiredAlways false.
One of
false
UNAUTHORIZED: The API key is invalid or revoked; API_KEY_REQUIRED: This endpoint needs an API key; INVALID_ACCESS_TOKEN: The OAuth access token is unknown, expired or revoked, or was sent to an endpoint other than /mcp, the only one it works on.
Type UnauthorizedResponse
codestringrequiredA machine-readable error code.
One of
UNAUTHORIZED,API_KEY_REQUIRED,INVALID_ACCESS_TOKENdetailsarray of ValidationErrorDetail or nullThe invalid fields.
3 fields inside details
locationstringWhere the error occurred, e.g. body.urls[3].
messagestringWhat went wrong.
valueanyThe rejected value.
errorstringrequiredA human-readable message.
successbooleanrequiredAlways false.
One of
false
INSUFFICIENT_CREDITS: The credit balance does not cover the request's URLs.
Type PaymentRequiredResponse
codestringrequiredA machine-readable error code.
One of
INSUFFICIENT_CREDITSdetailsarray of ValidationErrorDetail or nullThe invalid fields.
3 fields inside details
locationstringWhere the error occurred, e.g. body.urls[3].
messagestringWhat went wrong.
valueanyThe rejected value.
errorstringrequiredA human-readable message.
successbooleanrequiredAlways false.
One of
false
REQUEST_TIMEOUT: The request body took too long to arrive.
Type RequestTimeoutResponse
codestringrequiredA machine-readable error code.
One of
REQUEST_TIMEOUTdetailsarray of ValidationErrorDetail or nullThe invalid fields.
3 fields inside details
locationstringWhere the error occurred, e.g. body.urls[3].
messagestringWhat went wrong.
valueanyThe rejected value.
errorstringrequiredA human-readable message.
successbooleanrequiredAlways false.
One of
false
PAYLOAD_TOO_LARGE: The request body is too large.
Type PayloadTooLargeResponse
codestringrequiredA machine-readable error code.
One of
PAYLOAD_TOO_LARGEdetailsarray of ValidationErrorDetail or nullThe invalid fields.
3 fields inside details
locationstringWhere the error occurred, e.g. body.urls[3].
messagestringWhat went wrong.
valueanyThe rejected value.
errorstringrequiredA human-readable message.
successbooleanrequiredAlways false.
One of
false
UNSUPPORTED_MEDIA_TYPE: The request body must be JSON.
Type UnsupportedMediaTypeResponse
codestringrequiredA machine-readable error code.
One of
UNSUPPORTED_MEDIA_TYPEdetailsarray of ValidationErrorDetail or nullThe invalid fields.
3 fields inside details
locationstringWhere the error occurred, e.g. body.urls[3].
messagestringWhat went wrong.
valueanyThe rejected value.
errorstringrequiredA human-readable message.
successbooleanrequiredAlways false.
One of
false
VALIDATION_FAILED: One or more fields are invalid; TOO_MANY_URLS: The request has more URLs than your rate limit allows; IDEMPOTENCY_CONFLICT: The Idempotency-Key was already used with a different request.
Type ValidationFailedResponse
codestringrequiredA machine-readable error code.
One of
VALIDATION_FAILED,TOO_MANY_URLS,IDEMPOTENCY_CONFLICTdetailsarray of ValidationErrorDetail or nullThe invalid fields.
3 fields inside details
locationstringWhere the error occurred, e.g. body.urls[3].
messagestringWhat went wrong.
valueanyThe rejected value.
errorstringrequiredA human-readable message.
successbooleanrequiredAlways false.
One of
false
RATE_LIMITED: Rate limit exceeded, retry after Retry-After seconds.
Headers
Retry-AfterintegerSeconds to wait before retrying.
minimum 1
Type TooManyRequestsResponse
codestringrequiredA machine-readable error code.
One of
RATE_LIMITEDdetailsarray of ValidationErrorDetail or nullThe invalid fields.
3 fields inside details
locationstringWhere the error occurred, e.g. body.urls[3].
messagestringWhat went wrong.
valueanyThe rejected value.
errorstringrequiredA human-readable message.
successbooleanrequiredAlways false.
One of
false
INTERNAL_ERROR: Something went wrong on our side, try again.
Type InternalErrorResponse
codestringrequiredA machine-readable error code.
One of
INTERNAL_ERRORdetailsarray of ValidationErrorDetail or nullThe invalid fields.
3 fields inside details
locationstringWhere the error occurred, e.g. body.urls[3].
messagestringWhat went wrong.
valueanyThe rejected value.
errorstringrequiredA human-readable message.
successbooleanrequiredAlways false.
One of
false