Documentation

REST API and MCP server reference

Authentication

Generate an API key from the Settings page. Pass it as a bearer token in the Authorization header:

Authorization: Bearer YOUR_API_KEY

REST API

`GET /v1/health`

Returns service status. No authentication required.

curl https://small-giraffe-45.convex.site/v1/health

`POST /v1/scrape`

Scrape one or more URLs. Use sync mode for a single URL (returns results inline) or async mode for batch jobs (up to 5 URLs).

Request body

Field	Type	Description
`urls`	string[]	URLs to scrape (1–5). Sync mode requires exactly 1.
`mode`	string	`sync` (default) or `async`
`format`	string	`markdown` (default), `html`, or `both`
`config`	object	Optional config overrides (see below)
`prompt`	string	Optional LLM extraction prompt (max 10,000 chars). Response returned in the `extract` field.

Config options

Option	Default	Description
`timeoutMs`	30000	Request timeout (1–120,000 ms)
`maxConcurrency`	3	Parallel fetches per batch (1–5)
`cacheTtlSeconds`	3600	Cache TTL (60–3,600 seconds)
`userAgentProfile`	default	UA profile: `default`, `mobile`, or `tablet`
`jitterMs`	{ min: 500, max: 2000 }	Random delay range between requests (0–60,000 ms)

Sync example

curl -X POST https://small-giraffe-45.convex.site/v1/scrape \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "urls": ["https://example.com"],
    "format": "markdown"
  }'

Sync response

{
  "requestId": "abc123",
  "mode": "sync",
  "result": {
    "url": "https://example.com",
    "status": "success",
    "cached": false,
    "markdown": "# Example Domain\n\nThis domain is for use in illustrative examples...",
    "metadata": {
      "requestedUrl": "https://example.com",
      "finalUrl": "https://example.com/",
      "title": "Example Domain",
      "httpStatus": 200,
      "fetchDurationMs": 342,
      "fetchedAt": 1710800000000,
      "expiresAt": 1710803600000
    }
  }
}

Async example (batch)

curl -X POST https://small-giraffe-45.convex.site/v1/scrape \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "urls": ["https://example.com", "https://example.org"],
    "mode": "async"
  }'

# Response:
# { "requestId": "abc123", "mode": "async", "jobId": "job_456" }

LLM extraction example

curl -X POST https://small-giraffe-45.convex.site/v1/scrape \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "urls": ["https://example.com/pricing"],
    "prompt": "Extract all pricing tiers as JSON with name, price, and features[]"
  }'

`GET /v1/jobs/:jobId`

Poll the status of an async job.

curl https://small-giraffe-45.convex.site/v1/jobs/job_456 \
  -H "Authorization: Bearer YOUR_API_KEY"

# Response:
# {
#   "jobId": "job_456",
#   "status": "completed",
#   "totalCount": 2,
#   "completedCount": 2,
#   "successCount": 2,
#   "failureCount": 0,
#   "expiresAt": 1710803600000
# }

Job statuses: queued, running, completed, failed, partial.

`GET /v1/jobs/:jobId/results`

Fetch results for a completed (or partial) async job.

curl https://small-giraffe-45.convex.site/v1/jobs/job_456/results \
  -H "Authorization: Bearer YOUR_API_KEY"

Error codes

When a scrape fails, the error field contains a code and message:

Code	Description
`INVALID_URL`	URL is malformed
`UNSUPPORTED_SCHEME`	Only http/https URLs are supported
`PRIVATE_NETWORK_TARGET`	URL resolves to a private/internal IP
`TIMEOUT`	Request exceeded timeout
`FETCH_FAILED`	Network error or non-2xx response
`NON_HTML_RESPONSE`	Response content-type is not HTML
`TOO_MANY_URLS`	Exceeded max 5 URLs per request
`UNAUTHORIZED`	Missing or invalid API key
`USER_DISABLED`	Account is disabled
`RATE_LIMITED`	Too many requests
`NOT_FOUND`	Job or resource not found
`INTERNAL_ERROR`	Unexpected server error

MCP Server

scrape_ exposes an MCP server so AI agents can scrape pages directly. Authentication uses OAuth 2.0 — MCP clients handle the flow automatically.

`scrape_url` tool

Parameter	Required	Description
`url`	Yes	URL to scrape
`format`	No	`markdown` (default), `html`, or `both`
`timeout_ms`	No	Timeout in ms (default 30000, max 120000)
`user_agent_profile`	No	`default`, `mobile`, or `tablet`
`prompt`	No	LLM extraction prompt — response returned as JSON in the extract field

Setup

Add the following to your MCP client configuration. Replace the URL with your Convex site URL.

Claude Desktop / Claude Code

{
  "mcpServers": {
    "scrape": {
      "type": "http",
      "url": "https://small-giraffe-45.convex.site/mcp"
    }
  }
}

Cursor

{
  "mcpServers": {
    "scrape": {
      "url": "https://small-giraffe-45.convex.site/mcp"
    }
  }
}

On first use, your MCP client will open a browser window to authenticate via OAuth. After granting access, the token is cached and subsequent requests are automatic.

Questions?

Reach out at support@zaks.io.