batch_scrape

Fetch many URLs in one call with a shared worker pool. The request is synchronous — it returns the finished results, not a job handle — and every URL comes back with its own status, so one dead page never sinks the batch. Results are also stored for 24 hours and can be re-read with get_batch_results.

Use Cases

Competitor Price Sweeps

Pull the same CSS field off fifty product pages in one request, instead of fifty round trips.

Link Health Checks

Submit a URL list and read http_status per entry. Anything unreachable comes back as failed with the reason attached.

Content Inventories

Feed a sitemap into the batch and collect every title and canonical URL for an audit spreadsheet.

Post-Deploy Verification

Check that a set of critical pages still returns 200 and still contains the element you expect.

Dataset Assembly

Collect the first 5,000 characters of body text from a URL list as the input to a downstream pipeline.

Deferred Collection

Run the batch now and pull the stored results later from a different process with get_batch_results.

Endpoint

POST/api/v1/tools/batch_scrape

Auth Required

1 req/s on Free plan

5 credits

Parameters

The hosted REST API accepts the full MCP parameter set so the same request body works on both surfaces, but only urls, batch_config.concurrency and extraction_template.fields change what it does. Everything else is validated and then ignored; each one says so in the table below.

Name	Type	Required	Default	Description
urls	array	Required	-	The URLs to fetch. Between 1 and 50 entries; a longer list is rejected rather than truncated. Example: [{ "url": "https://example.com/widget-a", "id": "a" }]
batch_config	object	Optional	-	Execution settings for the batch. Example: { "concurrency": 5 }
extraction_template	object	Optional	-	Fields to pull out of every page in the batch. Example: { "fields": [{ "name": "price", "selector": ".price" }] }
output_config	object	Optional	-	Accepted and ignored in full. Responses are always JSON in the shape shown below; `format`, `include_metadata`, `include_errors` and `flatten_results` have no effect. Example: { "format": "json" }
options	object	Optional	-	Accepted and ignored, except that `javascript_enabled: true` adds an explanatory line to `notes`. `user_agent`, `follow_redirects`, `respect_robots_txt` and `rate_limit_per_domain` do not change fetching on the hosted REST API. Example: { "javascript_enabled": false }
respect_robots	boolean	Optional	true	Respect each target site's robots.txt. Left at `true`, a URL disallowed for `CrawlForge` is skipped and the rest of the batch runs on — the request still returns 200 with a partial result set rather than a 403, and a skipped URL is not charged. Set it to `false` only for targets you have your own agreement with — the response then carries a `warnings` entry and the override is recorded against your API key. Example: true

Execution Model

Worth knowing before you size a batch — this endpoint runs inside a 30-second serverless function.

Synchronous

One request in, finished results out. There is no job id to poll and no webhook — batch_id is a retrieval key, not a job handle.

20s fetch budget

URLs not started before the budget runs out come back with a skipped status and are not charged. Individual fetches time out at 8s.

Failures are per URL

A 404, a timeout or an unreadable body marks that one entry failed. The call itself still returns 200.

No JavaScript

Pages are fetched, not rendered. For client-rendered pages use scrape_with_actions or the CrawlForge MCP server.

Request Examples

terminalBash

curl -X POST https://crawlforge.dev/api/v1/tools/batch_scrape \
  -H "X-API-Key: cf_test_YOUR_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "urls": [
      { "url": "https://example.com/widget-a", "id": "a" },
      { "url": "https://example.com/widget-b", "id": "b" }
    ],
    "batch_config": { "concurrency": 5 },
    "extraction_template": {
      "fields": [
        { "name": "price", "selector": ".price" },
        { "name": "canonical", "selector": "link[rel=canonical]", "attribute": "href" }
      ]
    }
  }'

Response Example

200 OK3,140ms

{
  "success": true,
  "data": {
    "batch_id": "fd599a27-9ca9-4022-9a57-b02af57c387b",
    "total": 3,
    "succeeded": 2,
    "failed": 1,
    "skipped": 0,
    "results": [
      {
        "id": "a",
        "url": "https://example.com/widget-a",
        "status": "success",
        "http_status": 200,
        "title": "Widget A — Acme",
        "text": "Widget A$24.00In stock and ready to ship.",
        "fields": {
          "price": "$24.00",
          "canonical": "https://example.com/widget-a",
          "sku": null
        },
        "field_notes": [
          "sku: xpath selectors are not supported on the hosted REST API — use a CSS selector"
        ]
      },
      {
        "id": "b",
        "url": "https://example.com/widget-b",
        "status": "success",
        "http_status": 200,
        "title": "Widget B — Acme",
        "text": "Widget B$31.50Backordered until March.",
        "fields": {
          "price": "$31.50",
          "canonical": "https://example.com/widget-b",
          "sku": null
        }
      },
      {
        "id": "c",
        "url": "https://example.com/gone",
        "status": "failed",
        "http_status": 404,
        "error": "HTTP 404"
      }
    ],
    "notes": [
      "Results are stored for 24h and retrievable with get_batch_results (batch_id: fd599a27-9ca9-4022-9a57-b02af57c387b)."
    ],
    "completed_at": "2026-08-27T02:09:44.001Z"
  },
  "credits_used": 15,
  "credits_remaining": 990,
  "processing_time": 3140
}

Field Descriptions

data.batch_idPass this to [get_batch_results](/docs/api-reference/tools/get-batch-results) within 24 hours to re-read the full set

data.totalEntries submitted — always equal to the length of `results`

data.skippedURLs the time budget never reached. Not charged

data.results[].idYour `id` if you supplied one, otherwise the array index as a string

data.results[].status`success`, `failed` or `skipped` — check this per entry, not the HTTP status

data.results[].textVisible body text with scripts and styles removed, cut at 5,000 characters

data.results[].fieldsOne key per template field. `null` means the selector matched nothing

data.results[].field_notesPresent only when a field could not be applied — an xpath selector, or CSS the parser rejected

data.notesBatch-level remarks, including anything you asked for that the hosted API does not do

credits_used5 per URL attempted — 3 URLs here, none skipped, so 15

Error Handling

Invalid Batch (400 Bad Request)

VALIDATION_ERROR. Raised by an empty urls array, more than 50 entries, a malformed URL, or a concurrency outside 1-10. Nothing is fetched and nothing is charged.

Storage Unavailable (503 Service Unavailable)

STORAGE_UNAVAILABLE. Results are persisted before billing, so if that store cannot be reached the batch is refused rather than run and lost. Nothing is charged. Retry.

Batch Failed (500 Internal Server Error)

TOOL_ERROR. An unexpected fault. Per-URL problems never surface here — they appear as failed entries in a 200 response.

Disallowed by robots.txt (no error — the URL is skipped)

A URL that robots.txt disallows for CrawlForge is left out of the batch rather than failing it: the call still returns 200 with a partial result set and the skipped URL is not charged, so read the per-URL results before treating the batch as complete. Set respect_robots: false to override for targets you have your own agreement with — the override is recorded against your API key. The override does not reach a host on CrawlForge's permanent opt-out list: that one is refused whatever respect_robots is set to, and it is skipped and left uncharged in the same way, with the host named in that result's error. data.notes summarises how many URLs were skipped and why.

Note: A 200 response does not mean every URL worked. Read succeeded, failed and skipped — or each entry's status — before treating the batch as complete.

Credit Cost

5 credits

5 credits per URL attempted

A 10-URL batch costs 50 credits. URLs that came back skipped because the time budget ran out are not charged, and a batch rejected at validation costs nothing. Failed URLs are charged — the fetch was attempted.

What's Included:

Up to 50 URLs per call, fetched concurrently

Title and body text per page (5,000 characters)

CSS field extraction with per-field notes

Per-URL status and HTTP code

24-hour result storage for get_batch_results

Plan Recommendations:

Free Plan: 1,000 one-time trial credits = 200 URLs

Hobby Plan: 5,000 credits = 1,000 URLs ($19/mo)

Professional Plan: 50,000 credits = 10,000 URLs ($99/mo)

Related Tools

get_batch_results

Re-read a batch's stored results within 24 hours (1 credit)

map_site

Discover the URLs to feed into a batch (2 credits)

scrape

One URL, more output formats and options (2 credits)

crawl_deep

Follow links yourself instead of supplying a list (4 credits)

Ready to try batch_scrape? Sign up for free and get 1,000 credits to start building.

Advanced Tool5 credits

batch_scrape

Use Cases

Competitor Price Sweeps

Pull the same CSS field off fifty product pages in one request, instead of fifty round trips.

Link Health Checks

Submit a URL list and read http_status per entry. Anything unreachable comes back as failed with the reason attached.

Content Inventories

Feed a sitemap into the batch and collect every title and canonical URL for an audit spreadsheet.

Post-Deploy Verification

Check that a set of critical pages still returns 200 and still contains the element you expect.

Dataset Assembly

Collect the first 5,000 characters of body text from a URL list as the input to a downstream pipeline.

Deferred Collection

Run the batch now and pull the stored results later from a different process with get_batch_results.

Endpoint

POST/api/v1/tools/batch_scrape

Auth Required

1 req/s on Free plan

5 credits

Parameters

Name	Type	Required	Default	Description
urls	array	Required	-	The URLs to fetch. Between 1 and 50 entries; a longer list is rejected rather than truncated. Example: [{ "url": "https://example.com/widget-a", "id": "a" }]
batch_config	object	Optional	-	Execution settings for the batch. Example: { "concurrency": 5 }
extraction_template	object	Optional	-	Fields to pull out of every page in the batch. Example: { "fields": [{ "name": "price", "selector": ".price" }] }
output_config	object	Optional	-	Accepted and ignored in full. Responses are always JSON in the shape shown below; `format`, `include_metadata`, `include_errors` and `flatten_results` have no effect. Example: { "format": "json" }
options	object	Optional	-	Accepted and ignored, except that `javascript_enabled: true` adds an explanatory line to `notes`. `user_agent`, `follow_redirects`, `respect_robots_txt` and `rate_limit_per_domain` do not change fetching on the hosted REST API. Example: { "javascript_enabled": false }
respect_robots	boolean	Optional	true	Respect each target site's robots.txt. Left at `true`, a URL disallowed for `CrawlForge` is skipped and the rest of the batch runs on — the request still returns 200 with a partial result set rather than a 403, and a skipped URL is not charged. Set it to `false` only for targets you have your own agreement with — the response then carries a `warnings` entry and the override is recorded against your API key. Example: true

Execution Model

Worth knowing before you size a batch — this endpoint runs inside a 30-second serverless function.

Synchronous

One request in, finished results out. There is no job id to poll and no webhook — batch_id is a retrieval key, not a job handle.

20s fetch budget

URLs not started before the budget runs out come back with a skipped status and are not charged. Individual fetches time out at 8s.

Failures are per URL

A 404, a timeout or an unreadable body marks that one entry failed. The call itself still returns 200.

No JavaScript

Pages are fetched, not rendered. For client-rendered pages use scrape_with_actions or the CrawlForge MCP server.

Request Examples

terminalBash

curl -X POST https://crawlforge.dev/api/v1/tools/batch_scrape \
  -H "X-API-Key: cf_test_YOUR_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "urls": [
      { "url": "https://example.com/widget-a", "id": "a" },
      { "url": "https://example.com/widget-b", "id": "b" }
    ],
    "batch_config": { "concurrency": 5 },
    "extraction_template": {
      "fields": [
        { "name": "price", "selector": ".price" },
        { "name": "canonical", "selector": "link[rel=canonical]", "attribute": "href" }
      ]
    }
  }'

Response Example

200 OK3,140ms

{
  "success": true,
  "data": {
    "batch_id": "fd599a27-9ca9-4022-9a57-b02af57c387b",
    "total": 3,
    "succeeded": 2,
    "failed": 1,
    "skipped": 0,
    "results": [
      {
        "id": "a",
        "url": "https://example.com/widget-a",
        "status": "success",
        "http_status": 200,
        "title": "Widget A — Acme",
        "text": "Widget A$24.00In stock and ready to ship.",
        "fields": {
          "price": "$24.00",
          "canonical": "https://example.com/widget-a",
          "sku": null
        },
        "field_notes": [
          "sku: xpath selectors are not supported on the hosted REST API — use a CSS selector"
        ]
      },
      {
        "id": "b",
        "url": "https://example.com/widget-b",
        "status": "success",
        "http_status": 200,
        "title": "Widget B — Acme",
        "text": "Widget B$31.50Backordered until March.",
        "fields": {
          "price": "$31.50",
          "canonical": "https://example.com/widget-b",
          "sku": null
        }
      },
      {
        "id": "c",
        "url": "https://example.com/gone",
        "status": "failed",
        "http_status": 404,
        "error": "HTTP 404"
      }
    ],
    "notes": [
      "Results are stored for 24h and retrievable with get_batch_results (batch_id: fd599a27-9ca9-4022-9a57-b02af57c387b)."
    ],
    "completed_at": "2026-08-27T02:09:44.001Z"
  },
  "credits_used": 15,
  "credits_remaining": 990,
  "processing_time": 3140
}