Basic ToolMulti-Format2 credits

scrape

Unified single-fetch, multi-format extraction. Ask one page load for markdown, HTML, raw HTML, text, links, metadata, a screenshot, or JSON — every requested format is served from the same fetch, and a format that fails comes back as a warning instead of failing the whole call.

Use Cases

One Call Instead of Four

Get markdown, links, and metadata from a page in a single request instead of chaining fetch_url, extract_links, and extract_metadata.

LLM-Ready Markdown

Request markdown with onlyMainContent enabled to feed clean, boilerplate-free page text straight into a RAG pipeline or prompt.

Archival Snapshots

Ask for rawHtml and screenshot together to capture both the machine-readable and the visual state of a page.

Resilient Pipelines

Per-format warnings let a partial result through — a failed screenshot never discards the markdown you already paid for.

Endpoint

POST/api/v1/tools/scrape

Auth Required

1 req/s on Free plan

2 credits

Parameters

Every requested format is served from a single page fetch, so asking for six formats costs the same 2 credits as asking for one.

Name	Type	Required	Default	Description
url	string	Required	-	The URL to scrape (must include protocol: http:// or https://) Example: https://example.com
formats	array	Optional	["markdown"]	Output formats to return. One or more of `markdown`, `html`, `rawHtml`, `text`, `links`, `metadata`, `screenshot`, `json-schema`. Example: ["markdown", "links", "metadata"]
onlyMainContent	boolean	Optional	true	Strip navigation, headers, and footers so only the main article content is returned. Example: true

Request Examples

cURL

terminalBash

curl -X POST https://crawlforge.dev/api/v1/tools/scrape \
  -H "X-API-Key: cf_test_YOUR_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://example.com",
    "formats": ["markdown", "links", "metadata"],
    "onlyMainContent": true
  }'

TypeScript

scrape.tsTypescript

const response = await fetch('https://crawlforge.dev/api/v1/tools/scrape', {
  method: 'POST',
  headers: {
    'X-API-Key': process.env.CRAWLFORGE_API_KEY!,
    'Content-Type': 'application/json',
  },
  body: JSON.stringify({
    url: 'https://example.com',
    formats: ['markdown', 'links', 'metadata'],
    onlyMainContent: true,
  }),
});

const data = await response.json();

if (data.success) {
  console.log('Markdown:', data.data.markdown);
  console.log('Links found:', data.data.links.length);
  console.log('Title:', data.data.metadata.title);

  // A 200 with warnings is a PARTIAL success — some formats came back, others didn't.
  if (data.data.warnings.length > 0) {
    console.warn('Partial result:', data.data.warnings);
  }

  console.log('Credits used:', data.credits_used);
  console.log('Credits remaining:', data.credits_remaining);
} else {
  console.error('Error:', data.error);
}

Python

scrape.pyPython

import requests
import os

response = requests.post(
    'https://crawlforge.dev/api/v1/tools/scrape',
    headers={
        'X-API-Key': os.environ['CRAWLFORGE_API_KEY'],
        'Content-Type': 'application/json',
    },
    json={
        'url': 'https://example.com',
        'formats': ['markdown', 'links', 'metadata'],
        'onlyMainContent': True
    }
)

data = response.json()

if data['success']:
    print(f"Markdown: {data['data']['markdown']}")
    print(f"Links found: {len(data['data']['links'])}")
    print(f"Title: {data['data']['metadata']['title']}")

    # A 200 with warnings is a PARTIAL success — some formats came back, others didn't.
    if data['data']['warnings']:
        print(f"Partial result: {data['data']['warnings']}")

    print(f"Credits used: {data['credits_used']}")
    print(f"Credits remaining: {data['credits_remaining']}")
else:
    print(f"Error: {data['error']}")

Response Example

200 OK412ms

{
  "success": true,
  "data": {
    "url": "https://example.com",
    "onlyMainContent": true,
    "formats": [
      "markdown",
      "links",
      "metadata"
    ],
    "markdown": "# Example\n\nMain content scraped from https://example.com.",
    "links": [
      "https://example.com/about",
      "https://example.com/pricing"
    ],
    "metadata": {
      "title": "Example",
      "description": "Page at https://example.com"
    },
    "warnings": []
  },
  "credits_used": 2,
  "credits_remaining": 998,
  "processing_time": 412
}

Field Descriptions

data.formatsEchoes the formats that were requested

data.markdownMain content converted to markdown (present when `markdown` is requested)

data.linksEvery link discovered on the page (present when `links` is requested)

data.metadataTitle, description, and meta tags (present when `metadata` is requested)

data.warningsOne entry per format that could not be produced. An empty array means every requested format succeeded.

credits_usedCredits deducted for this request (2 per scrape, regardless of format count)

credits_remainingYour remaining credit balance

Error Handling

Invalid Input (400 Bad Request)

The URL format is invalid, or formats contains a value outside the supported list. At least one format is required when the field is supplied.

URL Blocked (403 Forbidden)

The target resolved to a private, internal, or link-local address and was rejected by SSRF protection. Only publicly reachable URLs can be scraped.

Insufficient Credits (402 Payment Required)

Your account doesn't have enough credits. Purchase more credits or upgrade your plan.

Rate Limit Exceeded (429 Too Many Requests)

You've exceeded your plan's rate limit. Wait a moment or upgrade your plan for higher limits.

Pro Tip: Check data.warnings on every response. A 200 with a non-empty warnings array means some formats came back and others didn't — treat it as partial success, not failure.

Credit Cost

2 credits

2 credits per request

Each successful scrape request costs 2 credits no matter how many formats you request, because every format is served from one fetch.

Free Plan: 1,000 one-time credits = 500 requests

Hobby Plan: 5,000 credits/month = 2,500 requests ($19/mo)

Professional Plan: 50,000 credits/month = 25,000 requests ($99/mo)

Business Plan: 250,000 credits/month = 125,000 requests ($399/mo)

Related Tools

fetch_url

Raw HTTP fetch when you want the untouched response body (1 credit)

extract_content

Readability-based main-content extraction for a single page (2 credits)

batch_scrape

Run the same extraction across many URLs as an async job (5 credits)

scrape_with_actions

Click, scroll, or fill forms before scraping a JS-heavy page (5 credits)

Ready to try scrape? Sign up for free and get 1,000 credits to start building.

Use Cases

One Call Instead of Four

Get markdown, links, and metadata from a page in a single request instead of chaining fetch_url, extract_links, and extract_metadata.

LLM-Ready Markdown

Request markdown with onlyMainContent enabled to feed clean, boilerplate-free page text straight into a RAG pipeline or prompt.

Archival Snapshots

Ask for rawHtml and screenshot together to capture both the machine-readable and the visual state of a page.

Resilient Pipelines

Per-format warnings let a partial result through — a failed screenshot never discards the markdown you already paid for.

Parameters

Every requested format is served from a single page fetch, so asking for six formats costs the same 2 credits as asking for one.

Name	Type	Required	Default	Description
url	string	Required	-	The URL to scrape (must include protocol: http:// or https://) Example: https://example.com
formats	array	Optional	["markdown"]	Output formats to return. One or more of `markdown`, `html`, `rawHtml`, `text`, `links`, `metadata`, `screenshot`, `json-schema`. Example: ["markdown", "links", "metadata"]
onlyMainContent	boolean	Optional	true	Strip navigation, headers, and footers so only the main article content is returned. Example: true

Request Examples

cURL

terminalBash

curl -X POST https://crawlforge.dev/api/v1/tools/scrape \
  -H "X-API-Key: cf_test_YOUR_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://example.com",
    "formats": ["markdown", "links", "metadata"],
    "onlyMainContent": true
  }'

TypeScript

scrape.tsTypescript

const response = await fetch('https://crawlforge.dev/api/v1/tools/scrape', {
  method: 'POST',
  headers: {
    'X-API-Key': process.env.CRAWLFORGE_API_KEY!,
    'Content-Type': 'application/json',
  },
  body: JSON.stringify({
    url: 'https://example.com',
    formats: ['markdown', 'links', 'metadata'],
    onlyMainContent: true,
  }),
});

const data = await response.json();

if (data.success) {
  console.log('Markdown:', data.data.markdown);
  console.log('Links found:', data.data.links.length);
  console.log('Title:', data.data.metadata.title);

  // A 200 with warnings is a PARTIAL success — some formats came back, others didn't.
  if (data.data.warnings.length > 0) {
    console.warn('Partial result:', data.data.warnings);
  }

  console.log('Credits used:', data.credits_used);
  console.log('Credits remaining:', data.credits_remaining);
} else {
  console.error('Error:', data.error);
}

Python

scrape.pyPython

import requests
import os

response = requests.post(
    'https://crawlforge.dev/api/v1/tools/scrape',
    headers={
        'X-API-Key': os.environ['CRAWLFORGE_API_KEY'],
        'Content-Type': 'application/json',
    },
    json={
        'url': 'https://example.com',
        'formats': ['markdown', 'links', 'metadata'],
        'onlyMainContent': True
    }
)

data = response.json()

if data['success']:
    print(f"Markdown: {data['data']['markdown']}")
    print(f"Links found: {len(data['data']['links'])}")
    print(f"Title: {data['data']['metadata']['title']}")

    # A 200 with warnings is a PARTIAL success — some formats came back, others didn't.
    if data['data']['warnings']:
        print(f"Partial result: {data['data']['warnings']}")

    print(f"Credits used: {data['credits_used']}")
    print(f"Credits remaining: {data['credits_remaining']}")
else:
    print(f"Error: {data['error']}")

Response Example

200 OK412ms

{
  "success": true,
  "data": {
    "url": "https://example.com",
    "onlyMainContent": true,
    "formats": [
      "markdown",
      "links",
      "metadata"
    ],
    "markdown": "# Example\n\nMain content scraped from https://example.com.",
    "links": [
      "https://example.com/about",
      "https://example.com/pricing"
    ],
    "metadata": {
      "title": "Example",
      "description": "Page at https://example.com"
    },
    "warnings": []
  },
  "credits_used": 2,
  "credits_remaining": 998,
  "processing_time": 412
}

Field Descriptions

data.formatsEchoes the formats that were requested

data.markdownMain content converted to markdown (present when `markdown` is requested)

data.linksEvery link discovered on the page (present when `links` is requested)

data.metadataTitle, description, and meta tags (present when `metadata` is requested)

data.warningsOne entry per format that could not be produced. An empty array means every requested format succeeded.

credits_usedCredits deducted for this request (2 per scrape, regardless of format count)

credits_remainingYour remaining credit balance

Error Handling

Invalid Input (400 Bad Request)

The URL format is invalid, or formats contains a value outside the supported list. At least one format is required when the field is supplied.

URL Blocked (403 Forbidden)

The target resolved to a private, internal, or link-local address and was rejected by SSRF protection. Only publicly reachable URLs can be scraped.

Insufficient Credits (402 Payment Required)

Your account doesn't have enough credits. Purchase more credits or upgrade your plan.

Rate Limit Exceeded (429 Too Many Requests)

You've exceeded your plan's rate limit. Wait a moment or upgrade your plan for higher limits.

Pro Tip: Check data.warnings on every response. A 200 with a non-empty warnings array means some formats came back and others didn't — treat it as partial success, not failure.

Credit Cost

2 credits

2 credits per request

Each successful scrape request costs 2 credits no matter how many formats you request, because every format is served from one fetch.

Free Plan: 1,000 one-time credits = 500 requests

Hobby Plan: 5,000 credits/month = 2,500 requests ($19/mo)

Professional Plan: 50,000 credits/month = 25,000 requests ($99/mo)

Business Plan: 250,000 credits/month = 125,000 requests ($399/mo)