API Reference

CrawlForge MCP API Reference

Complete reference for all 29 CrawlForge MCP tools. Build powerful web scraping applications with our developer-first API.

API Overview

Base URL

Bash

https://crawlforge.dev/api/v1

Authentication

All API requests require an API key passed in the X-API-Key header:

Bash

curl -H "X-API-Key: cf_test_YOUR_KEY" \
     https://crawlforge.dev/api/v1/tools/fetch_url

Get your API key from the dashboard.

Request Format

All tool endpoints accept JSON in the request body:

Json

{
  "url": "https://example.com",
  "timeout": 10000,
  "follow_redirects": true
}

Response Format

All responses follow a standard format:

Json

{
  "success": true,
  "data": {
    // Tool-specific response data
  },
  "credits_used": 1,
  "credits_remaining": 999,
  "processing_time": 245
}

A response can also carry an optional top-level warnings array of strings, sitting alongside data and credits_used rather than inside data. It is present only when a request produced a warning: you disabled the robots.txt check with respect_robots: false — which is also recorded against your API key — or a multi-URL tool skipped a URL or did not fetch a source. If you never override the check and every URL you request is allowed, no response carries the field at all; its absence is the normal case, not a missing value.

HTTP Status Codes

Code	Description
200	OK - Request successful
400	Bad Request - Invalid parameters
401	Unauthorized - Invalid or missing API key
402	Payment Required - Insufficient credits
403	Forbidden - robots.txt disallows the URL for `CrawlForge` (override with `respect_robots: false`), the host is on CrawlForge's permanent opt-out list (not overridable), or the URL resolves to a private or internal address. Nothing is fetched and no credits are charged
429	Too Many Requests - your plan's rate limit (slow down or upgrade), or the target site asked CrawlForge to wait via a `Retry-After` on its own 429 or 503, which no plan change affects and `respect_robots` does not override. Neither is charged
500	Internal Server Error

Rate Limits

Rate limits apply per API key and vary by plan. Exceeding the limit returns a 429 Too Many Requests response.

Free Plan

1 requests per second

Hobby Plan

2 requests per second

Professional Plan

4 requests per second

Business Plan

10 requests per second

All Tools

Basic Tools 1-2 credits

list_ollama_models

List locally installed Ollama models (discovery helper)

1 credit

fetch_url

Fetch and parse web pages with automatic handling

1 credit

get_batch_results

Retrieve paginated results for a batch_scrape job

1 credit

extract_text

Extract clean text from HTML with intelligent parsing

1 credit

extract_links

Discover and extract all links from a webpage

1 credit

extract_metadata

Extract page metadata, OpenGraph, and Twitter Card tags

1 credit

scrape_template

Pre-built scrapers for 10 popular sites (Amazon, GitHub, YouTube, and more)

1 credit

scrape

Unified single-fetch extraction — markdown, HTML, text, links, metadata, and more

2 credits

scrape_structured

Extract structured data using CSS selectors

2 credits

extract_content

Extract main content from web pages with readability detection

2 credits

map_site

Discover and map website structure and URLs

2 credits

process_document

Process documents from multiple formats including PDFs

2 credits

localization

Multi-language and geo-location management for scraping

2 credits

reddit_search

Search Reddit posts/comments or read a full thread via community archives

2 credits

Advanced Tools 3-5 credits

track_changes

Track page changes with baselines, scheduled monitoring, and alerts

3 credits

analyze_content

Analyze content for sentiment, language, and topics

3 credits

extract_embedded_state

Read the JSON state a page already ships in its own HTML - no LLM, no fabrication

2 credits

extract_structured

LLM-powered schema-driven extraction with CSS selector fallback

3 credits

extract_with_llm

LLM-powered extraction, local Ollama by default (OpenAI/Anthropic optional)

3 credits

summarize_content

Generate intelligent summaries of text content

4 credits

crawl_deep

Deep website crawling using breadth-first search

4 credits

stealth_mode

Anti-detection browser management with fingerprint randomization

5 credits

scrape_with_actions

Execute browser actions before scraping (clicks, forms, etc.)

5 credits

batch_scrape

Process multiple URLs simultaneously with async job management

5 credits

search_web

Search the web using Google Custom Search API

5 credits

serp_rank

Check a domain's organic Google SERP position for a keyword

5 credits

generate_llms_txt

Generate standard-compliant llms.txt and llms-full.txt files

5 credits

Premium Tools 8-10 credits

agent

Autonomous research from a natural-language prompt — no URLs required

8 credits

deep_research

Multi-stage research with AI synthesis and source verification

10 credits

Error Handling

All errors follow a standard format with error.code and error.message fields.

Json

{
  "error": {
    "code": "VALIDATION_ERROR",
    "message": "Invalid input parameters",
    "details": [
      {
        "field": "url",
        "message": "Invalid URL format"
      }
    ]
  }
}

See the FAQ for a complete list of error codes and solutions.

Ready to start building?

Pick a tool and explore its documentation

View Tool Docs Back to Getting Started

API Reference

CrawlForge MCP API Reference

Complete reference for all 29 CrawlForge MCP tools. Build powerful web scraping applications with our developer-first API.

API Overview

Base URL

Bash

https://crawlforge.dev/api/v1

Authentication

All API requests require an API key passed in the X-API-Key header:

Bash

curl -H "X-API-Key: cf_test_YOUR_KEY" \
     https://crawlforge.dev/api/v1/tools/fetch_url

Get your API key from the dashboard.

Request Format

All tool endpoints accept JSON in the request body:

Json

{
  "url": "https://example.com",
  "timeout": 10000,
  "follow_redirects": true
}

Response Format

All responses follow a standard format:

Json

{
  "success": true,
  "data": {
    // Tool-specific response data
  },
  "credits_used": 1,
  "credits_remaining": 999,
  "processing_time": 245
}

HTTP Status Codes

Code	Description
200	OK - Request successful
400	Bad Request - Invalid parameters
401	Unauthorized - Invalid or missing API key
402	Payment Required - Insufficient credits
403	Forbidden - robots.txt disallows the URL for `CrawlForge` (override with `respect_robots: false`), the host is on CrawlForge's permanent opt-out list (not overridable), or the URL resolves to a private or internal address. Nothing is fetched and no credits are charged
429	Too Many Requests - your plan's rate limit (slow down or upgrade), or the target site asked CrawlForge to wait via a `Retry-After` on its own 429 or 503, which no plan change affects and `respect_robots` does not override. Neither is charged
500	Internal Server Error

Rate Limits

Rate limits apply per API key and vary by plan. Exceeding the limit returns a 429 Too Many Requests response.

Free Plan

1 requests per second

Hobby Plan

2 requests per second

Professional Plan

4 requests per second

Business Plan

10 requests per second

All Tools

Basic Tools 1-2 credits

list_ollama_models

List locally installed Ollama models (discovery helper)

1 credit

fetch_url

Fetch and parse web pages with automatic handling

1 credit

get_batch_results

Retrieve paginated results for a batch_scrape job

1 credit

extract_text

Extract clean text from HTML with intelligent parsing

1 credit

extract_links

Discover and extract all links from a webpage

1 credit

extract_metadata

Extract page metadata, OpenGraph, and Twitter Card tags

1 credit

scrape_template

Pre-built scrapers for 10 popular sites (Amazon, GitHub, YouTube, and more)

1 credit

scrape

Unified single-fetch extraction — markdown, HTML, text, links, metadata, and more

2 credits

scrape_structured

Extract structured data using CSS selectors

2 credits

extract_content

Extract main content from web pages with readability detection

2 credits

map_site

Discover and map website structure and URLs

2 credits

process_document

Process documents from multiple formats including PDFs

2 credits

localization

Multi-language and geo-location management for scraping

2 credits

reddit_search

Search Reddit posts/comments or read a full thread via community archives

2 credits

Advanced Tools 3-5 credits

track_changes

Track page changes with baselines, scheduled monitoring, and alerts

3 credits

analyze_content

Analyze content for sentiment, language, and topics

3 credits

extract_embedded_state

Read the JSON state a page already ships in its own HTML - no LLM, no fabrication

2 credits

extract_structured

LLM-powered schema-driven extraction with CSS selector fallback

3 credits

extract_with_llm

LLM-powered extraction, local Ollama by default (OpenAI/Anthropic optional)

3 credits

summarize_content

Generate intelligent summaries of text content

4 credits

crawl_deep

Deep website crawling using breadth-first search

4 credits

stealth_mode

Anti-detection browser management with fingerprint randomization

5 credits

scrape_with_actions

Execute browser actions before scraping (clicks, forms, etc.)

5 credits

batch_scrape

Process multiple URLs simultaneously with async job management

5 credits

search_web

Search the web using Google Custom Search API

5 credits

serp_rank

Check a domain's organic Google SERP position for a keyword

5 credits

generate_llms_txt

Generate standard-compliant llms.txt and llms-full.txt files

5 credits

Premium Tools 8-10 credits

agent

Autonomous research from a natural-language prompt — no URLs required

8 credits

deep_research

Multi-stage research with AI synthesis and source verification

10 credits

Error Handling

All errors follow a standard format with error.code and error.message fields.

Json

{
  "error": {
    "code": "VALIDATION_ERROR",
    "message": "Invalid input parameters",
    "details": [
      {
        "field": "url",
        "message": "Invalid URL format"
      }
    ]
  }
}

See the FAQ for a complete list of error codes and solutions.

Ready to start building?

Pick a tool and explore its documentation

View Tool Docs Back to Getting Started