CrawlForge
API Reference

CrawlForge MCP API Reference

Complete reference for all 23 CrawlForge MCP tools. Build powerful web scraping applications with our developer-first API.

API Overview

Base URL

Bash
https://crawlforge.dev/api/v1

Authentication

All API requests require an API key passed in the X-API-Key header:

Bash
curl -H "X-API-Key: cf_test_YOUR_KEY" \
     https://crawlforge.dev/api/v1/tools/fetch_url

Get your API key from the dashboard.

Request Format

All tool endpoints accept JSON in the request body:

Json
{
  "url": "https://example.com",
  "timeout": 10000,
  "follow_redirects": true
}

Response Format

All responses follow a standard format:

Json
{
  "success": true,
  "data": {
    // Tool-specific response data
  },
  "credits_used": 1,
  "credits_remaining": 999,
  "processing_time": 245
}

HTTP Status Codes

CodeDescription
200OK - Request successful
400Bad Request - Invalid parameters
401Unauthorized - Invalid or missing API key
402Payment Required - Insufficient credits
429Too Many Requests - Rate limit exceeded
500Internal Server Error

Rate Limits

Rate limits apply per API key and vary by plan. Exceeding the limit returns a 429 Too Many Requests response.
Free Plan
1 requests per second
Hobby Plan
2 requests per second
Professional Plan
4 requests per second
Business Plan
10 requests per second

All Tools

Basic Tools 1-2 credits

list_ollama_models
List locally installed Ollama models (free discovery helper)

0 credits

fetch_url
Fetch and parse web pages with automatic handling

1 credit

extract_text
Extract clean text from HTML with intelligent parsing

1 credit

extract_links
Discover and extract all links from a webpage

1 credit

extract_metadata
Extract page metadata, OpenGraph, and Twitter Card tags

1 credit

scrape_template
Pre-built scrapers for 10 popular sites (Amazon, GitHub, YouTube, and more)

1 credit

scrape_structured
Extract structured data using CSS selectors

2 credits

extract_content
Extract main content from web pages with readability detection

2 credits

map_site
Discover and map website structure and URLs

2 credits

process_document
Process documents from multiple formats including PDFs

2 credits

localization
Multi-language and geo-location management for scraping

2 credits

Advanced Tools 3-5 credits

track_changes
Track page changes with baselines, scheduled monitoring, and alerts

3 credits

analyze_content
Analyze content for sentiment, language, and topics

3 credits

extract_structured
LLM-powered schema-driven extraction with CSS selector fallback

3 credits

extract_with_llm
LLM-powered extraction, local Ollama by default (OpenAI/Anthropic optional)

3 credits

summarize_content
Generate intelligent summaries of text content

4 credits

crawl_deep
Deep website crawling using breadth-first search

4 credits

stealth_mode
Anti-detection browser management with fingerprint randomization

5 credits

scrape_with_actions
Execute browser actions before scraping (clicks, forms, etc.)

5 credits

batch_scrape
Process multiple URLs simultaneously with async job management

5 credits

search_web
Search the web using Google Custom Search API

5 credits

generate_llms_txt
Generate standard-compliant llms.txt and llms-full.txt files

5 credits

Premium Tools 10 credits

deep_research
Multi-stage research with AI synthesis and source verification

10 credits

Error Handling

All errors follow a standard format with error.code and error.message fields.
Json
{
  "error": {
    "code": "VALIDATION_ERROR",
    "message": "Invalid input parameters",
    "details": [
      {
        "field": "url",
        "message": "Invalid URL format"
      }
    ]
  }
}

See the FAQ for a complete list of error codes and solutions.

Ready to start building?
Pick a tool and explore its documentation
View Tool DocsBack to Getting Started