API Reference
CrawlForge MCP API Reference
Complete reference for all 27 CrawlForge MCP tools. Build powerful web scraping applications with our developer-first API.
API Overview
Base URL
Bash
https://crawlforge.dev/api/v1Authentication
All API requests require an API key passed in the X-API-Key header:
Bash
curl -H "X-API-Key: cf_test_YOUR_KEY" \
https://crawlforge.dev/api/v1/tools/fetch_urlGet your API key from the dashboard.
Request Format
All tool endpoints accept JSON in the request body:
Json
{
"url": "https://example.com",
"timeout": 10000,
"follow_redirects": true
}Response Format
All responses follow a standard format:
Json
{
"success": true,
"data": {
// Tool-specific response data
},
"credits_used": 1,
"credits_remaining": 999,
"processing_time": 245
}HTTP Status Codes
| Code | Description |
|---|---|
| 200 | OK - Request successful |
| 400 | Bad Request - Invalid parameters |
| 401 | Unauthorized - Invalid or missing API key |
| 402 | Payment Required - Insufficient credits |
| 429 | Too Many Requests - Rate limit exceeded |
| 500 | Internal Server Error |
Rate Limits
Rate limits apply per API key and vary by plan. Exceeding the limit returns a
429 Too Many Requests response.Free Plan
1 requests per second
Hobby Plan
2 requests per second
Professional Plan
4 requests per second
Business Plan
10 requests per second
All Tools
Basic Tools 1-2 credits
list_ollama_models
List locally installed Ollama models (discovery helper)
1 credit
fetch_url
Fetch and parse web pages with automatic handling
1 credit
get_batch_results
Retrieve paginated results for a batch_scrape job
1 credit
extract_text
Extract clean text from HTML with intelligent parsing
1 credit
extract_links
Discover and extract all links from a webpage
1 credit
extract_metadata
Extract page metadata, OpenGraph, and Twitter Card tags
1 credit
scrape_template
Pre-built scrapers for 10 popular sites (Amazon, GitHub, YouTube, and more)
1 credit
scrape
Unified single-fetch extraction — markdown, HTML, text, links, metadata, and more
2 credits
scrape_structured
Extract structured data using CSS selectors
2 credits
extract_content
Extract main content from web pages with readability detection
2 credits
map_site
Discover and map website structure and URLs
2 credits
process_document
Process documents from multiple formats including PDFs
2 credits
localization
Multi-language and geo-location management for scraping
2 credits
Advanced Tools 3-5 credits
track_changes
Track page changes with baselines, scheduled monitoring, and alerts
3 credits
analyze_content
Analyze content for sentiment, language, and topics
3 credits
extract_structured
LLM-powered schema-driven extraction with CSS selector fallback
3 credits
extract_with_llm
LLM-powered extraction, local Ollama by default (OpenAI/Anthropic optional)
3 credits
summarize_content
Generate intelligent summaries of text content
4 credits
crawl_deep
Deep website crawling using breadth-first search
4 credits
stealth_mode
Anti-detection browser management with fingerprint randomization
5 credits
scrape_with_actions
Execute browser actions before scraping (clicks, forms, etc.)
5 credits
batch_scrape
Process multiple URLs simultaneously with async job management
5 credits
search_web
Search the web using Google Custom Search API
5 credits
serp_rank
Check a domain's organic Google SERP position for a keyword
5 credits
generate_llms_txt
Generate standard-compliant llms.txt and llms-full.txt files
5 credits
Premium Tools 8-10 credits
agent
Autonomous research from a natural-language prompt — no URLs required
8 credits
deep_research
Multi-stage research with AI synthesis and source verification
10 credits
Error Handling
All errors follow a standard format with
error.code and error.message fields.Json
{
"error": {
"code": "VALIDATION_ERROR",
"message": "Invalid input parameters",
"details": [
{
"field": "url",
"message": "Invalid URL format"
}
]
}
}See the FAQ for a complete list of error codes and solutions.
Ready to start building?
Pick a tool and explore its documentation