CrawlForge MCP Turns Any Website Into Clean Data for AI Agents
One API for scraping, search, and extraction — 31 MCP-native tools that plug straight into Claude, Cursor, LangChain, or your own code. Get clean markdown or structured JSON from any URL in a single call.
How It Works
Three steps from zero to live web data in your AI agent.
Install
Add the CrawlForge MCP server to your AI client with a single config entry.
Configure
Set your API key and choose which tools to enable for your workflow.
Scrape
Call any of 31 tools directly from your AI agent. Data flows back instantly.
Simple to Integrate, Powerful to Use
Add CrawlForge to your MCP client config and start calling tools immediately.
{
"mcpServers": {
"crawlforge": {
"command": "npx",
"args": ["-y", "crawlforge-mcp-server"]
}
}
}// Extract clean content from any URL
const content = await mcp.extract_content({
url: "https://example.com/blog/post",
format: "markdown",
});
// Search the web and get structured results
const results = await mcp.search_web({
query: "latest AI infrastructure trends",
max_results: 10,
});
// Deep research with conflict detection
const research = await mcp.deep_research({
query: "Compare pricing of top cloud providers",
sources: 5,
});Everything You Need for Web Scraping
31 real MCP tools with transparent credit-based pricing. From simple extraction to deep multi-source research.
Real-time Scraping
Get fresh data with millisecond latency and automatic retry logic
AI-Optimized Output
Structured data formats perfect for LLM training and inference
Credit-based Pricing
Pay only for what you use with transparent, predictable costs
fetch_url
Fetch raw HTML or rendered content from any URL
extract_text
Extract clean plain text from web pages
extract_links
Extract and categorize all links from a page
extract_metadata
Pull title, description, OG tags, and structured data
scrape_structured
Extract data matching a custom schema definition
extract_content
Get clean, AI-ready markdown or text content
map_site
Discover and map all pages on a website
process_document
Process PDFs, DOCs, and other document formats
localization
Detect and extract content across languages
extract_embedded_state
Read data from a page's embedded JavaScript state (Next.js, Nuxt, Redux)
track_changes
Track page changes with baselines, schedules, and alerts
analyze_content
Analyze readability, topics, and sentiment
extract_structured
LLM-powered schema extraction with selector fallback
summarize_content
Generate concise summaries of web page content
crawl_deep
Crawl entire sites following internal links
search_web
Search Google and get structured SERP results
reddit_search
Search Reddit posts, comments, or full threads via community archives
serp_rank
Check where a domain ranks in Google's organic results for a keyword
batch_scrape
Process hundreds of URLs in parallel
scrape_with_actions
Scrape pages that require clicks, scrolls, or form fills
stealth_mode
Bypass anti-bot protections with residential proxies
generate_llms_txt
Generate standard-compliant llms.txt for AI crawlers
deep_research
Multi-source research with conflict detection
extract_with_llm
AI extraction with local Ollama by default — no separate LLM-provider key required
list_ollama_models
Discover available local Ollama models for AI extraction.
scrape_template
Pre-built scrapers for GitHub, Amazon, LinkedIn, and 7 more platforms.
scrape
Unified multi-format scrape — markdown, HTML, links, metadata, and more in one fetch
agent
Autonomous research from a natural-language prompt — no URLs required
get_batch_results
Retrieve paginated results from a batch_scrape job
read_result
Read back a result that came back truncated — slice, search, page by line or json_path it by handle
Built for Real Production Workflows
From AI agent pipelines to competitive intelligence, see how teams use CrawlForge.
AI Agent Data Pipelines
Feed your AI agents live web data with structured extraction and multi-source research. Build pipelines with 31 MCP tools, from fetch_url to deep_research.
E-commerce Price Monitoring
Track competitor prices and product availability across hundreds of e-commerce sites. Detect undercutting the day it happens and reprice with confidence.
SEO Analysis & Auditing
Audit your site and competitors for metadata, broken links, content gaps, and ranking opportunities. Returns a structured report across every crawled page.
Competitive Intelligence
Gather competitor data discreetly using stealth browsing and structured extraction. Watch pricing, messaging, and launches without tripping bot defenses.
Built for Modern AI Workflows
Works with the tools and frameworks your team already uses.
Works with your favorite AI tools
Pricing that scales with your usage
Start with 1,000 free credits — no credit card. Upgrade only when your usage grows.
Free
Perfect for trying out CrawlForge MCP
Hobby
Great for personal projects and small applications
Billed $190/year· Save $38
Professional
Perfect for growing businesses and production applications
Billed $990/year· Save $198
How credits are spent
Paid plans roll unused credits over while your subscription is active. The Free plan's 1,000 credits are one-time.
Yearly plans get two months free and receive the full year's credits on day one.