On this page
Today we are launching CrawlForge MCP -- a complete web scraping platform built from the ground up for AI agents. Whether you are building with Claude, Cursor, or any MCP-compatible client, CrawlForge gives your AI the power to fetch, extract, analyze, and research web data through 18 specialized tools.
And yes, it is completely free to start. Every new account gets 1,000 credits -- no credit card required.
Table of Contents
- Why We Built CrawlForge
- 18 Tools for Every Scraping Scenario
- How MCP Integration Works
- Credit-Based Pricing That Scales
- Getting Started in 60 Seconds
- What Makes CrawlForge Different
- What Is Next
Why We Built CrawlForge
AI agents are powerful reasoners, but they are blind to the live web. Ask Claude to check a competitor's pricing page, scrape product reviews, or research a market -- and it hits a wall. Traditional scraping tools were never designed for the way AI agents work.
CrawlForge changes that. We built an MCP server that speaks the language AI agents already understand: the Model Context Protocol. No custom API wrappers. No SDK setup. Just install, configure, and your AI can scrape.
18 Tools for Every Scraping Scenario
We did not build one generic scraper and call it a day. CrawlForge ships with 18 purpose-built tools, each optimized for a specific task:
Basic Extraction
- fetch_url (1 credit) -- Retrieve any web page with full HTML
- extract_text (1 credit) -- Pull clean, readable text from any URL
- extract_links (1 credit) -- Discover all links on a page
- extract_metadata (1 credit) -- Get title, description, Open Graph tags, and more
Structured Data
- scrape_structured (2 credits) -- Extract data using CSS selectors or schemas
- extract_content (2 credits) -- Intelligent main content extraction
- map_site (3 credits) -- Generate a complete sitemap of any domain
- process_document (3 credits) -- Parse PDFs, DOCX, and other file formats
Advanced Capabilities
- search_web (5 credits) -- Google search integration with structured results
- crawl_deep (5 credits) -- Multi-page crawling with configurable depth
- batch_scrape (5 credits) -- Process hundreds of URLs in parallel
- scrape_with_actions (5 credits) -- Browser automation with clicks, scrolls, and form fills
AI-Powered
- deep_research (10 credits) -- Multi-source research with conflict detection
- analyze_content (3 credits) -- Sentiment, topic, and entity analysis
- summarize_content (2 credits) -- Intelligent content summarization
- stealth_mode (5 credits) -- Bypass anti-bot detection with residential proxies
- localization (3 credits) -- Access geo-restricted content from any country
- generate_llms_txt (2 credits) -- Create llms.txt files for any site
Every tool is available through the same MCP interface. Your AI agent picks the right one automatically based on what you ask it to do.
How MCP Integration Works
The Model Context Protocol is an open standard that lets AI assistants connect to external tools. CrawlForge implements this protocol natively, which means:
- Claude Code discovers all 18 tools automatically
- Cursor can use CrawlForge for code research and documentation fetching
- Any MCP client works out of the box -- no custom integration needed
Here is what it looks like in practice:
No code to write. No API keys to pass manually. Claude handles the tool selection, parameter construction, and result parsing.
Credit-Based Pricing That Scales
We designed pricing around a simple principle: pay for what you use, not what you might use.
| Plan | Monthly Credits | Price | Best For |
|---|---|---|---|
| Free | 1,000 | $0 | Trying it out, personal projects |
| Hobby | 10,000 | $19/mo | Side projects, small apps |
| Professional | 50,000 | $99/mo | Production workloads |
| Enterprise | 200,000 | $399/mo | High-volume, team access |
Simple tools like fetch_url cost just 1 credit. Advanced tools like deep_research cost 10. Most developers find the free tier covers their experimentation, and the Hobby plan handles production use cases comfortably.
View all pricing details on our site.
Getting Started in 60 Seconds
Three steps. That is all it takes.
Step 1: Install
Step 2: Get your free API key
Sign up at crawlforge.dev/signup -- you will receive 1,000 credits instantly.
Step 3: Configure Claude Code
That is it. Ask Claude to "search the web for the latest news on AI agents" and watch CrawlForge handle the rest.
For a detailed walkthrough, see our quick start guide.
What Makes CrawlForge Different
Native MCP, not an afterthought. We built CrawlForge as an MCP server from day one. Every tool is optimized for how AI agents discover, invoke, and process results.
18 specialized tools, not one generic scraper. Different scraping tasks have different requirements. A simple page fetch should not cost the same or use the same logic as a multi-source research operation.
Built-in anti-bot bypass. Our stealth mode uses residential proxies and browser fingerprint rotation to access sites that block traditional scrapers.
Deep research that actually works. The deep_research tool does not just fetch pages -- it queries multiple sources, cross-references claims, detects conflicts, and produces cited reports.
Transparent credit costs. Every tool lists its credit cost upfront. No surprise bills. No hidden fees. Your dashboard shows real-time usage.
What Is Next
This launch is just the beginning. Here is what we are working on:
- Webhooks -- Get notified when long-running scraping jobs complete
- Custom extractors -- Define reusable extraction schemas for your specific use cases
- Team workspaces -- Share API keys and manage usage across your organization
- More MCP clients -- Dedicated integrations for Windsurf, Cline, and other AI tools
We are building CrawlForge in the open. Follow us on Twitter for updates, or join our Discord to share feedback and feature requests.
Ready to give your AI the power to scrape the web? Start free with 1,000 credits -- no credit card required.
