31 Web Scraping Tools in One MCP Server: The Complete CrawlForge Guide

Web scraping has never been more critical for AI applications. Whether you're building RAG systems, training models, or creating intelligent agents, you need reliable access to web data. CrawlForge MCP delivers 31 specialized tools in a single package, designed specifically for developers building with Claude and other LLMs.

Why One MCP Server Changes Everything

Traditional scraping solutions force you to cobble together multiple tools:

• A basic HTTP client for simple fetches
• A browser automation framework for JavaScript-heavy sites
• A separate service for search
• Another tool for content extraction
• Yet another for monitoring changes

With CrawlForge, you get one unified API with consistent authentication, pricing, and response formats. Claude can intelligently choose the right tool for each task.

The Complete Tool Reference

Basic Tools (1 Credit Each)

These foundational tools handle the most common scraping tasks efficiently:

fetch_url

The simplest tool - fetches raw HTML from any URL with automatic redirect handling.

curl -X POST https://crawlforge.dev/api/v1/tools/fetch_url \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -d '{"url": "https://example.com"}'

Best for: Initial page loads, API endpoints, static content

extract_text

Strips HTML and returns clean, readable text content.

curl -X POST https://crawlforge.dev/api/v1/tools/extract_text \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -d '{"url": "https://example.com/article"}'

Best for: Content analysis, LLM context, text processing

extract_links

Parses all anchor tags and returns structured link data.

Best for: Site mapping, crawler seeds, SEO analysis

extract_metadata

Pulls SEO metadata, Open Graph tags, Twitter cards, and Schema.org data.

Best for: Content previews, SEO audits, social sharing analysis

Structured Extraction Tools (2 Credits Each)

When you need more than raw content:

scrape_structured

Use CSS selectors to extract specific elements into structured JSON.

{
  "url": "https://store.example.com/products",
  "selectors": {
    "title": "h1.product-title",
    "price": "span.price",
    "rating": "div.rating-stars"
  }
}

Best for: E-commerce data, listings, structured pages

extract_content

Intelligent main content extraction - removes navigation, ads, and boilerplate.

Best for: Articles, blog posts, documentation pages

map_site

Discovers and maps website structure, finding all accessible URLs.

{
  "url": "https://docs.example.com",
  "maxDepth": 3,
  "includePatterns": ["/docs/*"]
}

Best for: Pre-crawl planning, documentation indexing, sitemap generation

process_document

Extracts text from PDFs and other document formats via URL.

Best for: PDF scraping, document processing, academic papers

localization

Geo-targeted scraping with 26+ country proxies, timezone spoofing, and locale headers.

{
  "url": "https://store.example.com",
  "country": "UK",
  "language": "en-GB"
}

Best for: Price comparison, localized content, geo-restricted sites

Advanced Tools (3-5 Credits)

For complex scraping scenarios:

analyze_content (3 Credits)

AI-powered content analysis including sentiment, language detection, and topic extraction.

Best for: Sentiment analysis, content classification, language detection

stealth_mode (5 Credits)

Anti-detection browsing with fingerprint randomization and human behavior simulation -- the same techniques used by Playwright under the hood, with stealth patches applied.

Best for: Sites with bot detection, Cloudflare-protected pages

summarize_content (4 Credits)

AI-generated summaries with configurable length and focus.

Best for: Content digests, research summaries, quick overviews

crawl_deep (4 Credits)

Multi-page crawling with depth control, pattern matching, and content extraction. Flat 4 credits per call regardless of how many pages are crawled.

{
  "url": "https://blog.example.com",
  "maxDepth": 2,
  "maxPages": 50,
  "extractContent": true
}

Best for: Blog archives, documentation sites, full-site indexing

scrape_with_actions (5 Credits)

Browser automation with click, type, scroll, and screenshot capabilities.

{
  "url": "https://app.example.com/login",
  "actions": [
    {"type": "type", "selector": "#email", "text": "user@example.com"},
    {"type": "click", "selector": "button[type=submit]"},
    {"type": "wait", "timeout": 3000}
  ]
}

Best for: Login-gated content, interactive forms, SPA navigation

batch_scrape (5 Credits)

Process multiple URLs in parallel with unified response format. Flat 5 credits per call regardless of URL count.

{
  "urls": [
    "https://site1.com",
    "https://site2.com",
    "https://site3.com"
  ],
  "maxConcurrency": 10
}

Best for: Bulk data collection, comparison scraping, efficiency

search_web (5 Credits)

Google Custom Search integration for discovering relevant URLs.

Best for: Research starting points, topic discovery, competitive analysis

track_changes (3 Credits)

Monitor websites for content changes with configurable sensitivity. Flat 3 credits per call.

Best for: Competitor monitoring, price tracking, news alerts

AI Research Tool (10 Credits)

deep_research

The most powerful tool - multi-stage research with source verification and synthesis.

{
  "topic": "Next.js 15 performance improvements",
  "depth": "moderate",
  "maxSources": 10
}

Returns:

• Synthesized summary
• Key findings with confidence scores
• Verified sources with relevance ranking
• Conflict detection between sources

Best for: Competitive intelligence, market research, technical research, fact-checking

Credit Optimization Tips

1. Start cheap: Use fetch_url (1 credit) before trying expensive tools
2. Batch when possible: batch_scrape is more efficient than individual calls
3. Know your URLs: Don't use search_web (5 credits) when you have the URL
4. Cache results: Same URL = same content, don't re-scrape unnecessarily
5. Use the right tool: extract_content (2 credits) beats manual parsing

Pricing Comparison

Get Started

1. Sign up free at crawlforge.dev/signup
2. Get 1,000 credits instantly (no credit card)
3. Add to Claude Desktop in 5 minutes (guide)

Ready to start? Create your free account at crawlforge.dev and unlock all 31 tools today.