You want your AI to scrape the web. Here is how to make that happen in under a minute.
Prerequisites
- Node.js 18+ -- check with
node --version - Claude Code -- install with
npm install -g @anthropic-ai/claude-code
That is it. Let us go.
Step 1: Install CrawlForge
npm install -g crawlforge-mcp-serverStep 2: Get Your Free API Key
- Go to crawlforge.
dev/ signup - Create your account (no credit card required)
- Copy your API key (starts with
cf_live_)
You now have 1,000 free credits to use.
Step 3: Configure Claude Code
Run the setup wizard:
npx crawlforge-setupEnter your API key when prompted. The wizard verifies your connection and configures Claude Code automatically.
Alternative: Set the environment variable manually:
export CRAWLFORGE_API_KEY="cf_live_your_key_here"You Are Ready. Try These Examples.
Open Claude Code and paste any of these prompts:
Fetch a Web Page (1 credit)
Fetch https://news.ycombinator.com and show me the top 5 stories
Claude uses fetch_url to grab the page, then parses the HTML to extract story titles and links.
Search the Web (5 credits)
Search the web for "best MCP servers for web scraping 2026" and summarize the top 3 results
Claude uses search_web to query Google, then summarizes each result.
Extract Structured Data (2 credits)
Extract all product names and prices from https://books.toscrape.com
Claude uses scrape_structured with CSS selectors to pull structured data into a clean table.
Deep Research (10 credits)
Research the current state of the MCP ecosystem: key players, adoption stats, and where it is heading
Claude uses deep_research to query multiple sources, cross-reference claims, and produce a cited report.
Stealth Scrape (5 credits)
Use stealth mode to fetch the pricing page from cloudflare.com
Claude uses stealth_mode with browser fingerprint rotation and the proxy you supply to get past anti-bot protections.
Credit Costs at a Glance
| Tool | Credits | Use Case |
|---|---|---|
| fetch_url, extract_text, extract_links | 1 | Basic page fetching |
| scrape_structured, extract_content, map_site | 2 | Data extraction |
| analyze_content | 3 | Sentiment, topic, and entity analysis |
| crawl_deep | 4 | Multi-page crawling |
| search_web, batch_scrape | 5 | Search and bulk operations |
| deep_research | 10 | Multi-source AI research |
Your 1,000 free credits give you 1,000 basic fetches, 200 searches, or 100 deep research operations.
Next Steps
- Read the launch announcement for the full feature overview
- Browse the API reference for all 31 tools
- Check out the complete MCP guide for advanced patterns
- Set up Claude Desktop if you prefer the GUI
Questions? Join our Discord or open an issue on GitHub. Happy scraping!
