On this page
Claude Code can edit files, run shell commands, and write tests -- but it cannot fetch live web pages on its own. Connect it to CrawlForge MCP and it gains 20 scraping tools that run straight from your terminal.
This guide shows you how to scrape websites with Claude Code using CrawlForge MCP, from installation to stealth-mode bypass. Every code block below is runnable.
Table of Contents
- The Problem: Claude Code Cannot Fetch URLs
- Prerequisites
- Step 1: Install CrawlForge MCP
- Step 2: Get Your API Key
- Step 3: Register the MCP Server with Claude Code
- Step 4: Verify the Connection
- Step 5: Your First Scrape
- Full Working Example: Scrape a Pricing Page
- Advanced: Scrape JavaScript-Rendered Sites
- Troubleshooting
- FAQ
The Problem: Claude Code Cannot Fetch URLs
By default, Claude Code has no network access. Ask it to "read this blog post" and it will tell you it cannot open URLs. The built-in WebFetch helper exists in Claude Desktop but is limited, rate-capped, and frequently blocked by Cloudflare, Akamai, and other edge protections.
CrawlForge MCP solves this by exposing 20 scraping tools -- fetch_url, extract_content, scrape_structured, stealth_mode, deep_research, and more -- as Model Context Protocol tools that Claude Code can call like any other function. For more background on the protocol itself, see our complete guide to MCP web scraping.
Prerequisites
- Node.js 18+ -- check with
node --version - Claude Code -- install with
npm install -g @anthropic-ai/claude-code - A CrawlForge account -- free at crawlforge.dev/signup (1,000 credits included, no credit card)
Step 1: Install CrawlForge MCP
Verify the install:
Step 2: Get Your API Key
- Go to crawlforge.dev/signup and create an account.
- Open the dashboard at crawlforge.dev/dashboard/api-keys.
- Copy the key -- it starts with
cf_live_.
Step 3: Register the MCP Server with Claude Code
The fastest path is the setup wizard:
It writes the correct entry to ~/.config/claude-code/mcp.json (Linux/macOS) or %APPDATA%\claude-code\mcp.json (Windows).
If you prefer manual configuration, add this to your Claude Code MCP config:
Restart Claude Code so it picks up the new server.
Step 4: Verify the Connection
Open Claude Code and run:
/mcp
You should see crawlforge listed as connected with 20 tools available. If not, jump to Troubleshooting.
Step 5: Your First Scrape
Paste this prompt into Claude Code:
Fetch https://news.ycombinator.com using CrawlForge and give me the top 5 story titles with their URLs as a JSON array.
Claude Code will call fetch_url (1 credit), parse the HTML, and return something like:
That is it. You are scraping.
Full Working Example: Scrape a Pricing Page
Here is a realistic task: extract pricing tiers from a SaaS site. Paste this prompt:
Use scrape_structured to extract pricing from https://crawlforge.dev/pricing.
Return an array of { plan, price, credits, features[] }.
Behind the scenes, Claude Code builds a request like this:
Cost: 2 credits. Compare that to running a headless browser locally: zero infrastructure, no Puppeteer debugging, no Cloudflare roulette.
Advanced: Scrape JavaScript-Rendered Sites
Some sites render pricing or product data through client-side React. fetch_url returns the pre-hydration HTML shell and misses the data. Switch to scrape_with_actions (5 credits):
For Cloudflare and Akamai-protected sites, use stealth_mode (also 5 credits). We cover the fingerprint-rotation tradeoffs in our stealth mode deep dive.
Tool Quick Reference
| Tool | Credits | When to use |
|---|---|---|
fetch_url | 1 | Static HTML, you will parse yourself |
extract_text | 1 | Clean readable text from article pages |
extract_content | 2 | Readability-style main content extraction |
scrape_structured | 2 | CSS selectors into typed fields |
search_web | 5 | You do not know the URL yet |
scrape_with_actions | 5 | SPA requires clicks, waits, scrolls |
stealth_mode | 5 | Anti-bot systems (Cloudflare, DataDome) |
deep_research | 10 | Multi-source research with citations |
Full list in our 20-tools overview.
Troubleshooting
"MCP server failed to start" -- Confirm crawlforge-mcp-server is on your PATH. Run which crawlforge-mcp-server. If empty, reinstall globally: npm install -g crawlforge-mcp-server.
"Unauthorized" or 401 errors -- Your API key is missing or malformed. It must start with cf_live_. Re-export in your shell: export CRAWLFORGE_API_KEY="cf_live_..." and restart Claude Code.
"Insufficient credits" -- Check usage at crawlforge.dev/dashboard/usage. Free tier = 1,000 credits/month. Upgrade to Hobby ($19/mo) for 25,000.
Tools list is empty in /mcp -- MCP config is not being read. On macOS the file lives at ~/Library/Application Support/claude-code/mcp.json. On Linux: ~/.config/claude-code/mcp.json. On Windows: %APPDATA%\claude-code\mcp.json.
Cloudflare 403 on every fetch -- Swap fetch_url for stealth_mode. If you still get blocked, the target uses server-side JA3/JA4 fingerprinting; open an issue on GitHub with the URL.
FAQ
See the FAQ section below for quick answers to the most common questions.
Next Steps
- Read the CrawlForge quick start for five copy-paste examples
- Browse the getting started docs for full API reference
- Compare MCP clients in Claude Desktop vs Claude Code
- Evaluate alternatives at Firecrawl alternative
Start free with 1,000 credits at crawlforge.dev/signup. No credit card required.
