How to Actually Use CrawlForge MCP in Claude Code

You installed CrawlForge MCP. Claude Code says it is connected. Then you ask for a page and Claude summarizes it from memory, or reaches for its built-in fetch, or asks permission four times before it does anything useful.

That is not a broken install. It is a gap in how MCP tools get invoked — and it is fixable in about five minutes of configuration.

claude mcp list
# crawlforge: npx -y crawlforge-mcp-server - ✔ Connected

This guide covers what happens after the install: how Claude decides to call a tool, how to make it prefer CrawlForge over its built-in web helpers, how to stop the permission prompts, and how to pick the tool that does the job for the fewest credits.

Table of Contents

• Why Claude Ignores Your MCP Server
• How Tool Calls Actually Work
• Step 1: Confirm the Server Is Connected
• Step 2: Name the Tool When You Need To
• Step 3: Write a CrawlForge Policy Into CLAUDE.md
• Step 4: Stop the Permission Prompts
• Choosing the Right Tool
• Protect Your Context Window
• Prompt Patterns That Work
• Share the Setup With Your Team
• Run It Headless in Scripts and CI
• Troubleshooting

Why Claude Ignores Your MCP Server

There are three separate failure modes, and they need different fixes.

Claude does not know the tools exist yet. Claude Code enables tool search by default. At session start it loads only tool names and the server's own instructions — full tool definitions stay deferred until a task needs them. This keeps your context window free, but it means "I don't see 27 tools listed" is normal and not a symptom of anything.

Claude has a built-in that looks close enough. Claude Code ships WebFetch and WebSearch. Ask for "the content of this page" and a reasonable model may well reach for the built-in rather than search for an MCP tool it has to discover first. Nothing is wrong; you simply have not told it which one you prefer.

The call is happening, but every one needs approval. Each MCP tool call raises a permission prompt until you allow it. Four prompts into a research task, it feels like the integration is fighting you.

The rest of this guide fixes all three.

How Tool Calls Actually Work

You do not call an MCP tool. You describe an outcome, and Claude picks a tool — the same way it picks between Read and Grep.

Every MCP tool has a canonical name in the form mcp__<server>__<tool>. With CrawlForge registered under the name crawlforge, its tools are:

mcp__crawlforge__scrape
mcp__crawlforge__search_web
mcp__crawlforge__deep_research
mcp__crawlforge__stealth_mode

You will not usually type those names. They matter for two things: permission rules, which match on the canonical name, and the escape hatch of naming a tool explicitly when Claude picks the wrong one.

Step 1: Confirm the Server Is Connected

Before debugging prompts, confirm the transport works. From your shell:

claude mcp list

Each server gets a health status. The four you will actually see:

Inside a session, /mcp opens the same view with a per-server detail panel, including an Issue: row when a server failed.

One nuance worth knowing: claude mcp add prints Added ... as soon as the config is written, without validating anything. A typo in your API key still prints a success line. claude mcp list is the check that matters.

Step 2: Name the Tool When You Need To

When Claude picks the wrong tool — or no tool — say which one you want. Both of these work:

Use scrape to get https://example.com/pricing as markdown.

Use mcp__crawlforge__deep_research to compare the three vendors on
that page, then write the findings to research.md.

The bare tool name is enough in practice. Reach for the fully qualified mcp__crawlforge__* form when a name is ambiguous across servers, or when you are writing a slash command or script that must be unambiguous.

Naming the tool is a good debugging move and a bad habit. If you find yourself doing it on every prompt, fix the default instead — that is the next step.

Step 3: Write a CrawlForge Policy Into CLAUDE.md

This is the single highest-leverage change in this guide. CLAUDE.md loads into context on every session, so a short policy block permanently changes which tool Claude reaches for:

# Web Access Policy

Use CrawlForge MCP tools for all web search and page fetching.

- Web search: `search_web` (not the built-in WebSearch)
- Single page: `scrape` with `formats: ["markdown"]`
- Article text only: `extract_content`
- Site structure: `map_site`, then `crawl_deep` if you need page bodies
- Multi-source research: `deep_research`
- JS-rendered or anti-bot pages: `scrape_with_actions` or `stealth_mode`

Prefer one `batch_scrape` call over a loop of single fetches.

Put it in your project's CLAUDE.md, or in ~/.claude/CLAUDE.md to apply it everywhere. Two sessions later you will have forgotten it exists, which is the point.

State the preference once, plainly. A block that shouts ALWAYS and NEVER at every line tends to make models overcorrect — reaching for a scraping tool on questions that never needed one.

Step 4: Stop the Permission Prompts

Add an allow rule so CrawlForge tools run without interruption. In .claude/settings.json:

{
  "permissions": {
    "allow": [
      "mcp__crawlforge__*"
    ]
  }
}

The pattern rules, straight from the permissions reference:

• mcp__crawlforge — every tool from the server
• mcp__crawlforge__* — same thing, wildcard form
• mcp__crawlforge__scrape — that one tool only
• mcp__crawlforge__extract_* — the four extraction tools

Allow rules must be anchored to a literal server prefix. A bare "mcp__*" is skipped with a warning and auto-approves nothing, because it would name no server you actually configured.

If you would rather approve the cheap tools and keep a hand on the expensive ones, allow the reads and leave the rest prompting:

{
  "permissions": {
    "allow": [
      "mcp__crawlforge__scrape",
      "mcp__crawlforge__extract_content",
      "mcp__crawlforge__search_web",
      "mcp__crawlforge__map_site"
    ]
  }
}

deep_research at 10 credits and agent at 8 then still ask first.

Choosing the Right Tool

Twenty-seven tools is a lot of surface area, and Claude will happily spend 5 credits where 1 would have done. The costs:

Three rules that cover most of it:

Escalate, don't start at the top. Try fetch_url (1) or scrape (2) first. Move to scrape_with_actions (5) only when the content is client-rendered, and stealth_mode (5) only after a real block. Plenty of sites that look protected serve fine to a well-formed request.

Batch instead of looping. One batch_scrape call is 5 credits regardless of URL count. Twelve separate scrape calls are 24 credits and twelve round trips. Say "scrape all of these in one batch" and the difference is real.

Know what deep_research replaces. At 10 credits it is the most expensive tool, and it is a bargain when it stands in for a search plus eight scrapes plus synthesis. It is waste when you already know the URL. If you have the link, scrape it.

Protect Your Context Window

This is the failure nobody warns you about: a successful scrape that ruins your session. Web pages are large, and MCP results land directly in the conversation.

Claude Code has guardrails. It warns when any MCP tool output exceeds 10,000 tokens, and caps output at 25,000 tokens by default. You can raise the ceiling:

export MAX_MCP_OUTPUT_TOKENS=50000

Raising it is usually the wrong instinct. Three better moves:

Ask for the shape you need. scrape returns markdown by default and strips navigation, ads, and footers via Readability. Asking for rawHtml on a modern site can be 10x the tokens for no gain.

Route bulk output to disk. For anything you will process rather than read, have Claude write it out:

Batch scrape these 20 URLs and write each result to data/<domain>.md.
Then give me a one-line summary of each — do not paste the bodies.

Claude keeps 20 summary lines in context instead of 20 articles.

Use the cheap tool for triage. extract_metadata (1 credit) answers "is this page worth reading" with a few hundred tokens. map_site (2) gives you a URL list without fetching a single body. Survey first, then fetch what matters.

Prompt Patterns That Work

The difference between a vague request and a specific one is usually two extra clauses.

Name the output shape.

Get me the pricing from that page.

Scrape https://example.com/pricing and return JSON: [{ plan, monthly_price, annual_price, included_credits }]. Use null for anything not listed.

Say what happens to the data. Claude defaults to printing results. If the point is a file, a diff, or a test, say so:

Search for the top 10 results on "MCP web scraping", scrape each one,
and write a comparison table to docs/competitors.md. Skip anything
that 403s and note it at the bottom.

Give it the escalation path up front. This saves a whole round trip:

Scrape https://app.example.com/dashboard. If the content looks
client-rendered, retry with scrape_with_actions waiting on
.data-grid. If you get a 403, use stealth_mode.

Chain into work Claude Code is already good at. The real advantage of scraping in your terminal is that the data lands next to your code:

Scrape the Stripe webhook events reference, then check
src/lib/stripe/webhooks.ts for event types we handle that
no longer appear in their docs.

That is one prompt covering a fetch, a repo read, and a diff — none of which a browser tab does for you.

Share the Setup With Your Team

MCP servers install at three scopes. The default is local: private to you, scoped to the current project, stored in ~/.claude.json.

For a team, use project scope, which writes .mcp.json in the repo root:

claude mcp add crawlforge \
  --scope project \
  --env CRAWLFORGE_API_KEY=cf_live_your_key_here \
  -- npx -y crawlforge-mcp-server

Do not commit that file with a live key in it. .mcp.json supports environment variable expansion, so commit the reference instead:

{
  "mcpServers": {
    "crawlforge": {
      "command": "npx",
      "args": ["-y", "crawlforge-mcp-server"],
      "env": {
        "CRAWLFORGE_API_KEY": "${CRAWLFORGE_API_KEY}"
      }
    }
  }
}

Now every developer gets the server on clone and supplies their own key from their shell. The ${VAR:-default} form works too, and expansion applies to command, args, env, url, and headers.

Two things to expect with project scope. Claude Code asks each developer to approve the server the first time — that is deliberate, since a repo can otherwise ship a server that runs on clone. And if the variable is unset with no default, the config still loads: claude mcp list reports a missing-variable warning and passes the literal ${CRAWLFORGE_API_KEY} text through, which shows up later as a 401.

Use user scope (--scope user) for a server you want in every project on your machine.

Run It Headless in Scripts and CI

Everything above works in non-interactive mode, with one difference: there is nobody to answer prompts, so permissions must be settled in advance.

claude -p "Scrape https://news.ycombinator.com and write the top 10 \
stories as JSON to hn.json" \
  --allowedTools "mcp__crawlforge__scrape,Write"

Project-scoped servers from .mcp.json load in claude -p without the approval prompt, since it cannot be shown. That makes checked-in .mcp.json the natural choice for CI. If you need to keep a server out of an automated run, disabledMcpjsonServers blocks it in every mode.

A daily monitor is then a cron job and one prompt:

claude -p "Use track_changes on https://competitor.com/pricing. \
If anything changed since the last run, append it to CHANGES.md." \
  --allowedTools "mcp__crawlforge__track_changes,Read,Write"

Troubleshooting

✘ Failed to connect — Run claude mcp get crawlforge and read the Issue: line, which carries the HTTP status or error text. For the stdio server, check that npx -y crawlforge-mcp-server runs on its own first.

401 from every tool — The API key is wrong, or it has invisible whitespace. Pasting a key often brings a trailing newline; Claude Code flags this in claude mcp list and /mcp with a warning that names the field. Re-add the server and check that the key starts with cf_live_.

⏸ Pending approval — A project-scoped server needs interactive approval. Run claude in the directory and accept. In a freshly cloned repo you also need to accept the workspace trust dialog first — a repo cannot approve its own servers. claude mcp reset-project-choices clears earlier answers.

Claude keeps using WebFetch — Go back to Step 3. Without a stated preference, the built-in is a defensible choice.

Tool output truncated — You hit the 25,000-token cap. Prefer narrowing the request over raising MAX_MCP_OUTPUT_TOKENS; see Protect Your Context Window.

Insufficient credits — Check the usage dashboard. Free accounts get 1,000 one-time credits; Hobby is $19/mo for 5,000/month.

Empty content from a page that loads in your browser — Client-side rendering. Retry with scrape_with_actions and a wait on a selector that only exists after hydration.

Next Steps

• New to the setup? Start with the installation guide
• Scraping-specific walkthrough: How to Scrape Websites with Claude Code
• Protocol background: MCP Protocol Explained for Developers
• Full tool reference: getting started docs

Start free with 1,000 credits at crawlforge.dev/signup. No credit card required.