How to Scrape Websites in Cursor IDE with CrawlForge MCP

Cursor IDE is great at reasoning about your code but cannot see the live web. Add CrawlForge via Cursor's Model Context Protocol integration and the Composer gains 31 scraping tools -- no Python script, no curl, no leaving the editor.

// ~/.cursor/mcp.json
{
  "mcpServers": {
    "crawlforge": {
      "command": "crawlforge-mcp-server",
      "env": { "CRAWLFORGE_API_KEY": "cf_live_your_key" }
    }
  }
}

This guide walks through setting up web scraping in Cursor IDE step by step, with runnable examples for research, structured extraction, and competitor monitoring.

Table of Contents

• Why Scrape from Inside Cursor?
• Prerequisites
• Step 1: Install the MCP Server
• Step 2: Configure Cursor's MCP Settings
• Step 3: Restart and Verify
• Step 4: Your First Scrape in Composer
• Full Example: Build a Competitor Price Tracker
• Workflow: Use Scrapes to Write Code
• Troubleshooting
• FAQ

Why Scrape from Inside Cursor?

Cursor Composer treats MCP tools as first-class actions: it picks the right tool for a task, passes typed arguments, and feeds results back into the conversation. When you scrape websites in Cursor IDE through CrawlForge, the extracted data is immediately available for Cursor to generate tests, write TypeScript interfaces, or update dashboards. No copy-paste, no context switch.

If you already use Cursor rules to shape Composer's behavior, MCP tools slot right in -- rules describe how to code, tools expose what Composer can do.

Prerequisites

• Cursor IDE 0.42+ -- download from cursor.com
• Node.js 18+ -- node --version to check
• CrawlForge account -- free at crawlforge.dev/signup

Step 1: Install the MCP Server

npm install -g crawlforge-mcp-server

Confirm it is on your PATH:

which crawlforge-mcp-server
# /usr/local/bin/crawlforge-mcp-server

Step 2: Configure Cursor's MCP Settings

Cursor reads MCP servers from ~/.cursor/mcp.json. Create it if it does not exist:

mkdir -p ~/.cursor
touch ~/.cursor/mcp.json

Paste this config (replace the key):

{
  "mcpServers": {
    "crawlforge": {
      "command": "crawlforge-mcp-server",
      "env": {
        "CRAWLFORGE_API_KEY": "cf_live_your_key_here"
      }
    }
  }
}

On Windows the file lives at %USERPROFILE%\.cursor\mcp.json and the command should be crawlforge-mcp-server.cmd.

Step 3: Restart and Verify

1. Quit Cursor completely (Cmd+Q on macOS).
2. Reopen the project.
3. Go to Settings -> Features -> MCP. You should see crawlforge with a green dot and 31 tools listed.

If the server is red or the tools list is empty, skip to Troubleshooting.

Step 4: Your First Scrape in Composer

Open Composer (Cmd+I) and paste:

Cursor will call fetch_url (1 credit) and display the response. When you approve the tool call, Composer parses the HTML and returns a clean list.

Full Example: Build a Competitor Price Tracker

Say you want to track pricing changes on a competitor SaaS. Open Composer and paste:

Cursor issues a scrape_structured call with your selectors, returns JSON, and emits this TypeScript in the next editor chunk:

// types/competitor-pricing.ts
export interface CompetitorPricingTier {
  plan: string;
  price: string;
  features: string[];
}

export interface CompetitorPricingResponse {
  data: CompetitorPricingTier[];
  scraped_at: string;
}

// lib/fetch-competitor-pricing.ts
import { CompetitorPricingResponse } from '@/types/competitor-pricing';

export async function fetchCompetitorPricing(): Promise<CompetitorPricingResponse> {
  const response = await fetch('https://crawlforge.dev/api/v1/tools/scrape_structured', {
    method: 'POST',
    headers: {
      'Authorization': `Bearer ${process.env.CRAWLFORGE_API_KEY}`,
      'Content-Type': 'application/json',
    },
    body: JSON.stringify({
      url: 'https://competitor.example.com/pricing',
      selectors: {
        plan: '.pricing-card h3',
        price: '.pricing-card .price',
        features: '.pricing-card ul li',
      },
    }),
  });

  if (!response.ok) {
    throw new Error(`Scrape failed: ${response.statusText}`);
  }

  return response.json();
}

Total cost: 2 credits per run. Schedule it via Vercel Cron or GitHub Actions and you have a free-tier price tracker.

Workflow: Use Scrapes to Write Code

The real unlock is feeding scraped data into Cursor's code generation. Proven patterns:

• Type generation from live APIs: "Fetch https://api.example.com/users, then generate a Zod schema matching the response."
• Test fixtures from real pages: "Scrape the top 3 articles from Hacker News and save them as JSON fixtures in tests/fixtures/."
• Documentation extraction: "Use extract_content on the React docs for useState, then write an idiomatic example that matches."
• Competitor feature parity: "Use map_site on competitor.com and flag any URL patterns we do not have in our own sitemap."

Each pattern is 1-5 credits per run and keeps you inside Cursor.

Credit Costs Summary

Troubleshooting

Tools list empty in Cursor Settings -- Cursor caches MCP config. Fully quit (Cmd+Q), then reopen. Check ~/.cursor/logs/ for parse errors.

"Command not found: crawlforge-mcp-server" -- npm's global bin is not on Cursor's PATH. Fix by setting an absolute path in mcp.json: "command": "/usr/local/bin/crawlforge-mcp-server".

Every call returns 401 -- API key missing or still the placeholder. Verify with: curl -H "Authorization: Bearer $CRAWLFORGE_API_KEY" https://crawlforge.dev/api/v1/credits/balance.

Cursor asks for approval on every tool call -- That is expected default behavior. Enable "Auto-approve for trusted servers" in MCP settings if you want Composer to run scrapes silently.

Composer ignores the MCP tool -- Explicitly prompt: "Use CrawlForge's scrape_structured tool to...". Cursor sometimes defaults to its built-in web fetch, which is less capable.

Next Steps

• Read the Cursor rules guide to optimize Composer behavior for scraping
• Browse the 31-tools overview to see what else you can automate
• Check the getting started docs for API reference and credit pricing
• Compare vendors at Firecrawl alternative

Start free with 1,000 credits at crawlforge.dev/signup.