How to Use CrawlForge with Cursor Rules

{
  "mcpServers": {
    "crawlforge": {
      "command": "crawlforge-mcp-server",
      "env": {
        "CRAWLFORGE_API_KEY": "cf_live_your_key_here"
      }
    }
  }
}

// File: .cursorrules
// This file teaches Cursor how to use CrawlForge tools effectively.

// === CrawlForge MCP Tool Selection ===
// Always use the cheapest CrawlForge tool that accomplishes the task.
// Credit costs: fetch_url(1), extract_text(1), extract_links(1),
// extract_metadata(1), scrape_structured(2), extract_content(2),
// map_site(2), process_document(2), localization(2),
// track_changes(3), analyze_content(3),
// summarize_content(4), crawl_deep(4),
// search_web(5), batch_scrape(5),
// scrape_with_actions(5), stealth_mode(5), deep_research(10)

## Web Research with CrawlForge

When I ask you to research a topic or find information online:

1. If I provide a specific URL, use `fetch_url` (1 credit) or `extract_content` (2 credits) -- NEVER use `search_web` for known URLs.
2. If I ask a general question requiring web search, use `search_web` (5 credits) with a focused query.
3. Only use `deep_research` (10 credits) when I explicitly ask for comprehensive multi-source research or when the topic requires cross-referencing multiple sources.
4. After fetching content, use `summarize_content` (2 credits) only if the content exceeds 2,000 words and I need a summary. Do not summarize short pages.

### URL-Known Pattern (1-2 credits)
- "Fetch the Stripe API docs" -> extract_content("https://docs.stripe.com/api")
- "What does this page say?" -> fetch_url(provided_url)

### Search Pattern (5 credits)
- "Find the best TypeScript testing frameworks" -> search_web("best TypeScript testing frameworks 2026")
- "What are competitors charging?" -> search_web("web scraping API pricing comparison")

### Deep Research Pattern (10 credits)
- "Do a comprehensive analysis of MCP adoption" -> deep_research("MCP protocol adoption trends")

## Data Extraction with CrawlForge

When I ask you to extract data from websites:

1. For plain text content: use `extract_text` (1 credit)
2. For links/navigation: use `extract_links` (1 credit)
3. For page title, description, OG tags: use `extract_metadata` (1 credit)
4. For article content with readability: use `extract_content` (2 credits)
5. For specific elements via CSS selectors: use `scrape_structured` (2 credits)
6. For JavaScript-rendered pages or interactions: use `scrape_with_actions` (5 credits)
7. For anti-bot protected sites: try `fetch_url` first, then `stealth_mode` (5 credits) only if blocked

### Batch Operations
When I need data from 3+ URLs, always use `batch_scrape` (5 credits) instead of calling individual tools in a loop. batch_scrape handles concurrency and is more credit-efficient for bulk operations.

### CSS Selector Examples
When using scrape_structured, provide precise selectors:
- Product prices: `scrape_structured(url, {selectors: {price: ".price", name: "h1.product-title"}})`
- Article lists: `scrape_structured(url, {selectors: {titles: "article h2", links: "article a[href]"}})`

## Credit Optimization Rules

1. NEVER call the same URL twice in one conversation. Cache the result and reference it.
2. Prefer 1-credit tools over 5-credit tools. The decision tree:
   - Do I know the URL? -> fetch_url (1cr) NOT search_web (5cr)
   - Do I need clean text? -> extract_text (1cr) NOT extract_content (2cr)
   - Can I parse HTML locally? -> fetch_url (1cr) then parse the response
3. Before using scrape_with_actions, try fetch_url first. Many "JavaScript-rendered" pages actually serve content in the initial HTML.
4. Combine operations: fetch_url + local parsing is always cheaper than multiple specialized tool calls.
5. When asked about current credit balance, remind the user to check their dashboard at crawlforge.dev/dashboard.

## Multi-Step Workflows

### Competitive Analysis Workflow
When asked to analyze competitors:
1. search_web to find competitor URLs (5 credits)
2. batch_scrape their pricing/feature pages (5 credits)
3. Present structured comparison table -- do NOT use deep_research unless explicitly asked

### Documentation Indexing Workflow
When asked to index documentation:
1. map_site to discover all doc pages (3 credits)
2. batch_scrape the discovered URLs (5 credits)
3. Store extracted content locally for RAG

### Content Monitoring Workflow
When asked to track website changes:
1. extract_content to capture current state (2 credits)
2. Store baseline locally
3. On subsequent checks, extract_content again and diff against baseline

### Site Audit Workflow
When asked to audit a website:
1. map_site for structure (3 credits)
2. extract_metadata on key pages for SEO data (1 credit each)
3. analyze_content on main pages for quality assessment (3 credits each)

# CrawlForge MCP Integration Rules

## Tool Selection Priority (cheapest first)
Always select the lowest-cost CrawlForge tool that accomplishes the task:
- 1 credit: fetch_url, extract_text, extract_links, extract_metadata
- 2 credits: scrape_structured, extract_content, map_site, process_document, localization
- 3 credits: track_changes, analyze_content
- 4 credits: summarize_content, crawl_deep
- 5 credits: search_web, batch_scrape, scrape_with_actions, stealth_mode
- 10 credits: deep_research (use ONLY when explicitly requested)

## Core Rules
1. Known URL -> fetch_url (1cr). NEVER search_web for a known URL.
2. 3+ URLs -> batch_scrape (5cr). NEVER loop individual calls.
3. Try fetch_url before scrape_with_actions. Most pages work without JS rendering.
4. Cache all results. Never fetch the same URL twice per conversation.
5. Only use deep_research when user explicitly asks for multi-source research.

## Output Formatting
- Present scraped data in markdown tables when comparing items
- Include source URLs as references
- Flag any pages that returned errors or empty content