Protocol

MCP Protocol

Complete Model Context Protocol specification for CrawlForge MCP. Learn how to implement custom MCP clients and integrate with any AI framework.

What is MCP?

The Model Context Protocol (MCP) is an open standard created by Anthropic that enables AI assistants to securely access external data sources and tools. It defines a standardized way for AI models to discover, invoke, and receive results from external services.

JSON-RPC 2.0

Built on standard JSON-RPC for universal compatibility

Secure

API key authentication and rate limiting built-in

Extensible

Add custom tools without changing the protocol

Server Configuration

Configure your MCP client to connect to the CrawlForge MCP server.

Json

{
  "mcpServers": {
    "crawlforge": {
      "command": "crawlforge-mcp",
      "env": {
        "CRAWLFORGE_API_KEY": "cf_live_YOUR_KEY_HERE"
      }
    }
  }
}

crawlforge-mcp ships with the global crawlforge-mcp-server install. With no global install, use "command": "npx" and "args": ["-y", "crawlforge-mcp-server"]. If you ran crawlforge-setup, the env block is optional—the key is read from ~/.crawlforge/config.json.

Get your API key from the dashboard.

Tool Discovery

MCP clients discover available tools by calling the tools/list method.

Json

{
  "jsonrpc": "2.0",
  "id": "1",
  "method": "tools/list",
  "params": {}
}

Tool Manifest Fields

Name	Type	Required	Description
`name`	string	Required	Tool identifier (e.g., "fetch_url") · Example: fetch_url
`description`	string	Required	Human-readable tool description · Example: Fetch and parse HTML content from a URL
`inputSchema`	object	Required	JSON Schema defining tool parameters · Example: { "type": "object", "properties": {...} }

Tool Invocation

Invoke tools using the tools/call method with tool name and arguments.

Typescript

import { Client } from '@modelcontextprotocol/sdk/client/index.js';
import { StdioClientTransport } from '@modelcontextprotocol/sdk/client/stdio.js';

const transport = new StdioClientTransport({
  command: 'crawlforge-mcp',
  env: {
    CRAWLFORGE_API_KEY: process.env.CRAWLFORGE_API_KEY!
  }
});

const client = new Client({
  name: 'my-ai-app',
  version: '1.0.0'
}, {
  capabilities: {}
});

await client.connect(transport);

const result = await client.callTool({
  name: 'fetch_url',
  arguments: {
    url: 'https://example.com',
    timeout: 10000
  }
});

console.log(result);

Async Execution: All tool calls are asynchronous. Use await in JavaScript or async/await in Python to wait for results.

Response Format

Tool responses follow the standard MCP format with content, metadata, and error handling.

Json

{
  "jsonrpc": "2.0",
  "id": "2",
  "result": {
    "content": [
      {
        "type": "text",
        "text": "<!DOCTYPE html>\n<html>..."
      }
    ],
    "isError": false,
    "_meta": {
      "credits_used": 1,
      "credits_remaining": 999,
      "processing_time": 245
    }
  }
}

MCP Message Structure

Standard Message Fields

Name	Type	Required	Description
`method`	string	Required	MCP method name (e.g., "tools/call") · Example: tools/call
`params`	object	Required	Method-specific parameters · Example: { "name": "fetch_url", "arguments": {...} }
`id`	string	Required	Unique request identifier for correlation · Example: req_12345

Error Codes

Code	Name	Description
`-32700`	Parse Error	Invalid JSON
`-32600`	Invalid Request	Malformed request object
`-32601`	Method Not Found	Unknown MCP method
`-32602`	Invalid Params	Missing or invalid parameters
`-32603`	Internal Error	Server error during execution
`-32001`	Insufficient Credits	Not enough credits to execute tool

Best Practices

Use Unique Request IDs — Generate unique IDs for each request to correlate responses
Handle Errors Gracefully — Check for both JSON-RPC errors and tool-specific errors
Implement Timeouts — Set reasonable timeouts for tool calls (10-30 seconds)
Monitor Credit Usage — Track _meta.credits_remaining in responses

Troubleshooting

Connection Issues? Ensure your API key is valid and the server path is correct. Check environment variables and network connectivity.

For detailed troubleshooting, see the Claude Desktop guide or contact support.

MCP Resources & Prompts

Beyond tools, CrawlForge exposes MCP Resources (URI-addressable data) and Prompts (pre-built workflows). Both are available when using the server with any MCP-compatible client.

Resources

Resources live under the crawlforge:// URI scheme and address long-lived artifacts produced by earlier tool calls—research sessions, batch jobs, crawl sitemaps, screenshots, and page snapshots. The five registered URI templates are:

crawlforge://research/{sessionId}
crawlforge://job/{jobId}
crawlforge://crawl/{sessionId}/sitemap
crawlforge://screenshot/{actionId}
crawlforge://snapshot/{urlHash}/{timestamp}

Resources are read-only, TTL-bound, and do not consume credits.

Json

{
  "jsonrpc": "2.0",
  "id": "3",
  "method": "resources/read",
  "params": {
    "uri": "crawlforge://research/ses_a1b2c3"
  }
}

Prompts

Prompts are pre-built workflows the model can discover and invoke. CrawlForge ships five out of the box:

Prompt	Purpose
`competitive-analysis`	Side-by-side scrape and compare of competitor sites
`monitor-changes`	Set up a change-tracking workflow for a target URL
`rag-ingest`	Crawl a domain and emit clean, chunked content for RAG pipelines
`site-audit`	Full SEO + accessibility + content audit of a site
`research-deep-dive`	Multi-stage deep research with citations

Example: how Claude Desktop discovers and invokes a prompt.

Json

// 1. Client lists available prompts
{ "jsonrpc": "2.0", "id": "4", "method": "prompts/list", "params": {} }

// 2. Client invokes a prompt with arguments
{
  "jsonrpc": "2.0",
  "id": "5",
  "method": "prompts/get",
  "params": {
    "name": "competitive-analysis",
    "arguments": {
      "competitors": ["https://stripe.com", "https://paddle.com"]
    }
  }
}

Ready to integrate?

Choose your integration method and start building.

Claude Desktop Setup LangChain Integration

Name

Type

Required

Description

name

string

Required

Tool identifier (e.g., "fetch_url") · Example: fetch_url

description

string

Required

Human-readable tool description · Example: Fetch and parse HTML content from a URL

inputSchema

object

Required

JSON Schema defining tool parameters · Example: { "type": "object", "properties": {...} }

import { Client } from '@modelcontextprotocol/sdk/client/index.js'; import { StdioClientTransport } from '@modelcontextprotocol/sdk/client/stdio.js'; const transport = new StdioClientTransport({ command: 'crawlforge-mcp', env: { CRAWLFORGE_API_KEY: process.env.CRAWLFORGE_API_KEY! } }); const client = new Client({ name: 'my-ai-app', version: '1.0.0' }, { capabilities: {} }); await client.connect(transport); const result = await client.callTool({ name: 'fetch_url', arguments: { url: 'https://example.com', timeout: 10000 } }); console.log(result);

{ "jsonrpc": "2.0", "id": "2", "result": { "content": [ { "type": "text", "text": "<!DOCTYPE html>\n<html>..." } ], "isError": false, "_meta": { "credits_used": 1, "credits_remaining": 999, "processing_time": 245 } } }

Name

Type

Required

Description

method

string

Required

MCP method name (e.g., "tools/call") · Example: tools/call

params

object

Required

Method-specific parameters · Example: { "name": "fetch_url", "arguments": {...} }

id

string

Required

Unique request identifier for correlation · Example: req_12345

Code

Name

Description

-32700

Parse Error

Invalid JSON

-32600

Invalid Request

Malformed request object

-32601

Method Not Found

Unknown MCP method

-32602

Invalid Params

Missing or invalid parameters

-32603

Internal Error

Server error during execution

-32001

Insufficient Credits

Not enough credits to execute tool

Prompt

Purpose

competitive-analysis

Side-by-side scrape and compare of competitor sites

monitor-changes

Set up a change-tracking workflow for a target URL

rag-ingest

Crawl a domain and emit clean, chunked content for RAG pipelines

site-audit

Full SEO + accessibility + content audit of a site

research-deep-dive

Multi-stage deep research with citations

// 1. Client lists available prompts { "jsonrpc": "2.0", "id": "4", "method": "prompts/list", "params": {} } // 2. Client invokes a prompt with arguments { "jsonrpc": "2.0", "id": "5", "method": "prompts/get", "params": { "name": "competitive-analysis", "arguments": { "competitors": ["https://stripe.com", "https://paddle.com"] } } }