AI and MCP Terms

MCP is the foundation of CrawlForge's architecture. Instead of building custom integrations for each AI platform, CrawlForge exposes its 31 web scraping tools through the MCP standard. Any MCP-compatible AI client -- Claude, GPT, or custom agents -- can discover and use these tools immediately.

This means your AI agents gain web scraping capabilities by simply connecting to the CrawlForge MCP server, with no custom code or API wrappers needed. The protocol handles tool discovery, parameter validation, and response formatting automatically.

CrawlForge is an MCP server that provides 31 web scraping and data extraction tools. When an AI agent connects to CrawlForge, the server advertises its available tools -- fetch_url, extract_content, deep_research, and more -- along with their parameters and expected outputs.

The server handles all the complexity of web scraping (browser rendering, proxy management, rate limiting) while presenting a clean tool interface to the AI client. This separation of concerns lets AI developers focus on their application logic rather than scraping infrastructure.

MCP clients include AI assistants like Claude, development environments like Cursor, and custom applications built with MCP SDKs. Any MCP client can connect to CrawlForge and immediately gain access to all 31 web scraping tools without additional configuration.

The client-server architecture means you can use CrawlForge from any MCP-compatible environment. Whether you are building a chatbot, an autonomous agent, or a data pipeline, the same tools are available through the same protocol.

AI agents are the primary users of CrawlForge tools. An agent tasked with market research, for example, might use search_web to find relevant sources, extract_content to get clean data from each source, and deep_research to synthesize findings -- all without human intervention.

CrawlForge is designed specifically for agent workflows. Tools return structured data that agents can reason about, credit costs are predictable for budget planning, and the MCP interface means agents can discover and use tools dynamically.

LLMs like Claude, GPT, and Llama are powerful reasoners but have a fundamental limitation: they cannot access the live web. Their knowledge is frozen at training time, making them unreliable for tasks requiring current information.

CrawlForge bridges this gap by giving LLMs real-time web access through MCP tools. When an LLM needs current data -- live prices, recent news, updated documentation -- it calls CrawlForge tools to fetch and extract that information, then reasons about the fresh data.

RAG systems need high-quality source content to work well. Garbage in means garbage out -- if the retrieved documents are noisy HTML with navigation menus and ads, the generated answers suffer. Clean content extraction is a critical component of any RAG pipeline.

CrawlForge tools like extract_content and extract_text return clean, structured content stripped of boilerplate. This makes them ideal for building RAG pipelines that need to ingest web content. Pair them with deep_research for multi-source retrieval with built-in conflict detection.

Embeddings are the bridge between raw text and machine understanding. When you convert a web page's content into an embedding, you can compare it with other documents to find similar content, build recommendation systems, or power semantic search.

CrawlForge extract_content provides clean text that produces higher-quality embeddings. Raw HTML with navigation, footers, and ads creates noisy embeddings that degrade search quality. By extracting only the meaningful content, CrawlForge improves downstream embedding performance.

Vector databases like Pinecone, Weaviate, and pgvector are essential components of RAG systems and semantic search. They store document embeddings and retrieve the most relevant ones based on vector similarity when a query comes in.

CrawlForge integrates into vector database workflows as the content ingestion layer. Use batch_scrape to collect pages at scale, extract_content to get clean text, and then embed and store the results in your vector database. This pipeline keeps your knowledge base current with fresh web data.

AI agents need structured data to make decisions. Unstructured text requires additional parsing and is prone to misinterpretation. Structured output ensures that each data field is in a known location and format, enabling reliable automated processing.

CrawlForge tools like scrape_structured and extract_structured return data in clean JSON format matching your specified schema. This means your agents and pipelines can consume the output directly without post-processing or regex extraction.

Good prompt engineering determines whether an AI agent uses tools effectively. A well-designed prompt tells the agent when to use CrawlForge tools, which tool to choose for each task, and how to interpret the results.

When using CrawlForge with AI agents, prompt engineering helps optimize credit usage. For example, instructing an agent to use extract_text (1 credit) for simple content retrieval instead of deep_research (10 credits) when full research is unnecessary can significantly reduce costs.

Fine-tuning requires large, high-quality datasets of domain-specific text. Collecting this data from the web is one of the most common use cases for web scraping at scale. The quality of the training data directly impacts the fine-tuned model's performance.

CrawlForge batch_scrape and extract_content are designed for this workflow. Use batch_scrape to process hundreds of URLs in parallel, and extract_content to get clean, structured text suitable for training. This pipeline can build datasets from documentation sites, forums, academic papers, or any web source.

Understanding tokens is important when using CrawlForge with AI agents because the scraped content consumes context window space. A long web page might produce thousands of tokens, potentially filling the agent's context and increasing API costs.

CrawlForge tools like extract_text and summarize_content help manage token usage. extract_text returns only the main content without boilerplate, and summarize_content condenses long pages into concise summaries, reducing the token footprint sent to your LLM.

Context window size determines how much scraped content an AI agent can work with at once. Claude's 200K token context window can hold roughly 150,000 words, while smaller models may be limited to 4K-32K tokens. Exceeding the context window means data gets truncated or lost.

CrawlForge helps manage context window constraints through tools like summarize_content, which condenses long pages, and extract_text, which strips out boilerplate. For large-scale research, deep_research synthesizes multiple sources into a concise summary rather than dumping all raw content into context.

Function calling is the mechanism that enables AI agents to use CrawlForge tools. When an agent determines it needs web data, it generates a function call with the appropriate tool name and parameters. The MCP server executes the call and returns the results to the agent.

CrawlForge tools are designed to be easily discoverable through function calling interfaces. Each tool has clear parameter definitions and return types, making it straightforward for models to select the right tool and provide valid arguments.

Tool use transforms language models from passive text generators into active problem-solvers. Instead of hallucinating answers about current events, a tool-using model can search the web, fetch pages, and extract data to provide grounded, accurate responses.

CrawlForge provides 31 specialized web scraping tools for AI models to use. Each tool serves a specific purpose -- from simple URL fetching to complex multi-source research -- giving models fine-grained control over how they access web data.

In the Model Context Protocol, a server publishes a list of tools. Each has a name, a description the model reads to decide when to use it, and a JSON Schema for its arguments. The client passes the list to the model; when the model asks for a tool, the client runs it on the server and returns the result to the conversation.

CrawlForge's MCP server exposes 31 tools, from fetch_url (1 credit) to deep_research (10 credits). Each does one job with clear inputs, which helps the model pick the cheapest tool that fits the task. Clients that cap how many tools they register can load a subset with the CRAWLFORGE_TOOL_GROUPS setting.

A language model only knows what was in its training data. A search API gives an agent a way to find current sources: it sends a query, gets back a ranked list it can reason over, then fetches the pages worth reading. Some providers run their own search index; others query an existing engine.

CrawlForge's search_web (5 credits per query) runs a live web search on the Brave Search API's independent index and returns titles, URLs and snippets as JSON, with site, file type, language and recency filters and up to 10 queries per call. CrawlForge does not keep an index of its own. For research that needs search, reading and synthesis together, deep_research (10 credits) does all three in one call.

Some tasks need more than reading a page: logging in, filling a form, moving through a multi-step flow or opening a menu that only appears after a click. A browser agent handles these by alternating between observing the page and acting on it, usually in a headless browser running on a server.

CrawlForge offers two levels of this. scrape_with_actions (5 credits) runs a fixed sequence of clicks, typing and waits on one page and then extracts the result. browser_session keeps one browser page open across several calls -- 3 credits to open, then 1 to 2 per step -- so an agent can take a snapshot, choose an element, act on it and decide its next step from what it sees.

Raw HTML wastes a model's context window: menus, scripts, styles and tracking code can make up most of a page's bytes while carrying none of its meaning. Converting a page to markdown keeps headings, lists, tables and links and drops the rest, so more useful content fits in the same number of tokens. Structured JSON goes further and returns only the fields a task needs.

CrawlForge's scrape (2 credits) returns markdown with boilerplate stripped, and extract_content (2 credits) pulls a page's main article text. For specific fields, extract_structured (3 credits) fills a JSON Schema, and extract_embedded_state (2 credits) returns the exact JSON many modern sites already ship inside their HTML.