Model Context Protocol (MCP)
Definition
The Model Context Protocol is an open standard that enables AI models to interact with external tools and data sources through a unified interface. It provides a structured way for LLMs to call functions, access APIs, and retrieve real-time information.
MCP is the foundation of CrawlForge's architecture. Instead of building custom integrations for each AI platform, CrawlForge exposes its 29 web scraping tools through the MCP standard. Any MCP-compatible AI client -- Claude, GPT, or custom agents -- can discover and use these tools immediately.
This means your AI agents gain web scraping capabilities by simply connecting to the CrawlForge MCP server, with no custom code or API wrappers needed. The protocol handles tool discovery, parameter validation, and response formatting automatically.
Related Terms: MCP Server, MCP Client, Tool Use, Function Calling
MCP Server
Definition
An MCP server is a service that exposes tools and resources through the Model Context Protocol. It registers available functions, handles incoming tool calls from AI clients, and returns structured results.
CrawlForge is an MCP server that provides 29 web scraping and data extraction tools. When an AI agent connects to CrawlForge, the server advertises its available tools -- fetch_url, extract_content, deep_research, and more -- along with their parameters and expected outputs.
The server handles all the complexity of web scraping (browser rendering, proxy management, rate limiting) while presenting a clean tool interface to the AI client. This separation of concerns lets AI developers focus on their application logic rather than scraping infrastructure.
Related Terms: Model Context Protocol (MCP), MCP Client, Tool Use, AI Agent
MCP Client
Definition
An MCP client is an application or AI model that connects to MCP servers to discover and invoke tools. It sends tool call requests and processes the structured responses returned by the server.
MCP clients include AI assistants like Claude, development environments like Cursor, and custom applications built with MCP SDKs. Any MCP client can connect to CrawlForge and immediately gain access to all 29 web scraping tools without additional configuration.
The client-server architecture means you can use CrawlForge from any MCP-compatible environment. Whether you are building a chatbot, an autonomous agent, or a data pipeline, the same tools are available through the same protocol.
Related Terms: Model Context Protocol (MCP), MCP Server, AI Agent, Function Calling
AI Agent
Definition
An AI agent is an autonomous system powered by a large language model that can reason about tasks, make decisions, and take actions by using tools. Agents go beyond simple chatbots by planning and executing multi-step workflows.
AI agents are the primary users of CrawlForge tools. An agent tasked with market research, for example, might use search_web to find relevant sources, extract_content to get clean data from each source, and deep_research to synthesize findings -- all without human intervention.
CrawlForge is designed specifically for agent workflows. Tools return structured data that agents can reason about, credit costs are predictable for budget planning, and the MCP interface means agents can discover and use tools dynamically.
Related Terms: Large Language Model (LLM), Tool Use, Function Calling, Model Context Protocol (MCP)
Large Language Model (LLM)
Definition
A large language model is a neural network trained on vast amounts of text data that can understand and generate human language. LLMs power AI assistants, code generators, and autonomous agents.
LLMs like Claude, GPT, and Llama are powerful reasoners but have a fundamental limitation: they cannot access the live web. Their knowledge is frozen at training time, making them unreliable for tasks requiring current information.
CrawlForge bridges this gap by giving LLMs real-time web access through MCP tools. When an LLM needs current data -- live prices, recent news, updated documentation -- it calls CrawlForge tools to fetch and extract that information, then reasons about the fresh data.
Related Terms: AI Agent, Token, Context Window, Retrieval-Augmented Generation (RAG)
Retrieval-Augmented Generation (RAG)
Definition
RAG is an AI architecture that combines information retrieval with text generation. It first retrieves relevant documents from external sources, then uses them as context for the language model to generate accurate, grounded responses.
RAG systems need high-quality source content to work well. Garbage in means garbage out -- if the retrieved documents are noisy HTML with navigation menus and ads, the generated answers suffer. Clean content extraction is a critical component of any RAG pipeline.
CrawlForge tools like extract_content and extract_text return clean, structured content stripped of boilerplate. This makes them ideal for building RAG pipelines that need to ingest web content. Pair them with deep_research for multi-source retrieval with built-in conflict detection.
Related Terms: Embeddings, Vector Database, Large Language Model (LLM), Context Window
Embeddings
Definition
Embeddings are dense numerical vector representations of text, images, or other data. They capture semantic meaning in a format that enables similarity search, clustering, and other machine learning operations.
Embeddings are the bridge between raw text and machine understanding. When you convert a web page's content into an embedding, you can compare it with other documents to find similar content, build recommendation systems, or power semantic search.
CrawlForge extract_content provides clean text that produces higher-quality embeddings. Raw HTML with navigation, footers, and ads creates noisy embeddings that degrade search quality. By extracting only the meaningful content, CrawlForge improves downstream embedding performance.
Related Terms: Vector Database, Retrieval-Augmented Generation (RAG), Large Language Model (LLM), Structured Output
Vector Database
Definition
A vector database is a specialized database designed to store and efficiently query high-dimensional vector embeddings. It enables fast similarity search across millions of embedded documents.
Vector databases like Pinecone, Weaviate, and pgvector are essential components of RAG systems and semantic search. They store document embeddings and retrieve the most relevant ones based on vector similarity when a query comes in.
CrawlForge integrates into vector database workflows as the content ingestion layer. Use batch_scrape to collect pages at scale, extract_content to get clean text, and then embed and store the results in your vector database. This pipeline keeps your knowledge base current with fresh web data.
Related Terms: Embeddings, Retrieval-Augmented Generation (RAG), Structured Data, Data Pipeline
Structured Output
Definition
Structured output refers to data returned in a predictable, machine-readable format like JSON, rather than free-form text. It enables reliable downstream processing by AI agents and data pipelines.
AI agents need structured data to make decisions. Unstructured text requires additional parsing and is prone to misinterpretation. Structured output ensures that each data field is in a known location and format, enabling reliable automated processing.
CrawlForge tools like scrape_structured and extract_structured return data in clean JSON format matching your specified schema. This means your agents and pipelines can consume the output directly without post-processing or regex extraction.
Related Terms: JSON, Schema Markup, Structured Data, Function Calling
Prompt Engineering
Definition
Prompt engineering is the practice of designing and refining instructions given to language models to achieve desired outputs. It involves crafting system prompts, few-shot examples, and structured queries.
Good prompt engineering determines whether an AI agent uses tools effectively. A well-designed prompt tells the agent when to use CrawlForge tools, which tool to choose for each task, and how to interpret the results.
When using CrawlForge with AI agents, prompt engineering helps optimize credit usage. For example, instructing an agent to use extract_text (1 credit) for simple content retrieval instead of deep_research (10 credits) when full research is unnecessary can significantly reduce costs.
Related Terms: Large Language Model (LLM), AI Agent, Context Window, Token
Fine-Tuning
Definition
Fine-tuning is the process of further training a pre-trained language model on a specific dataset to specialize its behavior for a particular task or domain. It adapts general-purpose models to targeted use cases.
Fine-tuning requires large, high-quality datasets of domain-specific text. Collecting this data from the web is one of the most common use cases for web scraping at scale. The quality of the training data directly impacts the fine-tuned model's performance.
CrawlForge batch_scrape and extract_content are designed for this workflow. Use batch_scrape to process hundreds of URLs in parallel, and extract_content to get clean, structured text suitable for training. This pipeline can build datasets from documentation sites, forums, academic papers, or any web source.
Related Terms: Large Language Model (LLM), Token, Embeddings, AI Agent
Token
Definition
A token is the basic unit of text that language models process. Text is split into tokens (roughly 4 characters or 0.75 words each) before being processed by the model. Token counts determine costs and context limits.
Understanding tokens is important when using CrawlForge with AI agents because the scraped content consumes context window space. A long web page might produce thousands of tokens, potentially filling the agent's context and increasing API costs.
CrawlForge tools like extract_text and summarize_content help manage token usage. extract_text returns only the main content without boilerplate, and summarize_content condenses long pages into concise summaries, reducing the token footprint sent to your LLM.
Related Terms: Context Window, Large Language Model (LLM), Prompt Engineering, Fine-Tuning
Context Window
Definition
The context window is the maximum amount of text (measured in tokens) that a language model can process in a single request. It includes both the input prompt and the generated output.
Context window size determines how much scraped content an AI agent can work with at once. Claude's 200K token context window can hold roughly 150,000 words, while smaller models may be limited to 4K-32K tokens. Exceeding the context window means data gets truncated or lost.
CrawlForge helps manage context window constraints through tools like summarize_content, which condenses long pages, and extract_text, which strips out boilerplate. For large-scale research, deep_research synthesizes multiple sources into a concise summary rather than dumping all raw content into context.
Related Terms: Token, Large Language Model (LLM), Prompt Engineering, Retrieval-Augmented Generation (RAG)
Function Calling
Definition
Function calling is the ability of language models to invoke external functions or APIs during a conversation. The model decides when to call a function, generates the appropriate arguments, and processes the returned results.
Function calling is the mechanism that enables AI agents to use CrawlForge tools. When an agent determines it needs web data, it generates a function call with the appropriate tool name and parameters. The MCP server executes the call and returns the results to the agent.
CrawlForge tools are designed to be easily discoverable through function calling interfaces. Each tool has clear parameter definitions and return types, making it straightforward for models to select the right tool and provide valid arguments.
Related Terms: Tool Use, Model Context Protocol (MCP), AI Agent, Structured Output
Tool Use
Definition
Tool use is the capability of AI models to interact with external tools, APIs, and services to accomplish tasks beyond text generation. It extends model capabilities to include web browsing, code execution, data retrieval, and more.
Tool use transforms language models from passive text generators into active problem-solvers. Instead of hallucinating answers about current events, a tool-using model can search the web, fetch pages, and extract data to provide grounded, accurate responses.
CrawlForge provides 29 specialized web scraping tools for AI models to use. Each tool serves a specific purpose -- from simple URL fetching to complex multi-source research -- giving models fine-grained control over how they access web data.
Related Terms: Function Calling, Model Context Protocol (MCP), AI Agent, MCP Server
Start Scraping with 1,000 Free Credits
Get started with CrawlForge today. No credit card required.
Start scraping with 1,000 free credits