LangChain
LangChain Integration
Integrate CrawlForge MCP with LangChain to build powerful AI agents with web scraping capabilities. Use as a document loader, tool, or custom retrieval chain.
Use Cases
Document Loaders
Load web pages as documents for vector stores and RAG applications
AI Agents
Give agents web scraping tools to fetch real-time data
Retrieval Chains
Build custom chains that fetch and process web content
Research Pipelines
Create automated research workflows with deep_research tool
Installation
Install LangChain and the CrawlForge MCP adapter.
Bash
You'll also need a CrawlForge API key from the dashboard.
Document Loader
Use CrawlForge as a document loader to fetch web pages for RAG applications.
Typescript
Best Practice: Use
extract_text for clean content or extract_content for article extraction.RAG Pipeline with Vector Store
Build a complete RAG pipeline with CrawlForge document loader and vector store.
Typescript
Agent Tools
Give LangChain agents web scraping capabilities with CrawlForge tools.
Typescript
Agent Tips: Use descriptive tool names and descriptions to help the LLM choose the right tool. Set
verbose=true to see agent reasoning.Custom Retrieval Chain
Build a custom chain that searches, fetches, and summarizes web content.
Typescript
Best Practices
- Choose the Right Tool — Use
extract_text(1 credit) for simple content,deep_research(10 credits) for comprehensive analysis - Implement Caching — Cache fetched documents to avoid redundant API calls and save credits
- Handle Rate Limits — Implement retry logic with exponential backoff for production applications
- Monitor Credit Usage — Check document metadata for credit usage and set up alerts in your dashboard
Ready to build with LangChain?
Explore all 23 CrawlForge tools or check out other integrations.