LangChain Integration
Integrate CrawlForge MCP with LangChain to build powerful AI agents with web scraping capabilities. Use as a document loader, tool, or custom retrieval chain.
Use Cases
Load web pages as documents for vector stores and RAG applications
Give agents web scraping tools to fetch real-time data
Build custom chains that fetch and process web content
Create automated research workflows with deep_research tool
Installation
Install LangChain and the CrawlForge MCP adapter.
Document Loader
Use CrawlForge as a document loader to fetch web pages for RAG applications.
extract_text for clean content or extract_content for article extraction.RAG Pipeline with Vector Store
Build a complete RAG pipeline with CrawlForge document loader and vector store.
Agent Tools
Give LangChain agents web scraping capabilities with CrawlForge tools.
verbose=true to see agent reasoning.Custom Retrieval Chain
Build a custom chain that searches, fetches, and summarizes web content.
Best Practices
Choose the Right Tool
Use extract_text (1 credit) for simple content, deep_research (10 credits) for comprehensive analysis
Implement Caching
Cache fetched documents to avoid redundant API calls and save credits
Handle Rate Limits
Implement retry logic with exponential backoff for production applications
Monitor Credit Usage
Check document metadata for credit usage and set up alerts in your dashboard