On this page
LangGraph is LangChain's framework for building stateful, graph-based AI agents. By integrating CrawlForge tools as graph nodes, you can build agents that make intelligent decisions about what to scrape, when to dig deeper, and how to synthesize web data across multiple steps.
This guide shows you how to build a complete scraping agent with LangGraph and CrawlForge in TypeScript.
Table of Contents
- What Is LangGraph?
- Prerequisites
- Step 1: Project Setup
- Step 2: Define CrawlForge Tools for LangGraph
- Step 3: Design the Agent State
- Step 4: Build Graph Nodes
- Step 5: Wire the Graph Together
- Step 6: Run the Agent
- Credit Cost Reference
- LangGraph vs Direct LangChain for Scraping
- Next Steps
What Is LangGraph?
LangGraph is a low-level orchestration framework for building reliable AI agents. Unlike simple chain-based architectures, LangGraph models agent logic as a directed graph where:
- Nodes represent actions (tool calls, LLM invocations, data processing)
- Edges define transitions between nodes, including conditional routing
- State persists across the entire graph execution
This architecture is ideal for scraping agents because web scraping inherently involves decisions: Should I scrape deeper? Is this page blocked? Do I need to switch to stealth mode? LangGraph lets you model these decisions as conditional edges in a graph.
Prerequisites
- Node.js 18+ and TypeScript 5+
- A CrawlForge account with an API key (1,000 free credits)
- Familiarity with LangChain basics
Step 1: Project Setup
Create tsconfig.json:
Add your API keys to .env:
Step 2: Define CrawlForge Tools for LangGraph
Create typed tool wrappers that LangGraph can invoke:
Step 3: Design the Agent State
LangGraph agents maintain state across graph execution. Define a state shape that tracks scraping progress:
Step 4: Build Graph Nodes
Each node in the graph performs a specific action and updates state:
Step 5: Wire the Graph Together
Connect nodes with edges and conditional routing:
Step 6: Run the Agent
Run it:
The agent will search the web, discover relevant pages, extract content from the most promising results, and synthesize a comparison -- all while tracking credit usage in the graph state.
Credit Cost Reference
| Credits | Tools | LangGraph Node Role |
|---|---|---|
| 1 | fetch_url, extract_text, extract_links, extract_metadata | Lightweight data-gathering nodes |
| 2 | scrape_structured, extract_content, summarize_content, generate_llms_txt | Extraction and analysis nodes |
| 3 | map_site, process_document, analyze_content, localization | Discovery and processing nodes |
| 5 | search_web, crawl_deep, batch_scrape, scrape_with_actions, stealth_mode | Research and bulk-operation nodes |
| 10 | deep_research | Comprehensive analysis (use as a single-node subgraph) |
Typical LangGraph agent run: 5 (search) + 6 (3 extractions) + 0 (LLM analysis) = 11 credits.
LangGraph vs Direct LangChain for Scraping
| Aspect | LangGraph | Direct LangChain |
|---|---|---|
| State Management | Built-in, typed, persistent | Manual, requires custom code |
| Conditional Logic | First-class conditional edges | If/else in chain functions |
| Credit Tracking | Track in graph state automatically | Manual counter |
| Error Recovery | Route errors to fallback nodes | Try/catch in chain |
| Complexity | Higher initial setup | Simpler for linear workflows |
| Best For | Multi-step research with branching logic | Simple fetch-and-process pipelines |
Use LangGraph when your scraping agent needs to make decisions based on intermediate results. Use direct LangChain (see our LangChain integration guide) when the workflow is linear.
Next Steps
- LangGraph Documentation -- official LangGraph guides
- 5 Ways to Use CrawlForge with LangChain -- simpler LangChain patterns
- Build a Research Assistant -- related agent architecture
- CrawlForge API Reference -- full tool endpoint documentation
Build intelligent scraping agents today. Sign up for CrawlForge with 1,000 free credits, wire the tools into your LangGraph graph, and let your agent decide what to scrape next.
