On this page
Mastra is a TypeScript-first AI agent framework designed for building production-ready AI applications. CrawlForge gives those agents the ability to fetch, extract, and analyze live web data. Together, they let you build agents that can research topics, monitor competitors, and extract structured data from any website.
This guide shows you how to wire CrawlForge tools into Mastra agents with working TypeScript examples.
Table of Contents
- What Is Mastra?
- Prerequisites
- Step 1: Set Up Your Mastra Project
- Step 2: Create CrawlForge Tool Definitions
- Step 3: Build a Web Research Agent
- Step 4: Build a Data Extraction Workflow
- Step 5: Add Error Handling and Retries
- Credit Cost Reference
- Architecture Overview
- Next Steps
What Is Mastra?
Mastra is the modern TypeScript framework for AI-powered applications and agents. It provides primitives for agent creation, tool integration, workflows, and memory -- all with full type safety. Think of it as the Express.js of AI agents: minimal, composable, and production-oriented.
Mastra agents can use external tools through a standardized tool interface. CrawlForge tools map directly to this interface, giving your agents 18 web scraping capabilities without writing HTTP client code.
Prerequisites
- Node.js 18+ and TypeScript 5+
- A CrawlForge account with an API key (1,000 free credits)
- Basic familiarity with TypeScript and async/await
Step 1: Set Up Your Mastra Project
Create a new Mastra project and install dependencies:
Add your CrawlForge API key to .env:
Step 2: Create CrawlForge Tool Definitions
Create a tools file that wraps CrawlForge's API as Mastra-compatible tools:
Step 3: Build a Web Research Agent
Create an agent that can search the web and extract content for research tasks:
Run the agent:
Step 4: Build a Data Extraction Workflow
Mastra workflows let you chain tools into deterministic pipelines. Here is a competitive pricing monitor:
Step 5: Add Error Handling and Retries
Production agents need resilient error handling. Here is a pattern for CrawlForge tool calls:
Credit Cost Reference
| Credits | Tools | Mastra Use Case |
|---|---|---|
| 1 | fetch_url, extract_text, extract_links, extract_metadata | Quick data fetching in agent tools |
| 2 | scrape_structured, extract_content, summarize_content, generate_llms_txt | Workflow extraction steps |
| 3 | map_site, process_document, analyze_content, localization | Site audits, document processing |
| 5 | search_web, crawl_deep, batch_scrape, scrape_with_actions, stealth_mode | Research agents, bulk operations |
| 10 | deep_research | Comprehensive analysis agents |
Architecture Overview
| Component | Role |
|---|---|
| Mastra Agent | Orchestrates tool calls, maintains conversation context |
| Mastra Tools | Typed wrappers around CrawlForge API endpoints |
| Mastra Workflow | Deterministic multi-step pipelines for batch operations |
| CrawlForge API | Executes web scraping, returns structured data |
| Credit System | Tracks usage per API key, enforces limits |
The Mastra agent decides which CrawlForge tool to call based on the task. The tool wrapper handles HTTP communication, and CrawlForge executes the actual scraping. Credits are deducted atomically on each successful tool call.
Next Steps
- Mastra Quickstart Guide -- official Mastra documentation
- CrawlForge API Reference -- full endpoint documentation
- Build a Research Assistant -- similar pattern using Claude directly
- Deep Research Automation -- advanced research workflows
Build your first web-aware AI agent today. Sign up for CrawlForge (1,000 free credits), scaffold a Mastra project, and give your agents the power to scrape the entire web.
