The Problem
Training and fine-tuning AI models requires large, clean datasets from diverse web sources. Collecting this data manually is impractical, and raw HTML is too noisy for model training.
Use Cases
Collect and structure large-scale web datasets for fine-tuning and training AI models. Crawl entire sites, extract clean text, and export training-ready data.
Estimated cost: ~7 credits per document
01Quick Answer
Use CrawlForge batch_scrape (5 credits) to fetch hundreds of URLs in parallel, then extract_content (2 credits) to return clean, boilerplate-free text or markdown ready for a training pipeline. You collect structured content instead of noisy raw HTML, which improves dataset quality and lowers preprocessing cost -- about 7 credits per document.
02The brief
The Solution
CrawlForge batch_scrape processes hundreds of URLs in parallel for scale, while extract_content returns clean, structured text ready for training pipelines. Build datasets from any web source.
03In code
Code Example
ai-training-data.js
// Collect training data from documentation sitesconst batch = await mcp.batch_scrape({ urls: [ "https://docs.example.com/guide/intro", "https://docs.example.com/guide/setup", "https://docs.example.com/guide/advanced", // ... hundreds more URLs ], format: "markdown",}); // Extract clean content for each pageconst dataset = await Promise.all( batch.results.map(page => mcp.extract_content({ url: page.url, format: "text", remove_navigation: true, }) )); console.log(`Collected ${dataset.length} documents`);04The pipeline
Tools Used
7◆ credits
Estimated cost: ~7 credits per document
- https://…
- 01batch_
scrape 5 credits - 02extract_
content 2 credits - JSON · markdown
05Questions
Frequently Asked Questions
01How do I collect clean training data from the web at scale?
Use CrawlForge batch_scrape to fetch hundreds of URLs in parallel, then extract_content to return clean, boilerplate-free text ready for a training pipeline. You get structured content instead of noisy raw HTML.
02Why not just use raw HTML for model training?
Raw HTML is full of navigation, ads, and markup that adds noise and wastes tokens. extract_content uses a readability pass to return only the main content as clean text or markdown, which improves dataset quality and lowers preprocessing cost.
03Can I build a large dataset from many sources?
Yes. batch_scrape at 5 credits per batch parallelizes fetching across hundreds of URLs, and extract_content at 2 credits cleans each one. Combine with map_site to enumerate a source first, then batch the resulting URLs.
04Does CrawlForge respect robots.txt when collecting data?
CrawlForge honors robots directives and you control which sources you crawl. You are responsible for the rights to data you collect, so target sites you are permitted to use for training and keep your crawl scope deliberate.
06Keep exploring
Related Use Cases
AI Agent Data Pipelines
Feed your AI agents live web data with structured extraction and multi-source research. Build pipelines with 31 MCP tools, from fetch_url to deep_research.
deep_research 10 creditsextract_content 2 credits
Content Migration
Extract and restructure content from legacy sites for migration to modern platforms. Keeps formatting, media, and internal links intact as clean Markdown.
crawl_deep 4 creditsextract_text 1 credit
Start forging
Ready to Get Started?
Every new account gets 1,000 free credits. No credit card required.