The Problem
A RAG system is only as good as the text you feed it. Raw HTML carries navigation, ads, and markup that inflate token counts and pollute embeddings, and every documentation site is laid out differently, so one parser never fits them all.
Use Cases
Turn documentation sites into clean, chunk-ready markdown for retrieval-augmented generation. Crawl, strip the boilerplate, and summarize before you embed.
Estimated cost: ~6 credits per document
01Quick Answer
Use map_site (2 credits) to enumerate a docs site, scrape (2 credits) to return markdown with navigation and ads stripped, and summarize_content (4 credits) to add a short summary to each chunk. That is roughly 6 credits per document, and the markdown goes straight into your chunker and embedding model.
02The brief
The Solution
CrawlForge scrape returns markdown, metadata, and links from a single call with boilerplate stripped by Readability, and summarize_content condenses long pages into short summaries you can store as chunk metadata for better retrieval.
03In code
Code Example
rag-knowledge-base.js
// Turn a documentation site into RAG-ready markdownconst pages = await mcp.map_site({ url: "https://docs.example.com", include_sitemap: true, max_urls: 200,}); // One call returns markdown, metadata, and links per pageconst doc = await mcp.scrape({ url: pages.urls[0], formats: ["markdown", "metadata", "links"], onlyMainContent: true,}); // Condense long pages into chunk-level summariesconst digest = await mcp.summarize_content({ text: doc.markdown, options: { summaryLength: "short", summaryType: "abstractive" },}); console.log(doc.markdown); // chunk + embed thisconsole.log(digest.summary); // store as chunk metadata04The pipeline
Tools Used
8◆ credits
Estimated cost: ~6 credits per document
- https://…
- 01map_
site 2 credits - 02scrape 2 credits
- 03summarize_
content 4 credits - JSON · markdown
05Questions
Frequently Asked Questions
01How do I scrape a website for a RAG pipeline?
Use map_site to list every URL, scrape to return each page as markdown with navigation and ads stripped, and summarize_content to attach a short summary to each chunk. The markdown goes straight into your chunker and embedding model.
02Why is markdown better than raw HTML for RAG?
Raw HTML spends tokens on markup, menus, and ads, and that noise ends up in your embeddings. scrape runs Readability first and returns only the main content, so chunks stay semantically clean and cost less to embed.
03Can I keep the knowledge base up to date?
Yes. Re-run map_site and scrape on a schedule, and pair them with track_changes to spot the pages that actually changed, so you re-embed only those instead of rebuilding the whole index.
04How much does it cost to ingest a documentation site?
About 6 credits per document — 2 for scrape and 4 for summarize_content — plus 2 for one map_site call per site. A 200-page documentation site is roughly 1,200 credits, and skipping the summaries halves that.
06Keep exploring
Related Use Cases
AI Agent Data Pipelines
Feed your AI agents live web data with structured extraction and multi-source research. Build pipelines with 31 MCP tools, from fetch_url to deep_research.
deep_research 10 creditsextract_content 2 credits
AI Training Data Collection
Collect and structure large-scale web datasets for fine-tuning and training AI models. Crawl entire sites, extract clean text, and export training-ready data.
batch_scrape 5 creditsextract_content 2 credits
Start forging
Ready to Get Started?
Every new account gets 1,000 free credits. No credit card required.