RAG Knowledge Base Ingestion
Turn documentation sites into clean, chunk-ready markdown for retrieval-augmented generation. Crawl, strip the boilerplate, and summarize before you embed.
Quick Answer
Use map_site (2 credits) to enumerate a docs site, scrape (2 credits) to return markdown with navigation and ads stripped, and summarize_content (4 credits) to add a short summary to each chunk. That is roughly 6 credits per document, and the markdown goes straight into your chunker and embedding model.
The Problem
A RAG system is only as good as the text you feed it. Raw HTML carries navigation, ads, and markup that inflate token counts and pollute embeddings, and every documentation site is laid out differently, so one parser never fits them all.
The Solution
CrawlForge scrape returns markdown, metadata, and links from a single call with boilerplate stripped by Readability, and summarize_content condenses long pages into short summaries you can store as chunk metadata for better retrieval.
Code Example
// Turn a documentation site into RAG-ready markdown
const pages = await mcp.map_site({
url: "https://docs.example.com",
include_sitemap: true,
max_urls: 200,
});
// One call returns markdown, metadata, and links per page
const doc = await mcp.scrape({
url: pages.urls[0],
formats: ["markdown", "metadata", "links"],
onlyMainContent: true,
});
// Condense long pages into chunk-level summaries
const digest = await mcp.summarize_content({
text: doc.markdown,
options: { summaryLength: "short", summaryType: "abstractive" },
});
console.log(doc.markdown); // chunk + embed this
console.log(digest.summary); // store as chunk metadataTools Used
Estimated cost: ~6 credits per document
Frequently Asked Questions
How do I scrape a website for a RAG pipeline?
Use map_site to list every URL, scrape to return each page as markdown with navigation and ads stripped, and summarize_content to attach a short summary to each chunk. The markdown goes straight into your chunker and embedding model.
Why is markdown better than raw HTML for RAG?
Raw HTML spends tokens on markup, menus, and ads, and that noise ends up in your embeddings. scrape runs Readability first and returns only the main content, so chunks stay semantically clean and cost less to embed.
Can I keep the knowledge base up to date?
Yes. Re-run map_site and scrape on a schedule, and pair them with track_changes to spot the pages that actually changed, so you re-embed only those instead of rebuilding the whole index.
How much does it cost to ingest a documentation site?
About 6 credits per document — 2 for scrape and 4 for summarize_content — plus 2 for one map_site call per site. A 200-page documentation site is roughly 1,200 credits, and skipping the summaries halves that.
Ready to Get Started?
Every new account gets 1,000 free credits. No credit card required.
Start Free with 1,000 Credits