AI & LLMintermediate

Documentation Knowledge Base

Crawl documentation sites and build a structured knowledge base for AI-powered search.

MCP Configuration

{
  "tools": [
    {
      "name": "crawl_deep",
      "params": {
        "url": "https://docs.example.com",
        "max_depth": 3,
        "follow_links": true
      }
    },
    {
      "name": "extract_content",
      "params": {
        "format": "markdown",
        "remove_navigation": true
      }
    },
    {
      "name": "generate_llms_txt",
      "params": {
        "url": "https://docs.example.com"
      }
    }
  ]
}

How It Works

crawl_deep

5 credits

extract_content

2 credits

generate_llms_txt

2 credits

Estimated total: ~9 credits per run

Ready to Use This Template?

Every new account gets 1,000 free credits. No credit card required.

Start Free with 1,000 Credits

Related Templates

AI & LLMintermediate

AI Training Data Collector

Collect and clean large-scale web datasets for fine-tuning and training AI models.

batch_scrape (5 cr)extract_content (2 cr)

AI & LLMbeginner

LLMs.txt Generator

Generate an llms.txt file for your website to help LLMs understand your content structure.

generate_llms_txt (2 cr)map_site (3 cr)

AI & LLMintermediate

Documentation Knowledge Base

Crawl documentation sites and build a structured knowledge base for AI-powered search.

MCP Configuration

{
  "tools": [
    {
      "name": "crawl_deep",
      "params": {
        "url": "https://docs.example.com",
        "max_depth": 3,
        "follow_links": true
      }
    },
    {
      "name": "extract_content",
      "params": {
        "format": "markdown",
        "remove_navigation": true
      }
    },
    {
      "name": "generate_llms_txt",
      "params": {
        "url": "https://docs.example.com"
      }
    }
  ]
}