AI Engineering7m
Shipping an MCP Server Inside Your Product: Five Decisions
Ten tools out of twenty-nine, off by default, version-pinned and swappable — the architecture behind an AI feature that cites pages it actually opened.
Blog
Building AI applications with web data.
13 ai engineering
AI Engineering
4m
Web Search MCP Server With Local LLM: Which Model Actually Runs
search_web finds the pages; a local Ollama model reads them. CrawlForge picks that model by measured accuracy rather than parameter count — and a 4B model beats a 20B one.
AI Engineering
4m
Web Search MCP Server: Self-Hosted With SearXNG
Most "self-hosted" search MCP servers only run the process locally. CrawlForge can point search_web at your own SearXNG instance — and here is the part that still isn't self-hosted.
AI Engineering
12m
AI Crawler List 2026: Every Bot + Should You Block Them?
Every AI crawler that matters in 2026: 31 bots, their robots.txt tokens, and what blocking each one actually costs you. Copy-paste file included.
AI Engineering
6m
Reddit Data for AI Agents: The MCP Route
Reddit is where the unfiltered opinions live, and it is the most agent-hostile mainstream site on the web. Here is how to give your AI agent working Reddit search — posts, comments, and full threads — through one MCP tool.
AI Engineering
13m
Agent Scraper: What It Is and How to Build One
An agent scraper follows a goal, not a selector. What that means, the three ways to build one, working code, the real failure modes, and the credit math.
AI Engineering
9m
SSRF in MCP Servers: Why Scrapers Leak Cloud Secrets
A July 2026 study found 91.8% of audited MCP servers lack authentication. Here is why web-scraping servers leak cloud credentials -- and how to stop it.
AI Engineering
11m
Best Web Scraping Tools for AI Agents in 2026
The best web scraping tools for AI agents in 2026, ranked by agent-readiness: MCP-native tool discovery, typed schemas, and token-efficient output.
AI Engineering
9m
Extract Web Data With Local LLMs (Ollama + CrawlForge)
No API keys, no cloud, no data leaving your machine. Use extract_with_llm with local Ollama to pull structured data from any site.
AI Engineering
10m
MCP Protocol Explained: A Developer Guide for 2026
Learn how the Model Context Protocol works, why it matters for AI agents, and how to build MCP servers and clients with architecture diagrams and code.
AI Engineering
11m
How to Build a RAG Pipeline with Web Data
Build a production RAG pipeline that crawls websites, extracts content, chunks text, generates embeddings, and serves retrieval-augmented answers.
AI Engineering
14m
Stealth Mode Scraping: How CrawlForge Bypasses Anti-Bot Detection
Technical deep-dive into anti-bot detection systems and how CrawlForge's stealth mode features help you scrape protected websites ethically and effectively.
AI Engineering
14m
Web Scraping for AI Training Data: A Complete 2026 Guide
Learn how to collect, clean, and structure web data for AI training. Best practices for ethical scraping, data quality, and LLM fine-tuning preparation.