CrawlForge MCP v6.24: A Reliable Web Scraping MCP Server

An agent that reads the web is only as useful as the worst answer its tools hand back. A page that comes back half-empty, an error that blames the wrong thing, or a cited link nobody fetched all count. Between September 29 and October 22 we shipped 24 releases of CrawlForge MCP, v6.12.1 through v6.24.0, and almost none of them added a feature you would put on a landing page. They were about one thing: making CrawlForge a reliable web scraping MCP server, one whose output an agent can act on without a human checking it.

Most of the fixes came from two live sweeps of all 31 tools against real sites. The first made about 470 calls on v6.21.1, and each defect was reproduced by a second tester before it was fixed. The second covered v6.22.3, with at least three live sites per tool. This post is the record of what changed, grouped by what it changes for you.

Table of Contents

• What Shipped, Release by Release
• Failed Calls Are Not Charged, Anywhere
• A Page Cache for Repeat Reads
• Two Live Sweeps Toward a Reliable Web Scraping MCP Server
• Agent Answers You Can Check
• Embedded State From More Frameworks
• Browser Automation That Waits for the Page
• Security: Every Hop Goes Through the Guard
• Search Moved to the Brave Search API
• Breaking Changes to Check Before You Upgrade
• How to Upgrade

What Shipped, Release by Release

No tool was added or renamed, and no list price changed. Every line above comes from the server's changelog, and the website's changelog carries the same entries.

Failed Calls Are Not Charged, Anywhere

Most scraping APIs answer the question "do failed requests get charged?" with a status-code list. CrawlForge's answer since v6.22.0 is shorter: a failed call costs 0 credits, on the MCP server and on the REST API.

Before v6.22.0 the MCP server charged half the price on an error. Three tools also had a quieter problem: extract_structured, extract_with_llm and summarize_content returned success: false without marking the result as an error, so those calls were billed in full and logged as successes. Now a failure is any of three things, and none of them is billed:

• a result with isError set
• a thrown error
• a top-level success: false body

Refusals were already free and still are. A robots.txt disallow, a blocklisted host, an SSRF refusal or a bot wall that stops every rung of an escalation all cost nothing. Since v6.16.0 that also covers scrape_with_actions and browser_session chains that our own compliance gate stopped partway. The _cost block that reports what a call was charged now appears in structuredContent too for tools that declare an output schema, not only in the text result, so an agent can read its own spend without parsing prose.

A Page Cache for Repeat Reads

Agents re-read pages. A model scrapes a page for markdown, decides it needs the metadata, then asks for one value out of the embedded JSON, and that is three fetches of the same URL. v6.24.0 adds a session page cache so the second and third reads cost less and leave the site alone.

A repeat request for the same page within 10 minutes reuses the copy already fetched and costs 1 credit, whatever it asks for this time: other formats, another JSON path, other selectors. Parallel calls for one page share one request. It covers nine tools: fetch_url, scrape, extract_metadata, extract_text, extract_links, scrape_structured, extract_embedded_state, extract_structured and extract_with_llm.

What is never cached matters as much as what is:

• non-2xx responses, bot walls and challenge pages
• responses marked Cache-Control: no-store
• requests that carry Authorization or Cookie headers

A cache hit still runs the blocklist and robots.txt checks, so a cached page is never a way around a site's rules. The cache is bounded by size (CRAWLFORGE_PAGE_CACHE_MB, default 64, 0 turns it off). It runs in a local MCP install only: it is off under --http and for requests through the REST API.

When you need the live page, say so with maxAge (milliseconds). maxAge: 0 forces a fresh fetch:

{
  "name": "scrape",
  "arguments": {
    "url": "https://example.com/pricing",
    "formats": ["markdown"],
    "maxAge": 0
  }
}

Every result from those nine tools carries cache: { state, fetchedAt, ageMs, note }, so the agent can tell whether it is looking at a 3-second-old copy or a fresh one.

Two Live Sweeps Toward a Reliable Web Scraping MCP Server

The scrape and crawl_deep fixes below are a sample, and the changelog has the full list. What they share is that each one turned a wrong answer reported as success into a right answer or an honest error.

Text that reads like the page

• List pages keep their text. Readability took a list item's author for the article byline, so quotes.toscrape.com lost "by Albert Einstein". A list page where main-content extraction keeps under 60% of the visible text now falls back to the whole page. This reaches scrape, extract_content, batch_scrape, crawl_deep and scrape_with_actions.
• Blocks no longer weld together. One shared text flattener replaced several, so "July 2023If you…" reads as two lines, and links written back to back each get their own line.
• Code blocks come out fenced, with their language. Sphinx code on docs.python.org had come back as escaped prose.
• Wikipedia keeps its section headings and drops the "[edit]" links.

Crawls that finish and count honestly

• crawl_deep resolves links against the page it landed on. docs.python.org/3/tutorial redirects to /3/tutorial/, and every chapter link had resolved one level too high and returned 404.
• Link analysis no longer stalls the server. On a 7.3 MB page with 16,766 links it took 50 seconds and starved every other call. It now takes 0.5 seconds.
• crawl_deep stops at max_pages. It used to drain every queued link first: 232 seconds for 3 pages on one site.
• map_site includes the start URL, so a one-page site returns 1 URL, not 0.

Extraction that says what it could not find

• extract_structured returns every field you asked for. A value it cannot find is null and named in a new missing_fields array, instead of the field silently disappearing.
• analyze_content stops scoring other languages with English formulas. Spanish text had scored a clear complaint at −0.08. Sentiment now uses the AFINN-165 lexicon with negation for English, and returns not_applicable for other languages, as readability does.
• summarize_content tries an abstractive summary by default at medium length when an LLM is reachable, and strips cookie lines and navigation menus before summarising.

Errors that name the cause

A DNS failure used to read as a bare fetch failed, and the next-step hint suggested stealth_mode, sending the agent into a stealth browser to fetch a typo. Errors now carry the cause (ENOTFOUND, ECONNREFUSED), and the hint says to check the URL. A selector that matches nothing is reported as a selector miss that names the selector, not as a slow site. The hints themselves are now keyed on the class of error (robots, blocklist, SSRF, 404, timeout, validation) rather than one generic hint per tool.

Agent Answers You Can Check

The agent tool answers a question from the open web. Its fixes in this cycle are about provenance, because an answer is only useful if you can see where it came from.

• A cited URL must come from a source. Since v6.22.1, every URL in an answer has to be a page the agent fetched, a link on one, or written in its source text. One that is not gets a corrective rewrite. One that survives the rewrite is named in provenance.unverified_urls and in a warning inside the answer. Before, an answer with the right Hacker News title and a link to a 2021 story passed as verified.
• The agent can link the exact item. Each source now carries up to eight of its links into the synthesis, so "today's top story" comes with that story's URL.
• The official source wins ties. A tied domain vote used to promote the first-ranked site, often Wikipedia.
• "Last updated" is not a release date. An answer that gives a page's update date as a release date is corrected or flagged, in provenance.update_dates.
• It keeps to its 120-second limit. One sweep run had taken 183 seconds. A run that hits the limit now returns degraded: true with the evidence it has.

The price is unchanged: 8 credits, plus 5 for each stealth retry that gets its page, 18 at most.

Embedded State From More Frameworks

Many sites ship their data as JSON inside the HTML, for the front-end framework to hydrate. extract_embedded_state reads that JSON from a plain fetch, with no browser, for 2 credits. v6.15.0 and v6.16.0 widened what it can read:

• Nuxt 3's __NUXT_DATA__, decoded into objects as nuxt_data (it used to come back as an index array)
• SvelteKit data as sveltekit_data, and Inertia's data-page as inertia_page
• YouTube's ytInitialData and ytInitialPlayerResponse, so a watch page returns videoDetails from a plain fetch
• Remix, Shopify product JSON and analytics meta, and globals assigned a JavaScript literal rather than JSON, read by a static evaluator with no page code run

Three parameters make large payloads workable. find returns every property with a given key name, each with the path to pass straight back. keys_only returns two levels of keys with their types instead of the data. Over max_inline_chars, the result is a preview plus a handle you read with read_result, instead of a 2 MB payload in the context window.

{
  "name": "extract_embedded_state",
  "arguments": {
    "url": "https://shop.example.com/products/widget",
    "find": "price"
  }
}

A walled page can be re-read in the stealth browser with escalate: true, which projects 7 credits and charges 2 when the plain fetch got the page. Since v6.24.0, a path that does not resolve returns the keys two levels deep where it stopped, with a next step built from them. Before, it told the agent to re-read the whole payload.

Browser Automation That Waits for the Page

scrape_with_actions and browser_session drive a real browser. Their most common failure was acting on a page that had not finished rendering.

• Snapshots wait for the page to settle: load (up to 5 seconds), then a quiet window with no DOM change for 300 ms and no request in flight for 500 ms (up to 3 seconds). The Amazon homepage snapshot went from 0 nodes to 236, search box included.
• Snapshots see inside shadow roots and iframes, cross-origin ones included, through Playwright's native accessibility snapshot.
• A replaced element is found again. When a page swaps an element after the snapshot (Wikipedia's search box does), the ref is looked up once more by role and name instead of failing as detached.
• Cookie banners are answered for you. browserOptions.consent: "reject" runs DuckDuckGo's autoconsent opt-out after each navigation. The Guardian's Sourcepoint dialog took about a second.
• A burst of calls queues instead of failing. "Maximum concurrent sessions (3) reached" is gone. Calls wait up to 60 seconds for a browser slot and report queued_ms.
• Failures cost less time. maxRetries now defaults to 0, so a click on a missing selector with a 4-second timeout fails in about 4 seconds instead of 20.8. Stealth clicks on Camoufox went from 12.5–20 seconds to 1.4.

Every navigation in a chain is now checked for bot walls, not only the final page. The result lists each one under navigations, with AWS WAF and F5 named alongside Cloudflare, DataDome and the rest. For what happens after a wall, the v6.12 release post covers the escalation ladder.

Security: Every Hop Goes Through the Guard

A scraping server fetches URLs that a model chose, and that model may have read them off a hostile page. The SSRF guard has been in place since v5. This cycle closed the paths that went around it:

• Browser traffic goes through a guard proxy. Since v6.23.0, every browser context connects through a local forward proxy that checks each connection (redirect hops, iframes, subresources, fetch, XHR and WebSockets) and connects to the address it checked, so DNS rebinding no longer gets past the browser path. Before, a page that redirected to 127.0.0.1 reached it before the block was reported.
• [::] is blocked. The unspecified IPv6 address reaches the local host, like 0.0.0.0.
• Every redirect hop is gated like a first request. Since v6.16.0, the blocklist and robots.txt are checked again before each hop, and each hop waits out the host's Crawl-delay. A redirect into a disallowed path is never requested.
• Local file reads are limited. process_document with a local path now accepts only document file types and refuses dot-directories such as ~/.ssh. Set CRAWLFORGE_ALLOWED_DIRS to restrict reads to the directories you list.
• Your paths and credentials stay on your machine. A local file path no longer reaches the usage report. Proxy credentials are redacted from results, error messages and usage reports, and credentials in URLs are redacted from the compliance audit log.
• MCP SDK 2.3.1, for three advisories in the SDK's HTTP transport and auth handling.

Search Moved to the Brave Search API

On October 19 the CrawlForge API moved search_web from Google Custom Search, which Google discontinues on January 1, 2027, to the Brave Search API. Results come from Brave's independent index, and provider.backend reports which backend served the call. total_results is null because Brave does not report a total, and a 0 would read as "no results". Paging is exact: an offset that is not a multiple of the page size now returns the window you asked for.

serp_rank gained a decoy check. Google now serves some automated lookups a realistic page of results unrelated to the query. When fewer than 1 in 5 of the top results mention the keyword, the result carries suspect: true and says how many did. The default depth is now 30 on both the MCP server and the REST API, still 5 credits.

Breaking Changes to Check Before You Upgrade

Most of the 24 releases change nothing you call. These do:

How to Upgrade

npm install -g crawlforge-mcp-server@latest
crawlforge --version   # 6.24.0

# Optional: size the session page cache in MB (default 64), or 0 to turn it off
export CRAWLFORGE_PAGE_CACHE_MB=64

An MCP client that launches the server with npx picks up v6.24.0 on its next restart. If you installed the Agent Skills, refresh them with crawlforge install-skills --target claude-code --force so their tool notes match the server.

Want to see the difference on the pages your agent reads? Start free with 1,000 credits, point your agent at the sites that gave it trouble, and check what each call cost in its _cost block. Failed calls will show 0. Prices for every tool are on the pricing page.