CrawlForge MCP v6.7–v6.12: Scrape Cloudflare-Protected Sites

"What browser MCP to bypass Cloudflare?" is a live question on r/ClaudeAI, and the honest answer for most of 2026 was: none of them reliably, ours included. A bot-detection benchmark we ran against v6.6.2 in mid-September recommended routing through residential proxies. We configured it. It did nothing. Finding out why turned into six releases in eleven days, and this post is the record of what they changed for anyone who needs to scrape Cloudflare-protected sites from an MCP server without pretending to be something they are not.

CrawlForge MCP v6.7.0 through v6.12.0 are all about one page: the one that comes back as a challenge wall. The plain fetch still runs first and still costs 2 credits. What changed is everything that happens after it is refused.

Table of Contents

• What Shipped, Release by Release
• Why a Plain Fetch Fails on a Cloudflare-Protected Site
• One Call, Three Tries: The Escalation Ladder
• A Solved Challenge Stays Solved
• The Agent Retries Walled Pages Itself
• Bring Your Own Proxies, and They Route Now
• Measured Against a Benchmark, Not Claimed
• What CrawlForge Will Not Do
• Running It for Clients: The Agency Setup
• How to Upgrade

What Shipped, Release by Release

No tool was added or renamed, and no price changed except the agent ceiling, which now depends on what got through. Every line above is drawn from the server's changelog.

Why a Plain Fetch Fails on a Cloudflare-Protected Site

Cloudflare scores a request before the page is served, and a Node HTTP client fails that score in three places at once: the IP is a known datacenter range, the TLS handshake does not look like a browser's, and the interstitial needs JavaScript the client will never run. The result is a 403, or a 200 carrying nothing but a challenge shell. The stealth mode guide covers the detection layers in detail; this post is about what the server now does when it hits one.

Since v5.6.11, scrape has reported that wall as success: false with blocked.vendor naming Cloudflare, DataDome, PerimeterX, Akamai, Amazon or Vercel, whatever the HTTP status, and charged nothing for it. Since v5.9.0, escalate: true has let the same call render the page once in the stealth browser instead of returning the block. The six releases below are what that escalation stage turned into.

One Call, Three Tries: The Escalation Ladder

With escalate: true and the default engine of "auto", one scrape call now climbs a ladder, and stops at the first rung that returns the page:

1. The plain fetch, with the CrawlForge/<version> User-Agent. 2 credits. If the host walled a request in the last 24 hours, the MCP server skips this rung rather than repeat a doomed fetch.
2. A Chrome TLS handshake through impit, new in v6.11.0. It presents Chrome's TLS fingerprint and still carries the honest CrawlForge User-Agent. No browser starts. From a residential IP where Cloudflare blocks the plain fetch, indeed.com came back in about a second.
3. The stealth browser, Camoufox first and Chromium with a warning when the Camoufox binary is absent (v6.8.0). The clearance jar from v6.10.0 is applied here, and on Chromium a wall that is still up after the usual wait, with a challenges.cloudflare.com frame on the page, gets one click on the Turnstile checkbox (v6.11.0).

The price is 2 + 5 whichever of the last two rungs got the page, and a call the plain fetch served still pays 2. A page that meets the wall on every rung comes back as a block carrying escalated: true and is charged nothing.

// npm install crawlforge-sdk
import { CrawlForge } from 'crawlforge-sdk';

const client = new CrawlForge({ apiKey: process.env.CRAWLFORGE_API_KEY });

// One call. The plain fetch runs first; only a wall triggers the rest.
const result = await client.scrape({
  url: 'https://www.indeed.com/cmp/Burger-King/reviews',
  formats: ['markdown', 'metadata'],
  escalate: true
});

const data = result.data as {
  escalated?: boolean;
  stealth?: { engine: string };
  content: { markdown: string };
};

// "impit" means the TLS rung got it and no browser ever launched.
console.log(data.escalated, data.stealth?.engine, data.content.markdown.length);

Two guards sit inside the impit rung. Every redirect hop is SSRF-checked with its own DNS resolution, because impit resolves names itself. And a page with under 200 characters of visible text goes on to the browser rather than being returned: quora.com answers a Chrome handshake with its client app's "Something went wrong" fallback under a perfectly normal title, and v6.12.0 now treats that fallback as a soft block on every path, not only this one.

One deliberate difference: the hosted REST API's scrape escalation stays browser-only. Measured from the hosted instance's datacenter IP, the impit step cleared none of the benchmark's walls, so the hosted service runs with CRAWLFORGE_IMPIT=off while an npm install keeps it on.

A Solved Challenge Stays Solved

Before v6.10.0, every stealth render started cold. If Cloudflare's interstitial let the browser through, that clearance died with the browser context, and the next call to the same host solved it again.

The clearance jar keeps exactly three cookies from a render that got past a wall: cf_clearance, __cf_bm and datadome. They are replayed to the next stealth context with the same engine, User-Agent and proxy, which is the identity the clearance was issued to. That covers the scrape escalation stage, stealth_mode, the agent's stealth retry, browser_session and scrape_with_actions.

Measured on stackoverflow.com from a residential IP: the first stealth call went through Cloudflare's interstitial in 4.1 seconds, with the orchestrate and fo challenge requests visible in the trace. The second went straight to 200 with neither, in 1.6 seconds. A second process reused a clearance from disk and loaded indeed.com with zero challenge-platform requests.

The boundaries matter as much as the feature:

• No other cookie is ever kept. A session cookie from one caller's login cannot reach another caller's context, because the jar does not store it.
• A render that meets the wall again drops that host's clearances, so a stale clearance cannot loop.
• Expiry is the cookie's own, capped at 24 hours. The jar holds at most 32 identities with 200 cookies each.
• It persists at ~/.crawlforge/stealth-clearance.json, mode 0600, keys hashed. CRAWLFORGE_CLEARANCE_JAR=off turns it off.

A persistent Chromium profile pool was considered for the same phase and deliberately not built. A shared profile keeps logins and site storage, and on a hosted instance that would leak between customers.

The Agent Retries Walled Pages Itself

Before v6.9.0, the agent tool's act stage ran only the plain fetch. A challenged seed page was dropped and the answer was assembled from search snippets. In our own review that produced the sentence "3.3 stars, based on 3.3 reviews" for a Burger King page on Indeed, which is what happens when a rating leaks into the count field of a snippet.

Now a page that hits a challenge, a 403 or 429, an empty shell or a timeout is retried through the same escalation stage scrape uses, with the same compliance gate, engine resolver and proxies. URLs the caller named are retried first. A discovered URL is retried only when it has no relevant search snippet. Refusals, 404s and 5xx errors are never retried, a run makes at most two retries, and none starts with under 20 seconds of wall clock left.

With the Chromium engine, the same Indeed test read the seed page itself and answered 3.3 stars from 58,942 reviews, with the evidence marked via: "stealth".

Pricing follows what got through:

The result reports stealth_retries (attempts) and stealth_retries_charged (billed). A retry that meets the wall again, or throws, is free. The REST API charges the same 8 + 5 per charged retry.

Bring Your Own Proxies, and They Route Now

CrawlForge supplies no proxies. What v6.7.0 fixed is that the ones you supply are actually used. stealthConfig.proxyRotation had been a no-op in three separate ways: the proxy went onto Chromium's --proxy-server flag, which cannot carry the user:pass every residential proxy is issued with, so authenticating proxies answered 407; Camoufox's launch path returned before the argument list was built, so the Firefox engine ran unproxied whatever was asked; and the list was read once at launch, so rotationInterval could never elapse.

All three are fixed. Proxies are ordinary URLs (http, https, socks4, socks5, credentials percent-encoded), applied per browser context so both engines authenticate, and a malformed entry is an error rather than a silently unproxied request. Camoufox now runs with its own geoip, block_webrtc and humanize features on, so behind a proxy it derives locale, timezone and location from the exit IP and agrees with the address the site sees.

v6.8.0 added the server-level form, CRAWLFORGE_STEALTH_PROXIES: a comma-separated list used by the escalation stage, stealth_mode, browser_session, scrape_with_actions and the agent's retry when a call passes none. A proxy on the call always wins.

The same release fixed a Camoufox identity problem that predated all of this. About two thirds of Camoufox contexts had presented a Chrome User-Agent on a Gecko engine and sent sec-ch-ua client hints Firefox has never implemented. A detector could act on that from the request headers alone, before a line of script ran. Camoufox now presents its own Firefox identity, with nothing Chromium-shaped injected over it.

Measured Against a Benchmark, Not Claimed

Every number in this post comes from npm run bench:stealth, which v6.8.0 turned from a hand-run table into a command. It drives the stealth browser and the plain fetch against twelve bot walls and five detector pages, and heads its matrix with the host OS, exit-IP class, engine and browser versions, because a Cloudflare result from a residential connection and one from a datacenter are different measurements.

Measured against it, the fingerprint leaks were closed: navigator.webdriver reads false instead of being deleted, nothing is an own property of navigator, userAgentData drops its HeadlessChrome brand, the Chrome major comes from the installed binary, and a Web Worker answers what the document answers. Detector self-probes went from a known list of failures to 19 pass, 0 fail, 1 skip on both engines. npm run bench:stealth:ci runs ten of those probes with no third-party site and fails only on a regression, with a negative control that forces navigator.webdriver to true and must be caught.

Two findings from the benchmark are worth carrying with you:

• Which engine passes depends on the exit IP. From a residential connection Camoufox cleared indeed.com where Chromium failed. From the hosted instance's datacenter address two runs found the exact reverse. That is why CRAWLFORGE_STEALTH_ENGINE exists to pin a deployment, why the hosted service pins Chromium, and why the npm default still prefers Camoufox.
• A single DataDome cell is not a result. leboncoin on Chromium passed on one run and was blocked an hour later from the same IP with nothing changed in between.

What is not closed is stated in the changelog rather than glossed. Chromium still leaks HeadlessChrome from a SharedWorker, a target Playwright attaches nothing to. And the Turnstile click was verified against Cloudflare's forced-interactive test sitekey, which proves the mechanism only; whether a real site accepts the click still depends on the IP and fingerprint it scores.

What CrawlForge Will Not Do

The searcher's word is "bypass". Ours is narrower, and these releases held the line on it:

• No CAPTCHA solving and no forged challenge tokens. The Turnstile click is a single click on a checkbox Cloudflare put on the page, on Chromium only. No token API is called.
• The User-Agent is honest on every rung. The impit handshake presents Chrome's TLS fingerprint and still identifies itself as CrawlForge/<version>. On indeed.com, the honest User-Agent passed 3 runs in 4 where a Chrome User-Agent passed 0, so the honest header was not the thing costing us the page.
• Clearance cookies are never moved between identities. Replaying a cf_clearance cookie through impit was declined, because the cookie is bound to the User-Agent it was earned with, and replaying it would mean sending a browser User-Agent instead of our own.
• robots.txt is checked before any rung runs, against the same CrawlForge product token the plain fetch uses. A respect_robots: false override is written to the compliance audit log.
• Every escalation is audited. Since v6.11.0, scrape with escalate, the agent's stealth retry and stealth_mode each write a stealth_escalation row to logs/compliance-audit.log, after the compliance gate and before the browser navigates, carrying the URL, the tool, the resolved engine and a truncated hash of the API key. A refused request writes none.

That list is the difference between a scraper an agency can put its name next to and one it cannot.

Running It for Clients: The Agency Setup

If you run CrawlForge for several clients, the pieces above compose into one deployment:

# Your residential or ISP proxies. CrawlForge supplies none.
CRAWLFORGE_STEALTH_PROXIES=http://user:pass@proxy-a:8000,socks5://user:pass@proxy-b:1080

# Pin the engine once you have measured which one passes from your exit IP.
CRAWLFORGE_STEALTH_ENGINE=chromium

# Leave the Chrome TLS rung on unless your exit IP is a datacenter range.
# CRAWLFORGE_IMPIT=off

# Clearances persist per engine, User-Agent and proxy; off if you would rather start cold.
# CRAWLFORGE_CLEARANCE_JAR=off

Then measure before you promise anything: npm run bench:stealth from the box that will do the work, so the matrix carries your exit-IP class and not ours. The audit log gives you a per-key record of every stealth escalation to hand a client who asks what was fetched on their behalf, and the clearance jar's three-cookie rule means one client's login never sits in a context another client's job reuses.

For the hosted API, the honest description is this: stealth traffic exits from a datacenter IP, the escalation is browser-only, and which walls that clears is a property of that IP. Where the client's targets sit behind IP-reputation checks, the self-hosted MCP server with your own proxy list is the setup that measures well.

How to Upgrade

npm install -g crawlforge-mcp-server@latest
crawlforge --version   # 6.12.0

An MCP client that launches the server with npx picks up v6.12.0 on its next restart. impit is an optional dependency with prebuilt binaries; when it is absent, escalation behaves exactly as it did in v6.10.0. Camoufox is optional too, needs Node 22 for its transitive dependencies, and "auto" falls back to Chromium with a warning that says so. No schema or output shape changed on any tool.

Want to see what the ladder clears from your IP? Start free with 1,000 credits, run scrape with escalate: true against the page that has been blocking you, and read the scrape API reference for every parameter the escalation stage takes.