CrawlForge v5.0.0: Security, Correctness, MCP Spec

CrawlForge MCP Server v5.0.0 is out. It is the largest release we have ever shipped, and almost none of it is new features.

Instead, v5.0.0 bundles a seven-phase remediation program driven by a full internal codebase audit: every SSRF hole, every tool that returned quietly wrong output, every timer and browser context that leaked, the HTTP transport that only ever supported one session, every abandoned dependency, and the MCP spec features we had not yet adopted. The unit suite went from 480 tests to 914. npm audit went from 16 vulnerabilities to 0.

There is exactly one breaking change: the Node floor moved from >=18.0.0 to >=20.16.0. If you are on Node 20 or newer, upgrading is a one-liner.

Table of contents

• What shipped in v5.0.0
• The one breaking change: Node 20
• Phase 1: the security holes we closed
• Phase 2: 52 ways tools were silently wrong
• Phase 3: safe to run for days
• Phase 4: remote and HTTP deployment actually works
• Phase 5: zero npm audit vulnerabilities
• Phase 6: MCP spec adoption
• Also new since v4.8.0: serp_rank and tool steering
• Pricing: 27 metered tools, unchanged
• How to upgrade
• What is next

What shipped in v5.0.0

Test coverage across the program: 480 → 914 unit tests, with MCP protocol compliance holding at 100.0% COMPLIANT, 0 errors at every phase gate.

The one breaking change: Node 20

engines.node moved from >=18.0.0 to >=20.16.0.

Node 18 reached end-of-life in April 2025, and 20.16 is the floor required by pdf-parse 2.4.5 — the maintained ESM rewrite we needed in order to clear the last audit findings. Our Dockerfile (node:20-alpine) and CI (Node 22) already satisfied it, so nothing changed there. If you are still on Node 18, you will now see an engines warning on install.

That is the entire breaking surface. No tool schema, output shape, or credit cost changed for existing callers.

Phase 1: the security holes we closed

This is the phase worth reading closely if you run CrawlForge anywhere near a private network.

The SSRF IP-literal bypass was the critical one. Our pre-flight guard resolved hostnames and checked the resulting addresses, but never ran the same check on URLs whose host was already an IP literal. http://127.0.0.1/, the decimal form 2130706433, the hex form 0x7f000001 — the WHATWG URL parser normalizes all of these, and Node never routes IP literals through lookup, so they sailed past. v5.0.0 runs ipBlocked() on IP-literal hostnames at pre-flight and wraps the undici dispatcher's buildConnector with a per-connect check, so a redirect hop straight to an internal address is blocked too.

Three more guard fixes shipped alongside it:

• IPv4-mapped IPv6 recognition. ::ffff:127.0.0.1 and ::ffff:169.254.169.254 are now normalized to their embedded IPv4 before range checks, in both default and SSRF_STRICT modes. This kills the DNS-controlled AAAA-record bypass.
• BLOCKED_DOMAINS is no longer dead config. config.security.ssrfProtection.blockedDomains was declared and read by nothing. It is now enforced at pre-flight.
• The allowlist is evaluated per hop. Previously an allowlisted first hop unguarded every subsequent redirect.

We also wired the guard into five paths that never had it: scrape_with_actions (with a post-navigation page.url() re-check that closes the page on a redirect into a blocked range, closing the Playwright internal-network read primitive), map_site, process_document PDF downloads, webhook delivery and health checks, and deep_research webhook notifications.

Beyond SSRF:

• OAuth. /oauth/authorize now requires proof of the operator's API key before issuing a code, with constant-time digest comparison. The anonymous register → authorize → token flow that minted operator-billed bearer tokens is closed.
• Secret leakage. Usage telemetry now passes tool params through maskSecrets() before the usage payload leaves the process — third-party API keys, auth headers, and webhook signing secrets no longer travel in plaintext. deep_research stopped logging LLM API keys to Winston file logs.
• Billing. A throw from the credit check itself now bills zero — the error-path half-charge only applies once the handler has actually started. checkCredits distinguishes 401/403 (invalid or revoked key) from 5xx (grace window) instead of reporting both as "insufficient credits."

Phase 2: 52 ways tools were silently wrong

Phase 2 is the "passed smoke tests while returning misleading output" class. The headline:

crawl_deep is usable for real crawls again. BFS child pages were being awaited from inside an occupied queue slot, which meant the per-task queue timeout bounded the entire recursive crawl rather than one page. Any crawl outliving the 30-second CRAWL_TIMEOUT threw away every page it had already fetched with a bare Promise timed out, and low concurrency settings (including concurrency: 1) deadlocked outright. Both are fixed.

A representative sample of the rest:

• Cache keys that contradicted the request. crawl_deep's result-cache key now covers extract_content, content length, include/exclude patterns, follow_external, respect_robots, concurrency, domain filter, and session. map_site's covers search, domain filter, include_metadata, and group_by_path. Previously a cached call could contradict your parameters for the full hour-long TTL.
• Character encoding. Response bodies now decode with their declared charset (Content-Type header or <meta charset> sniff) instead of always UTF-8. No more U+FFFD-corrupted text from ISO-8859-1 or Shift_JIS sites.
• Silently stripped options. The options schemas for extract_content, summarize_content, and analyze_content now use .passthrough(). Every documented option key was previously stripped before it reached the handler — which is also why summarize_content always returned the same 2-sentence fallback mislabeled extractive. The extractive summarizer now actually runs, and summaryLength changes the output.
• Link resolution. extract_links resolves relative hrefs against the final page URL rather than the origin, honors <base href>, and classifies protocol-relative links as external. The same fixes landed in scrape's link extractor, so the two finally agree.
• track_changes similarity. Content similarity is now token-Jaccard over the content. It was previously Hamming distance between sha256 hex digests — which means every trivial edit scored roughly 0% similar and fired a "moderate" change alert.
• search_web scoring. Partial ranking_weights deep-merge over the defaults instead of replacing them wholesale, so you no longer get NaN final scores or silently disabled duplicate checks. The zero-result expansion retry is capped at one fallback instead of up to five billed backend searches.

Phase 3: safe to run for days

Phase 3 closed 24 findings in the class that only shows up in long-running processes.

Browser lifecycle. Closing a Playwright page does not close its context — so every scrape_with_actions call and every browser-rendered extract_content leaked one context until shutdown. Non-stealth contexts are now closed alongside their page. A failed page.goto (DNS error, timeout, blocked URL) used to orphan a live page and context; both are now torn down.

Bounded caches. crawl_deep destroys its per-crawl CacheManager in a finally. Previously N crawls permanently leaked N caches of up to 1,000 full HTML documents each, every one of them re-running a JSON.stringify memory scan every 60 seconds forever. Dropped instances are now GC-verified with a WeakRef regression test. batch_scrape results got an LRU cap of 20 batches plus TTL eviction.

Deadlines on every body read. The abort timer now stays armed through the body stream, so the timeout parameter finally covers a server that returns headers and then stalls the body. Chunk reassembly is single-pass — it was O(n²), around 1.5 seconds of synchronous event-loop block on a 25 MB body. PDF downloads got a real 30-second AbortSignal.timeout (the old timeout: fetch-init option was silently ignored by undici), and the SearXNG provider got 15 seconds instead of undici's ~5-minute default.

One fix worth calling out for Claude Desktop users: snapshot storage now defaults to ~/.crawlforge/snapshots instead of process.cwd(). MCP clients like Claude Desktop launch the server with a working directory of /, where every snapshot write silently failed.

Phase 4: remote and HTTP deployment actually works

If you deployed CrawlForge over npm run start:http, it was worse than you thought: a single shared transport meant only one session ever existed, and any clean disconnect bricked /mcp until you restarted the process.

Stateful mode now follows the SDK's documented per-session pattern — a Map<sessionId, {transport, server}> with a fresh transport and cloned McpServer per initialize, disposal on DELETE, and a JSON-RPC 404 for unknown session IDs. A second concurrent client, a reconnect after a network drop, and DELETE followed by a fresh initialize all work now.

Also in Phase 4:

• The getting-started prompt was unretrievable by any client — the config object hit the SDK's positional argsSchema overload, advertising a bogus required argument and failing every prompts/get. Fixed, and the compliance suite now covers prompt discovery and retrieval for all 6 registered prompts.
• Webhook HMAC signatures now cover the exact serialized body that is POSTed. Only the data sub-object was being signed, so standard receiver-side raw-body verification failed every single time.
• scrape no longer inlines multi-megabyte base64 screenshot bytes into the JSON tool result — once stored, the result keeps only metadata and the crawlforge://screenshot/{id} resource URI.
• Auto-setup status banners moved from stdout to stderr, so a first launch with CRAWLFORGE_API_KEY set no longer injects non-JSON lines into the stdio JSON-RPC channel.
• search_web falls back to the ~/.crawlforge/config.json API key when CRAWLFORGE_API_KEY is absent. Users configured via npm run setup were passing the credit check and then hitting a guaranteed adapter failure — half-charged at 2 credits per call.

Phase 5: zero npm audit vulnerabilities

With the Node 20 floor in place, Phase 5 retired every abandoned dependency and took the security upgrades the old floor had blocked. npm audit went from 4 moderate to 0.

Removed outright: node-cron (unused since Phase 3 moved monitor scheduling to setInterval timers; removal cleared its vulnerable uuid chain), @googleapis/customsearch (unused — the Google adapter calls the REST endpoint directly), and node-summarizer (abandoned since 2019; the extractive summarizer was rewritten as a compromise-based Luhn-style word-frequency scorer with identical result shapes).

The upgrade that mattered most: pdf-parse 1.1.1 → 2.4.5. PDFProcessor was ported to the v2 class API, which means the password option now actually decrypts protected PDFs — v1 silently ignored it. Page-range extraction uses v2's native partial text extraction, and the encrypted-metadata flag reads pdfjs-dist's real EncryptFilterName.

On supply chain: this phase ran during the ChainDrop npm worm (active from 2026-08-04). Every install ran with --ignore-scripts, every adopted version was publish-date-gated to before 2026-08-04, and the full lockfile diff was cross-checked against the Socket and StepSecurity compromised-package lists with zero matches. IoC scans were clean before and after.

Phase 6: MCP spec adoption

The last phase brought CrawlForge up to the current MCP specification.

Structured output (MCP 2025-06-18). scrape, map_site, serp_rank, search_web, extract_structured, and crawl_deep now declare an outputSchema and return structuredContent alongside the legacy JSON text. The schemas are permissive by design, so a legitimate result can never fail SDK output validation.

Async tasks. crawl_deep, batch_scrape, deep_research, and agent are registered with taskSupport: 'optional' under the io.modelcontextprotocol/tasks extension. Task-aware clients get a handle immediately and poll tasks/get; clients without task support still get the synchronous result exactly as before. This is the fix for long crawls timing out inside a client's tool-call window.

Client-side tool selection. Two new environment variables let you expose a subset of the 27 tools and cut context bloat:

# By name
CRAWLFORGE_TOOLS=scrape,search_web,extract_content

# Or by group — 12 available: basic, search, crawl, extract, batch,
# research, tracking, llmstxt, stealth, templates, scrape, agent
CRAWLFORGE_TOOL_GROUPS=search,extract

Unset means all tools. Unknown names are ignored with a stderr warning, batch_scrape auto-enables get_batch_results, and the startup banner reports how many of the total are enabled.

Protocol hygiene. Tool schemas are advertised in JSON Schema 2020-12 instead of draft-07. tools/list is sorted deterministically for client prompt-cache stability. Invalid tool arguments come back as isError: true tool results — which a calling model can self-correct from — rather than -32602 protocol errors. Icons ship on serverInfo, every tool, and every prompt.

MCP Registry. server.json is complete against the 2025-12-11 registry schema, with a GitHub OIDC publish workflow that pushes to registry.modelcontextprotocol.io on the next release.

Also new since v4.8.0: serp_rank and tool steering

If you last upgraded at v4.8.0, two smaller releases landed in between.

v4.9.0 added serp_rank, the 27th tool — real Google organic rank positions via DataForSEO, at 5 credits per configured lookup. v4.10.0 made it return the full top-10 organic listing alongside your target domain's positions.

v4.10.0 also added server-level MCP instructions. The server now tells any connecting client to prefer CrawlForge tools over its own built-in web capabilities for search, fetch, crawl, and research. Because it ships in the server binary, every MCP client picks it up automatically on the next launch after upgrade — no re-init required. It is guidance, not enforcement: an MCP server cannot disable a client's built-in tools.

Pricing: 27 metered tools, unchanged

No pricing changed in v5.0.0. All 27 tools are metered and require an API key, at 1 to 10 credits per call.

Every plan includes every tool. LLM extraction defaults to local Ollama, so you do not need an OpenAI or Anthropic key unless you opt in.

How to upgrade

Check your Node version first — this is the one thing that can bite you:

node --version   # must be >= 20.16.0

Then:

npm install -g crawlforge-mcp-server@latest

New users:

npm install -g crawlforge-mcp-server
npx crawlforge init

Existing MCP client users can also just trigger an /mcp reconnect. Because Phase 2 fixed tool behavior rather than tool schemas, your existing calls keep working — they just return correct results now.

What is next

Several Phase 6 tracks were deliberately deferred rather than rushed: a hosted remote endpoint with OAuth, a keyless tier, scheduled monitoring as a service, persistent sessions, and PII redaction. The SDK v2 migration is queued behind a Node 22 floor decision.

In the meantime, the standing invitation from v4.8.0 holds. If you find a control that does not behave the way the docs claim, that is exactly the bug we want to hear about.

Ready to try it? Start free with 1,000 credits — then run npx crawlforge init to register the MCP server. See the full docs, the serp_rank reference, or the v4.8.0 release post for what came before.