CrawlForge MCP v5.2: Every Change Across 28 Tools, Tested Against Live Sites

Every one of our amazon-product tests passed. The tool returned null for the price currency, null for the rating, null for every image, and the string "Brand: Amazon" where the brand should have been. Nothing looked broken in CI, because the fixtures had been written to match the selectors instead of the site.

That is the shape of most of CrawlForge MCP v5.2. Over two days we ran all 28 tools against real sites — Wikipedia, Hacker News, Amazon, Shopify storefronts, npm, YouTube, GitHub, Reddit, an IRS PDF — and shipped six releases: v5.2.0 through v5.2.5, plus four releases of the shared crawlforge-extractors package. This is the whole set, organised by what it does for you rather than by version number.

Tool count stays at 28. One price moved. Nothing was renamed and no output shape changed, so it is a drop-in upgrade.

Table of Contents

• The Release at a Glance
• Why So Many Fixes Landed at Once
• Product Data: Shopify, Amazon, npm, YouTube
• Reddit-Wide Search After PullPush Went Dark
• Change Tracking That Actually Fires
• Scraping Fidelity: Content That Was Being Deleted
• Local LLMs: Ollama Finally Works
• Browser Automation and the Stealth Wedge
• Language Detection and PDF Tables
• Infrastructure: Caches, Transport, Chromium
• One Copy of the Extractors
• Credit Costs
• How to Upgrade

The Release at a Glance

Why So Many Fixes Landed at Once

Because we stopped trusting the test suite and pointed the tools at live sites instead.

The amazon-product failure above is the clearest case. Six unit tests covered that template and all six passed, because every selector they exercised — a priceCurrency meta tag, #acrPopover .a-size-base, img.a-thumbnail-image — exists nowhere on Amazon today, and the fixture HTML had been written to contain them. The tests were checking that our code agreed with itself.

That pattern repeated. scrape_with_actions had seven Playwright defects invisible to its suite because the fake pages implemented whatever the executor called, including APIs Playwright does not have. The youtube-video template read an attribute that appears nowhere on a YouTube watch page while its fixture kept the test green. process_document read a field that the PDF library declares and never writes.

So the new suites were written to fail against the pre-fix code first — 15 of 24 for amazon-product, 8 of 10 for the crawl_deep cache scope, 7 for the scrape_with_actions Playwright APIs. The suite is now 1,122 unit tests with MCP protocol compliance at 100% across all 28 tools, but the number that matters is the one above it: every fix below was found by a real page, not by CI.

Product Data: Shopify, Amazon, npm, YouTube

Four of the scrape_template templates changed, and three of them moved the same direction: stop parsing the rendered page, read the structured data the site already publishes.

shopify-product, the new one

Shopify's Dawn theme ships every price badge in the markup unconditionally and hides the inapplicable ones with component CSS. A scraper reading the DOM sees a "Sold out" badge on a product with a hundred units in stock. Ask an LLM for a compare-at price on a product that has none and it will produce a plausible one — in our testing it invented 27.99.

Shopify serves the same data as JSON at /products/<handle>.json, on every storefront including custom domains, so the template reads that:

{
  "tool": "scrape_template",
  "params": {
    "template": "shopify-product",
    "url": "https://shop.example.com/products/kelpie-bandana"
  }
}

You get exact price, compare-at price, on_sale, currency, the price range across variants, per-variant stock, options, images and tags — no HTML parsing, no LLM in the path. Store quirks found in live captures are handled: an absent compare-at price is "" on one store and "0.00" on another (both read as null, while a genuinely free product keeps its 0.00 price), tags arrive as an array or a comma-joined string, and because the endpoint carries no available flag, stock is derived from inventory management, policy and quantity — reporting null rather than guessing "in stock" when the payload does not say.

Making that possible needed a change one level down: TemplateRegistry gained two optional hooks, resolveUrl(url) to redirect the tool's single fetch and extractRaw(body, url) to parse a non-HTML response. The tool still owns the SSRF-guarded fetch and now reports fetchedUrl when a rewrite happened. HTML templates are unchanged.

amazon-product, rebuilt

Selectors were re-derived from live captures of three pages — a first-party device, a branded storefront and a book:

• Currency comes from the hidden add-to-cart form field, not a meta tag.
• Rating is parsed to a number from the title attribute on #acrPopover, so you get 4.7 rather than "4.7 out of 5 stars".
• Review count parses either "(198,594)" or "198,594 global ratings" — Amazon uses both — and returns 198594.
• Brand reduces all three byline shapes to the bare name.
• Images drop Amazon's size token from the URL. The tokened URL is a 1 KB thumbnail; the same URL without it is the 16 KB original.

npm-package, now reading the registry

npmjs.com answers plain HTTP fetches with 403, and where a body did arrive the selectors keyed off class-name fragments that no longer match — version came back null, weekly downloads null, and repository pointed at the stargazers link. It now resolves a package page to its registry document and reads that, returning version, license, repository, homepage, maintainers, keywords, dependencies and any deprecation notice. Repository URLs arrive as git+ssh://, git+https://, git:// or a bare owner/repo and are normalised to browsable https.

Weekly downloads are omitted rather than returned as null, deliberately. They live on a separate endpoint, and the registry endpoint that carries them alongside package data is a search endpoint: ask it for left-pad and it answers with pad-left. Returning another package's download count is worse than returning none.

youtube-video, counting the right statistic

The template read meta[itemprop="interactionCount"], which appears nowhere on a watch page, so views came back null against every live video. The real markup is one schema.org InteractionCounter block per statistic, distinguished only by a sibling interactionType — and YouTube emits the LikeAction counter first, so the obvious fix silently returns likes where views are meant. Selection is now by interactionType, parsed to a number so zero views reads as 0 rather than null, and the likes field the description already promised now exists.

Reddit-Wide Search After PullPush Went Dark

If your Reddit pipeline started returning 429s this month, the fix is not a longer sleep, a rotating user agent or a proxy. We tested all three. Every request to PullPush now returns:

{ "error": "Rate limit exceeded. This website does not provide free scraping resources for agents..." }

Any user agent, no user agent, a browser user agent — no difference, and from some server IPs a Cloudflare 403 challenge instead. This is policy, not throttling.

It matters more than one dead endpoint because of what PullPush uniquely did. Arctic Shift, the other public Reddit archive, cannot keyword-search across all of Reddit by design. Ask it for a query that names no subreddit and no author and it returns HTTP 400:

So scoped searches (a subreddit, a user) still worked, and unscoped ones — where you know the phrase but not the community — had no backend at all. That is the search most people actually want, and the one Reddit's own API has never exposed cleanly.

reddit_search now serves it in two steps. Discovery: a site-restricted web search over reddit.com returns post URLs in relevance order, and post IDs come out of the /comments/<id>/ segment. Hydration: those IDs go to the Arctic Shift archive in one batch lookup, which returns the real post rows — score, comment count, subreddit, author, timestamp, full selftext — and the archive's ordering is remapped back to relevance order.

The distinction that matters: these are archive rows, not search snippets. Discovery only supplies the addresses. Responses carry source: "web_discovery" so you can tell which route served them.

Two limits, both reported rather than hidden: after/before cannot apply on the discovery route, and the response says the filter was ignored instead of returning results that look filtered; and an unscoped comment search has no backend at all now, so it asks for a subreddit or author scope rather than failing generically. PullPush is no longer tried automatically anywhere, including as the fallback for a scoped search — there it could only spend a request and bury the real Arctic Shift error behind a second failure. Passing source: "pullpush" still reaches it.

Anything still telling you PullPush is the one tool that can search across subreddits was written before August 2026.

Change Tracking That Actually Fires

track_changes had seven defects across this release, and the first one meant price monitoring — the tool's headline use case — did not work.

Price moves were scored by how much of the page they occupy. Significance was purely volumetric, so $19.99 → $29.99 and $19.99 → $99.99 both scored "minor", and unscoped the change did not register at all. With notificationThreshold defaulting to "moderate", a monitor set up the obvious way never fired. Monetary amounts are now compared directly and their relative magnitude raises significance to at least "moderate", or "major" at 20%+. Only currency-tagged numbers count, so view counters and review totals do not fire; thousands separators parse, so $1,299 reads as 1299. The pair is surfaced in details.valueChanges so you can see why a monitor fired.

customSelectors never scoped anything. It was read only inside section-level analysis, where it added hashes — so scoping a comparison made it worse. On an Amazon product page, scoping to the price block raised modified elements 456 → 3204 and payload 5.35 MB → 6.18 MB, and reported changes on a page whose price had not moved. Analysis now narrows the document to the matched subtrees, which scopes hashing, similarity and diffs together. A selector matching nothing falls back to the full document and warns rather than silently tracking nothing.

structuralSimilarity lied in both directions. It reported 0 when it had not measured at all — and zero is a real score meaning "the structure changed completely", so opting out of structural tracking produced the strongest possible signal that the structure had changed. It is null when not measured. Then in 5.2.1: the score could never fall below 0.5, because the hierarchy half of the average compared an object that was initialised empty and never written, returning a constant 1. A page rebuilt from the same tags in a completely different nesting scored a perfect 1.0 — precisely the case the metric exists to catch. It is now an element-count-by-depth histogram compared as a weighted Jaccard.

Diff payloads are bounded. line_diff embedded the full before+after document, because whitespace collapsing degenerates the diff into "remove everything, add everything". Both diffs now carry an explicit omittedEntries marker.

Three more surfaced in 5.2.5, while testing price tracking on Zillow and Newegg.

Scoping to anything but a handful of tags tracked nothing. Element-level analysis indexed a fixed allowlist — h1-h6, p, div, span, a — so a monitor scoped to address, td, li, tr or dd built a baseline of zero elements and could never report an element-level change. A Zillow page scoped to ['address'] built a baseline of 0 elements from 9 matching nodes. Matches outside the allowlist are now hashed too; allowlisted tags are skipped, so div-scoped counts are unchanged.

A scoped compare silently ran unscoped. compareWithBaseline discards the caller's tracking options in favour of the baseline's. That is correct and required for a valid diff — both sides must be analysed identically, and a scoped baseline no longer holds the full document to re-scope — but it said nothing, so a scoped compare returned results identical to an unscoped run with no way to tell. The ignored options now come back in a warnings array.

"Text content changed" appeared on compares that found no changes, because sub-threshold token noise still populates textChanges even when hasChanges is false. The summary now defers to significance.

Verified against the live Zillow page that exposed them: the baseline went from 0 to 9 elements, and a compare that had claimed true/"moderate"/28 modified now reports false/"none"/0 modified at 100% similarity.

Scraping Fidelity: Content That Was Being Deleted

Three scrape defects shared a root: the cleanup passes that strip invisible content were removing content that was visible.

Framework-streamed content was deleted. The hidden-content strip removed <div id="S:0" hidden>, which is where the Next.js App Router streams the rendered page. On a pricing page that wrapper is the whole visible page, so markdown came back empty and every price with it. The wrapper guard now takes the larger of the text and markup shares.

Script payload counted as page text. The bulk-removal guard sized elements with $('body').text(), which includes the source of every inline <script>. On a Shopify storefront the denominator was 62,269 characters of which 4,295 was visible copy — so a wrapper holding the entire product section measured under the threshold and was deleted along with the price, after which the json path had no price to extract and the model invented one. Both sides of the ratio now exclude script/style/noscript/template.

An LLM's schema echo was returned as data. The json format read only success from extract_with_llm and discarded the rest, so three failures reached callers looking like clean extractions: a schema document returned instead of page data (well-formed JSON, and with no required fields it passed validation too), output that failed validation, and input silently clipped at the 50,000-character cap. extract_with_llm now detects a schema echo, retries once, and fails with an actionable error; scrape surfaces schema-mismatch and truncation as warnings while keeping the data.

And in 5.2.2, the markdown format stopped returning raw HTML. turndown-plugin-gfm converts a table only when its first row is entirely <th>; every other table hits the plugin's keep filter and is emitted verbatim. Real pages are full of layout tables — Hacker News is built out of them — so scrape with formats: ["markdown"] and extract_text with output_format: "markdown" handed back <table> markup. Rules registered after the plugin now match those tables first and flatten them to their cell content. Tables with a genuine heading row still render as GFM pipe tables; we checked a Wikipedia data table before and after and it is byte-identical.

Local LLMs: Ollama Finally Works

Ollama was never registered as an LLM provider. LLMManager registered only OpenAI and Anthropic, both gated on an API key. So on a machine running Ollama with no cloud keys, extract_structured skipped LLM extraction entirely and reported css_fallback — producing values like "$79.99$79.99" and dropping fields — and deep_research silently disabled query expansion, semantic ranking and synthesis. extract_with_llm has its own private client, which is why it worked and masked the gap. A failed LLM call also no longer reports extraction_method: "llm" with confidence 0.9.

Model routing picks the best installed model instead of always llama3.2. Benchmarked against three live product pages with verified ground truth, gemma3:4b scored 18/18 at 1040 ms while llama3.2 scored 16/18 — and the failures are systematic, not sampling noise: over five runs llama3.2 invented a compare-at price on all five. Parameter count did not predict accuracy; the 4B model beat both a 12B and a 20B. selectOllamaModel() picks the highest-ranked installed model; OLLAMA_DEFAULT_MODEL still wins if you set it.

Remote Ollama endpoints work via OLLAMA_API_KEY. Every Ollama HTTP call sends Authorization: Bearer when the variable is set, so a hosted deployment can point at Ollama Cloud or any auth-fronted instance with no OpenAI or Anthropic key. Unset, nothing changes.

Browser Automation and the Stealth Wedge

scrape_with_actions carried seven Playwright API and error-recovery defects, all invisible to the suite because its fake pages implemented whatever the executor called — including APIs Playwright does not have:

• scroll toElement called scrollIntoView(), which does not exist on a handle or locator, so the branch threw every time it ran.
• The wait action advertised enabled/disabled/stable but passed them to an API that rejects them.
• The per-action Promise.race shared a deadline with the work it raced and won, replacing Playwright's real error with a bare "Action timeout" — and left a live timer per action.
• Clicks and keypresses did not wait on the document they replaced.
• A chain retry replayed against whatever the failed attempt left behind, never reloading.
• Every recovery strategy sat behind retries > 0 while the schema defaulted retries to 0, so none could ever run.

The stealth browser wedged itself. stealth_mode create_page never closed its page, leaking one Chromium renderer per call until the instance ran out of memory. A wedged browser was then reused forever behind truthiness-only checks, and cleanup hung on protocol calls to the dead browser, so it could not be unwedged remotely. There is now isConnected() corpse detection, a disconnected handler, cleanup racing closes against 5 s deadlines with a SIGKILL fallback and pool recreation, and an in-flight launch mutex.

Hosted images ignored the system Chromium. chromium.launch now honours PLAYWRIGHT_CHROMIUM_EXECUTABLE_PATH — the Dockerfile has set it all along, but Playwright never reads env vars itself — so browser paths stop dying on Alpine looking for a chrome-headless-shell that was never downloaded.

Language Detection and PDF Tables

Five languages were undetectable. analyze_content filters franc's output against a map of language names, and five keys in that map were ISO 639-2/B codes — chi, gre, ara, nor, msa — which franc, emitting ISO 639-3, never produces. The effect was silent and total: a page written entirely in Chinese returned null. So did Greek, Arabic, Norwegian and Malay. The codes are now cmn, ell, arb, nob and zlm/zsm.

A second bug sat behind it: franc scores whichever script is most common, and a Chinese or Japanese page carries the usual run of English product names and code samples, so it scored as Latin-script prose. Detection now short-circuits on a Han, kana or hangul share of at least 10% of letters — measured across live pages, Latin-script pages sit at 0% and genuine CJK pages at 24–51%.

PDF table extraction returned [] for every document ever passed to it. The hosted process_document route read TableResult.mergedTables, a field the PDF library declares but never writes; the tables live on result.pages[].tables. Against three live PDFs — the arXiv "Attention Is All You Need" paper, the IRS W-9 and the IRS 1040 — the field being read held 0 tables in every case while the per-page arrays held 7, 6 and 11. Alongside the fix, all-empty tables are dropped (they are drawn boxes — form outlines and figure borders — not data), each table reports its page number, the 20-table cap says how many it left out, and a table ruled only horizontally carries a note rather than reading as a genuine one-column table.

Infrastructure: Caches, Transport, Chromium

CACHE_DIR and CACHE_ENABLE_DISK did nothing. Both have always been exposed in config, but CacheManager read neither: the directory was hard-wired and the disk cache unconditionally on. Every test process therefore shared one directory that survived every run — 1,778 files had accumulated — and because crawl_deep keys on the crawled URL while test servers bind ephemeral ports the OS recycles, a run could be handed an earlier run's crawl of a different site. That was the cause of a rare parallel-run flake whose tell was in the timings: two failures took 13 ms and 5 ms for tests that perform two full crawls each. Defaults are unchanged — the variables simply do what they always claimed.

crawl_deep's cacheEnabled: false switched off only half the caching. BFSCrawler builds its own cache and stored every fetched page body with no way to turn it off, so a caller who explicitly asked for none was still served cached pages. Both caches also wrote to disk despite being described in-code as per-session; on disk, bodies outlived the crawler by an hour and crossed process boundaries. Both are memory-only now.

Task-capable tools failed over Streamable HTTP. cloneServerForSession dropped capabilities and the task store, so tools/call on agent, crawl_deep, batch_scrape or deep_research threw "No task store provided for task-capable tool."

One Copy of the Extractors

The MCP server and the CrawlForge REST API each carried their own copy of the template extractors, in two languages, with nothing detecting divergence — and it diverged twice in two days. amazon-product was repaired on the MCP side on 25 August while the REST copy kept returning a null rating and "Brand: Amazon" until the 26th, and shopify-product existed on one side only.

There is now one implementation, published as crawlforge-extractors, which both surfaces install. TemplateRegistry is re-exported with an unchanged API — nothing about scrape_template changes for callers. It is a runtime dependency, not a peer dependency, so npm install -g crawlforge-mcp-server pulls it in automatically; there is nothing extra to install.

Two more behaviours moved there because one surface had them and the other did not:

• readBody decodes a response using its own charset and refuses to buffer past a cap. The MCP server did both; the REST API called response.text(), which assumes UTF-8 — so pages served as Shift_JIS, GBK or ISO-8859-1 came back as mojibake — and would buffer a body of any size into a serverless function.
• structureSignature fingerprints a document as its tag vocabulary plus an element-count-by-depth histogram, small enough to store next to a change-tracking baseline, and takes an optional root so a caller tracking a CSS selector scores that region alone.

The alternative was a parity test that told us after the fact which copy was wrong. Deleting the second copy is cheaper than detecting drift in it.

Credit Costs

One price moved in this release. reddit_search went from 2 credits to 5, matching search_web.

The reason is the mechanism above: a Reddit-wide search now spends the same upstream search call search_web does, at the same cost to us. A scoped search still queries the archive directly and costs us nothing extra, but bills 5 as well, because the tool has one published price rather than a per-request one you would have to model. On the free plan's 1,000 one-time credits that is 200 Reddit searches.

Everything else is unchanged: scrape_template is 1 credit per call whichever template you use, fetch_url is 1, track_changes is 3, crawl_deep is 4, batch_scrape is 5 per URL. Failed requests are never charged.

How to Upgrade

npm install -g crawlforge-mcp-server@latest
crawlforge --version   # 5.2.5

If your MCP client launches the server with npx, it picks up the release on the next restart. 5.2.5 depends on crawlforge-extractors@^1.2.1 directly, so the youtube-video view-count fix comes with it.

There are no breaking changes: no tool was renamed, no output shape changed, and only reddit_search moved price. The one thing to check before upgrading is your budget maths if you call reddit_search in a loop — it is 5 credits now, not 2.

The same fixes are live on the hosted REST API, so there is nothing to install if you call CrawlForge over HTTP.

Want to try any of this without setting up a server? Start free with 1,000 credits — no credit card — then browse the API reference for all 28 tools. Related reading: Reddit API alternatives that still work in 2026 and extracting data with local LLMs and Ollama.