CrawlForge MCP v5.2.0: Shopify Product Data Without Parsing HTML

Every one of our amazon-product tests passed. The tool returned null for the price currency, null for the rating, null for every image, and the string "Brand: Amazon" where the brand should have been. Nothing was broken in CI, because the fixtures had been written to match the selectors instead of the site.

CrawlForge MCP v5.2.0 is largely a release about that. One new template, three tools that now report facts they always had, and a long run of fixes to things that were quietly wrong while their tests were green.

Table of Contents

• What Shipped
• The Bug That Passed Six Tests
• Shopify Prices Without Parsing HTML
• Templates Can Now Read an API
• Price Monitoring That Notices Prices
• Two Tools Report What They Always Knew
• Browser Automation Against Real Playwright
• Local LLMs Actually Work Now
• One Copy of the Extractors
• Credit Costs
• How to Upgrade

What Shipped

Tool count stays at 28. No schema, output-shape or credit-cost change to any existing tool — this is a drop-in upgrade. The suite is 1,122 unit tests, 1,121 passing, and MCP protocol compliance is 100% across all 28 tools.

The Bug That Passed Six Tests

amazon-product was the worst case, so it is worth being specific about how it failed.

Run against three live Amazon pages, the template returned null for currency, null for rating, null for images, null for breadcrumbs, "Brand: Amazon" for the brand and the literal string "(198,594)" for the review count. Six unit tests covered it. All six passed.

They passed because every selector they exercised — a priceCurrency meta tag, #acrPopover .a-size-base, img.a-thumbnail-image — exists nowhere on Amazon today, and the fixture HTML had been written to contain them. The tests were checking that our code agreed with itself.

The fix was to capture three real pages — a first-party device, a branded storefront and a book — and re-derive every selector from what Amazon actually serves:

• Currency comes from the hidden add-to-cart form field, not a meta tag.
• Rating is parsed to a number from the title attribute on #acrPopover, so you get 4.7 rather than "4.7 out of 5 stars".
• Review count is parsed from either "(198,594)" or "198,594 global ratings" — Amazon uses both — and returns 198594.
• Brand reduces all three byline shapes to the bare name.
• Images drop Amazon's size token from the URL. The tokened URL is a 1 KB thumbnail; the same URL without it is the 16 KB original.

The rewritten suite has 24 tests, and 15 of them fail against the pre-fix code. That is the property the old suite lacked.

Shopify Prices Without Parsing HTML

The headline addition is a shopify-product template, and it exists because every Shopify failure we hit this cycle came from parsing the rendered page.

Shopify's Dawn theme ships every price badge in the markup unconditionally and hides the inapplicable ones with component CSS. A scraper reading the DOM sees a "Sold out" badge on a product with a hundred units in stock. Ask an LLM for a compare-at price on a product that has none and it will produce a plausible one — in our testing it invented 27.99.

So the template does not read the page. Shopify serves the same data as JSON at /products/<handle>.json, on every storefront including custom domains, and that is what it reads:

{
  "tool": "scrape_template",
  "params": {
    "template": "shopify-product",
    "url": "https://shop.example.com/products/kelpie-bandana"
  }
}

You get the exact price, the compare-at price, on_sale, currency, the price range across variants, per-variant stock, options, images and tags — with no HTML parsing and no LLM anywhere in the path.

Storefronts differ in ways that only show up against real stores, and the template handles the ones we found in live captures:

• An absent compare-at price is "" on one store and "0.00" on another. Both read as null, because both render no badge — while a genuinely free product keeps its 0.00 price.
• Tags arrive as an array on some stores and a comma-joined string on others.
• The endpoint carries no available flag at all. Stock is derived from inventory management, inventory policy and quantity together, and reports null rather than guessing "in stock" when the payload does not say.

Point it at a site that is not Shopify and it fails with a clear message instead of returning a row of empty fields.

Templates Can Now Read an API

shopify-product needed something the template system could not do: fetch something other than the page the caller named. Rather than let one template open its own socket — which would have put it outside the SSRF guard — TemplateRegistry gained two optional hooks.

resolveUrl(url) redirects the tool's single fetch, and extractRaw(body, url) parses a non-HTML response. The tool still owns the fetch, the guard and the timeout. When a rewrite happens, the response reports the URL that was actually read as fetchedUrl, so nothing is hidden from the caller.

HTML templates are untouched. But the door is now open for any site that publishes structured data next to its rendered page, which is most storefront platforms.

Price Monitoring That Notices Prices

track_changes had a flaw that made its main use case not work.

Change significance was purely volumetric — scored by how much of the page had changed. A price is a handful of characters, so $19.99 → $29.99 and $19.99 → $99.99 both scored "minor". With notificationThreshold defaulting to "moderate", a price monitor configured the obvious way never fired. Unscoped, the change often did not register at all.

Monetary amounts are now compared as numbers, and their relative magnitude raises significance to at least "moderate" — or "major" past 20%. Only currency-tagged numbers count, so view counters and review totals do not trip it, and thousands separators parse, so $1,299 reads as 1299. The before-and-after pair is surfaced in details.valueChanges, so you can see why a monitor fired rather than guessing.

Two more in the same tool:

customSelectors never scoped anything. It was read only inside section-level analysis, where it added hashes. Scoping a comparison therefore made it worse: on an Amazon product page, scoping to the price block took modified elements from 456 to 3,204 and the payload from 5.35 MB to 6.18 MB — and reported changes on a page whose price had not moved. Analysis now narrows the document to the matched subtrees, so hashing, similarity and diffs are all scoped together. A selector that matches nothing falls back to the whole document and says so.

structuralSimilarity reported 0 when it had not measured anything. Zero is a real score meaning the structure changed completely, so opting out of structural tracking produced the strongest possible signal that the structure had changed. It is null when not measured.

Two Tools Report What They Always Knew

fetch_url returned status, headers, body, size and content type — enough to answer "is this URL up?" and nothing that could answer "how slow is it?", despite being the raw-HTTP tool with nothing else to fall back on. It now returns responseTime.

The measurement is more careful than a stopwatch around the call. It starts after the per-host politeness throttle, so a monitor polling one host in a loop does not read its own waiting as the site being slow, and it closes after the body is fully read, so a server that answers instantly and then trickles is correctly reported as slow. Live: example.com at 96 ms, our own health endpoint at 547 ms.

crawl_deep gains cached and crawled_at. A replayed crawl used to be indistinguishable from a fresh one; crawled_at now carries when the pages were really fetched. Both are declared in the tool's output schema.

The same two fields have landed on the hosted REST API this week, so a latency check reads the same on either surface:

curl -X POST https://www.crawlforge.dev/api/v1/tools/fetch_url \
  -H "X-API-Key: your_api_key" \
  -H "Content-Type: application/json" \
  -d '{"url": "https://example.com"}'

Browser Automation Against Real Playwright

scrape_with_actions carried seven defects, none of which its test suite could see — because its fake pages implemented whatever the executor called, including APIs Playwright does not have.

scroll toElement called scrollIntoView(), which exists on neither a handle nor a locator, so that branch threw every single time it ran. The wait action advertised enabled, disabled and stable and passed them to an API that rejects them. A per-action Promise.race shared its deadline with the work it was racing and won, replacing Playwright's real error with a bare "Action timeout" and leaking a timer per action. Clicks and keypresses did not wait on the document they replaced. A chain retry replayed against whatever the failed attempt had left behind instead of reloading. And every recovery strategy sat behind retries > 0 while the schema defaulted retries to 0, so not one of them could ever run.

The new suite drives a real Chromium against a local fixture server, and skips cleanly when no browser binary is installed.

stealth_mode had a matching problem in production rather than in tests: create_page never closed its page, leaking a Chromium renderer per call until the hosted instance ran out of memory. A wedged browser was then reused forever behind truthiness-only checks, and cleanup hung on protocol calls to the dead browser, so it could not be unwedged remotely either. v5.2.0 adds corpse detection via isConnected(), a disconnect handler, cleanup that races closes against 5-second deadlines with a SIGKILL fallback and pool recreation, and a mutex around in-flight launches.

Local LLMs Actually Work Now

LLMManager registered OpenAI and Anthropic and nothing else, both gated behind an API key. On a machine running Ollama with no cloud keys, extract_structured skipped LLM extraction entirely and quietly reported css_fallback — producing values like "$79.99$79.99" and dropping fields — while deep_research silently disabled query expansion, semantic ranking and synthesis. extract_with_llm has its own private client, which is why it kept working and masked the gap for so long.

Ollama is now a registered provider. A failed LLM call also stops reporting extraction_method: "llm" with confidence 0.9.

Model selection changed too. Routing hardcoded llama3.2. Benchmarked against three live product pages with verified ground truth, gemma3:4b scored 18/18 at 1,040 ms while llama3.2 scored 16/18 — and its failures were systematic rather than sampling noise: across five runs it invented a compare-at price all five times. Parameter count did not predict accuracy at all. The 4B model beat both a 12B and a 20B. selectOllamaModel() now picks the highest-ranked installed model, and OLLAMA_DEFAULT_MODEL still overrides it.

For hosted setups, every Ollama HTTP call now sends Authorization: Bearer when OLLAMA_API_KEY is set, so a deployment can point at Ollama Cloud or any auth-fronted instance with no OpenAI or Anthropic key at all. Unset, nothing changes.

One Copy of the Extractors

The last change in this release is structural, and it is the reason the amazon-product story could happen twice.

This server and the CrawlForge REST API each carried their own copy of the same eleven site extractors, written in two different languages, with nothing detecting divergence. It diverged twice in two days: amazon-product was repaired here and the REST copy kept returning nulls, and shopify-product existed on one side only.

There is now one implementation, published as crawlforge-extractors, which both surfaces install. TemplateRegistry is re-exported from it with an unchanged API — nothing about the scrape_template tool changes for callers. The per-template tests moved with it.

The alternative was a parity test that told us after the fact which copy was wrong. Deleting the second copy is cheaper than detecting drift in it.

Credit Costs

Nothing changed. scrape_template remains 1 credit per call regardless of which template you use, fetch_url is 1, and track_changes is 3. Failed requests are never charged.

The free plan includes 1,000 one-time credits, which is 1,000 template scrapes — enough to pull a full Shopify catalogue before deciding anything.

How to Upgrade

npm install -g crawlforge-mcp-server@latest
crawlforge --version   # 5.2.1 or newer

If your MCP client launches the server with npx, it picks up the release on the next restart. There are no breaking changes: no tool was renamed, no output shape changed, and no credit cost moved.

Update: 5.2.1 followed the same day. Two of the fixes above had reached the hosted server on release day but not the npm tarball — shopify-product was missing from the tool description an MCP client reads when it picks a template, and structuralSimilarity could not score below 0.5 because the hierarchy half of it compared an object nothing ever wrote to. Take @latest.

Want structured product data without writing a selector? Start free with 1,000 credits, then read the scrape_template API reference for the full template list and field-by-field output.