# CrawlForge MCP - Full Documentation
> Enterprise web scraping MCP server with 30 specialized tools for AI applications

## Overview

CrawlForge MCP is a Model Context Protocol server providing 30 web scraping and data extraction tools. It integrates natively with Claude, Cursor, LangChain, and any MCP-compatible AI framework.

### v6.5.0 Highlights

The current release is 6.5.0, with **30 tools and 18 scrape templates**. `extract_embedded_state`, the 29th tool, returns the JSON a page already ships inside its own HTML — Next.js `__NEXT_DATA__` and React Server Component payloads, Nuxt, Apollo and Redux stores, plus inline JSON-LD — so a scraper reads the site's own data structures instead of re-parsing rendered markup. The template catalogue covers Shopify storefronts (product and collection), Amazon, GitHub, YouTube, Reddit threads, Hacker News, Product Hunt, Stack Overflow, npm, the Greenhouse, Lever, Ashby, Workable, Recruitee and Teamtailor job boards, NHTSA VIN decode and the NPI provider registry; `linkedin-profile` and `tweet` were retired in 5.5.0 because both sites' robots.txt disallow every path for generic agents, so naming either now returns the reason, fetches nothing and costs nothing. The 5.6 line fixes everything four live regressions across all 29 tools found: `scrape` had been dropping inline links on pages whose CSS uses native nesting — the hidden-content stripper read a nested `a{display:none}` as a top-level rule and deleted 56 of one article's 68 anchors — and `batch_scrape`, `crawl_deep`, `generate_llms_txt` and `track_changes` now decode non-UTF-8 pages through the same charset-aware reader the single-page tools already used, so an ISO-8859-1 page no longer arrives as mojibake. Later 5.6 patches keep data tables intact — `extract_content`, `process_document` and `scrape_with_actions` re-attach the tables Readability drops, and a selector that matches a table in `scrape_with_actions`, `batch_scrape` or `scrape_structured` returns one line per row with cells joined by ` | ` instead of every cell run together — send an empty `Accept-Language` on page fetches (the header Amazon's captcha keyed on), make LLM extraction answer null rather than invent a value, report a Cloudflare, DataDome or PerimeterX challenge from `stealth_mode` as a block rather than a scrape, and read the `amazon-product` price from the buy box only, never a carousel item. 5.6.7 changes how a client picks a tool rather than what a tool returns: the server's instructions are a decision ladder with credit costs and a never-re-fetch rule, every description says when not to use the tool and what it costs, `scrape`, `search_web` and `deep_research` load at session start, and every error result ends with a `Next step:` line naming the tool to try — a 30-day log review had found one call in five repeating a call already made. 5.6.8 closes a fifth live round: every fetch path decodes a page by its declared charset (Shift_JIS sites such as kakaku.com had arrived as mojibake from `scrape` while `batch_scrape` read them), the `agent` tool checks every version, date and count in its answer against the fetched sources and names any it cannot back up, `stealth_mode` reports a Vercel Security Checkpoint as a block, keeps a Chromium context alive across a camoufox scrape and spoofs hardware inside Web Workers, and `analyze_content` segments Hindi, Finnish and Japanese into words instead of clauses. 5.6.9, from a pricing sweep across travel, auto and retail sites, makes `stealth_mode` report an HTTP error page or a titled soft block as a failure carrying its status, wait up to 8 s for a blank document to paint, and carry that verdict into `scrape_with_actions`; `crawl_deep` fetches URLs with their trailing slashes; `deep_research` searches a 12-word form of a long topic; `localization` sets Accept-Language for the configured country; and the `shopify-product` template reads the product page's JSON-LD when the store refuses its JSON endpoint. 5.6.10 reads that Shopify fallback from the shared crawlforge-extractors package the REST API uses too. 5.6.11 makes plain `scrape` report a challenge wall (Cloudflare, Amazon, DataDome, PerimeterX, Akamai, Vercel) whatever its HTTP status, and an empty JavaScript shell or a short error page served as HTTP 200, as `success: false` with `blocked.vendor` on both the MCP server and the REST API, reports the client's real version in usage records, and masks text typed into forms by `scrape_with_actions` before the usage report leaves the machine. 5.7.0 adds two query-scoped formats to `scrape`: a `highlights` format with a `query` returns the sentences, table rows and code blocks that match it, and a `question` format returns an answer assembled from them — both verbatim page text with an `offset` and `length` into the `markdown` of the same call and no model in the path, for 1 credit on top of the scrape, unlike summarising fetch tools that paraphrase a page and invent what they cannot find; `mode: "model"` is opt-in on the MCP server for 3 credits more, with a grounding check that flags any number or proper noun the evidence does not contain, while the REST API serves both formats in extractive mode and rejects model mode with 400. 5.8.0 adds result handles: the large-output tools (`scrape`, `fetch_url`, `extract_content`, `crawl_deep`, `batch_scrape`, `stealth_mode` scrape, `scrape_with_actions`, `process_document`, `deep_research`, and `extract_embedded_state`, which is never truncated but still offers a handle) take `max_inline_chars` (default 40,000), and a result whose JSON exceeds it comes back as a `preview`, a `result_handle`, `total_chars`, `truncated: true` and `expires_at` instead of a body that floods the context; the 30th tool, `read_result` (1 credit), slices, searches, paginates by line or reads a JSON path from the stored result — on the MCP server the store is on the customer's own machine under `~/.crawlforge/results/` (1-hour TTL, 200 MB LRU, nothing uploaded), on the REST API a per-account value with a 1-hour TTL — and `batch_scrape` jobs share the same store. 5.9.0 adds an opt-in `escalate` to `scrape`: the plain fetch still runs first, and only when the page comes back as a bot wall — a Cloudflare, Amazon, DataDome, PerimeterX, Akamai or Vercel challenge, an empty JavaScript shell or a short error-titled placeholder — does the same call render it once in the stealth browser and derive every requested format from that render, instead of returning a block that costs a second, separate call; the 5-credit escalation stage is charged only when it ran, so a call that asked to escalate and was served the page by the plain fetch still pays the base 2, robots.txt is respected on the escalated path against the same `CrawlForge` product token, and on the MCP server a host that walled a request is remembered for 24 hours so the next `escalate: true` call to it skips the doomed plain fetch. A wall the stealth render is refused at too is reported as a block carrying `escalated: true` and charged nothing: the feature saves a round trip and a second call, it does not promise the page, and it is most reliable where the plain fetch failed only because the page needs JavaScript to render. 5.10.0 adds two parameters rather than any new tool. `search_web` takes `queries`, between one and ten searches in a single call as an alternative to `query` — exactly one of the two is required — each running the full pipeline a single search runs; the payloads come back as `results_by_query`, a list in the order they were sent with one entry per query, so a repeated query appears twice and a query that failed carries `error` in place of its results. A `search_web` call costs 5 credits for every query that returned results — a single `query` costs 5, and a `queries` batch is projected at 5 x the number of queries sent, which is a ceiling: each query that failed is not charged, so a batch in which none succeeded costs nothing. Both surfaces bill this identically, and a batch in which none succeeded is a 200 carrying the normal envelope — `count` as sent, one `{ query, error }` entry per query, and `credits_used: 0` — rather than an error; the single-`query` path is unchanged, and one failing search still errors as it always did. A single-`query` call is byte-identical to what it was before. `redact_pii` removes personal data from the text nine tools return — `scrape`, `extract_content`, `extract_text`, `batch_scrape`, `crawl_deep`, `stealth_mode` on `operation: "scrape"`, `scrape_with_actions`, `process_document` and `search_web` — across four regex classes (`EMAIL`, `PHONE`, `FINANCIAL` with card numbers validated by Luhn and IBANs by mod-97, and `SECRET` for API keys, bearer tokens and labelled passwords), replaced as a tag, a mask or nothing, with `SECRET` keeping its label and losing only its value in every style. It runs before the result is stored rather than after, so an oversized result read back later with `read_result` is already redacted, and the report — `redaction: { entities, count, mode }`, at `result.redaction` on the MCP server and inside `data` on the REST API — is present whenever redaction was asked for, `count: 0` included, so "found nothing" is distinguishable from "the parameter was ignored". The default `fast` mode is regex over text the call already fetched and adds nothing; `mode: "model"` covers `PERSON` and `LOCATION` for 3 credits once per call on the MCP server only, dropping the charge when no model answered, and is a 400 charged nothing on the REST API, which runs no LLM. Counters derived from the text (`content_length`, `word_count`, `character_count`) describe it before redaction, and address-like fields (`url`, `link`, `href`, `canonical_url`) are deliberately never redacted. Earlier in the 5.5 line, `scrape` at its default `onlyMainContent` had been deleting every code example on Nextra-built documentation sites — 12 `<pre>` blocks and 96 inline `<code>` spans on one page — while leaving the prose intact, so the result read as a complete page rather than a failure; `stealth_mode` silently served Chromium when `engine: "camoufox"` was requested and a browser was already running; `extract_structured` could return a version number that is not on the page, now caught by a provenance guard over digit-bearing literals with one full-text retry; `localization`'s `localize_search` runs the search instead of returning a config; and `reddit_search` gained an unscoped comment search. The same line moved every browser-navigation path behind the SSRF guard, bound `--http` to loopback by default, and fenced scraped page text with a per-call nonce before it reaches a model CrawlForge itself calls (`extract_with_llm`, `summarize_content`, and the `agent` tool's synthesis step) — mitigation rather than a solution, with the full trust model in docs/SECURITY.md. 6.0.0 is a major, for three reasons a caller can see. The MCP server now speaks the 2026-07-28 protocol revision statelessly on the same `/mcp` endpoint that already served the 2025 era — no handshake, no session id, `server/discover` in place of `initialize` — with the era decided by the MCP SDK's own classifier rather than a rule of ours, so the endpoint can never disagree with the SDK about a borderline request; header/body mismatches and a non-JSON content type are refused before the tool runs, so a rejected call is never billed, and authentication is unchanged and identical on both eras. This is the HTTP endpoint only: a stdio connection, which is how `npx crawlforge-mcp-server` and every desktop client connects, still negotiates the 2025 era and nothing about it changes. The `task` parameter and async task mode are removed from `crawl_deep`, `batch_scrape`, `deep_research` and `agent` — they were built on the SDK's experimental tasks API and SEP-2663 deleted it outright rather than moving it, so the four tools now run synchronously, which is what every caller who never passed `task` already got; `batch_scrape`'s async webhook mode and the `result_handle` / `read_result` pattern cover long work that should not hold a connection open. The `--legacy-http` flag and the v3.1 stateless mode it gated are removed, four minor versions after its own warning said they would be; 2025-era clients are unaffected. And the hosted HTTP endpoint now returns the same `tools/list` a stdio client has always had — it had never applied the protocol-hygiene pass, so it served tools in registration order with no icons and no schema dialect, which means output is now reordered and gains both. 6.1.0 makes the confirmation prompts work on 2026-07-28-era clients. The five tools that ask before an expensive run — `crawl_deep` over 500 pages, `batch_scrape` in sync mode over 25 URLs, `deep_research` over 50 URLs, `agent` on the pro model, and `extract_structured` with no LLM configured and more than three required fields — plus the low-credit warning, were built on the 2025-era inline `elicitation/create` request, and the 2026-07-28 revision has no server-to-client request channel at all, so on that era the prompts silently did not fire and the operation proceeded unasked; each is now a multi-round-trip `input_required` return, which the SDK serves on both eras, so the prompt reaches the user either way. A confirmation round trip is never billed: a call that returns asking for confirmation has done no work, so it costs zero credits and produces no usage record, the charge is taken once when the call actually completes, and declining costs nothing. The same release fixes elicitation over HTTP, which had never worked — both HTTP legs serve from a cloned server instance and only a clone is ever connected, so the helper read undefined client capabilities on every HTTP request and every session proceeded unasked, broken since v3.2.0 and failing safe the whole time; the transport now stamps the serving clone on the request context. That is the hosted HTTP endpoint only, and stdio was never affected. In CI, the `.mcpb` bundle smoke test no longer sets a placeholder API key. 6.2.0 gives the MCP server's scheduled monitors a hosted kind: `scheduledMonitorOptions: { hosted: true }` on `create_scheduled_monitor` registers the monitor with the hosted service at `/api/v1/monitors` using the configured API key and returns the hosted id and a dashboard URL, after which it is the same hosted monitor the REST API and the dashboard manage — CrawlForge's own scheduler fetches, diffs and records every check and notifies by email (`notificationOptions.email.recipients`) and signed webhook (`notificationOptions.webhook`) with nothing running on the customer's side, the interval converted to a cron with a 5-minute floor, timezone UTC — while a local monitor, `create_scheduled_monitor` as before, is persisted under `~/.crawlforge/monitors`, fires in-process only while the MCP server process is alive, catches up missed runs on restart, is guaranteed only by `crawlforge monitor:run-due` from system cron, notifies by webhook or Slack from the customer's machine and never sends email; `list_scheduled_monitors` lists both kinds with a `hosted` flag, `stop_scheduled_monitor` deletes either, a hosted create or stop from the MCP server costs nothing like the REST monitors API while a local create still costs 3 credits, a local monitor with email notification settings now reports an error pointing at `hosted: true` instead of the fake success its placeholder email path used to return, the `track_changes` input schema is declared once and shared by the server registration and the tool with no behaviour change, and `AlertNotificationSystem.js`, 601 lines imported by nothing, is deleted — its live counterpart is the notifier module, and the hosted service sends the email and signed webhooks. 6.3.0 closes the R19 live sweep: `extract_structured` validates nested shapes (an array of stray page text no longer passes a schema of objects; one validator in `src/utils/schemaValidate.js` serves every consumer, and `success` is false when a required field is present but the wrong shape), its output budget scales with the schema's array-ness with one retry at twice the budget so the workable table range goes from roughly 43 rows to roughly 156, `agent` will not answer a version question from a discussion page and flags what it cannot back up in `provenance.unsupported_versions`, and `analyze_content` reports `notApplicable` for entities and readability on non-Latin text instead of inventing them. 6.3.1 fixes `crawlforge login` on Linux, where one wrong-case import had crashed the command since 6.1.0. 6.4.0 closes the R20 sweep over retail, travel and aviation sites: `map_site` reads sitemaps written with relative `<loc>` paths (boeing.com's 1,878-URL sitemap had parsed as empty and the tool returned 75) and never caches or serves an empty parse; `scrape` recovers a dropped table whose author marked `<th>` header cells whatever its size, so a small fee table survives the default `onlyMainContent`, and renders a table with an empty corner cell as a pipe table; `scrape_with_actions` serves `markdown` and `html` from the post-action DOM instead of a placeholder when Readability finds no article; `agent` adds the task's own words as a second query when a current-state plan stops at the bare entity name; `crawl_deep` content is the page's main content rather than its menu; `get_batch_results` takes `max_inline_chars`; `extract_links` skips `javascript:` pseudo-links; and `reddit_search` names the caller's window when Arctic Shift times out inside it. 6.5.0 closes the R21 sweep across thirteen verticals: `process_document` reads DOCX (mammoth) and routes a PDF served under `sourceType: "url"` to the PDF parser, refusing an archive or image by name; `scrape`'s hidden-content pass skips print stylesheets, reads inline styles per element rather than as universal rules and splits selector lists on top-level commas only, so irs.gov's tax tables return; a truncated `scrape` keeps `highlights`, `answer` and `json` inline; `track_changes` registers a text-only document that grew by a record; `extract_structured` keeps the complete rows of a cut-off response as `partial: true` and names the real fallback reason; `map_site` scopes to the seed's path and ranks by the search terms in each URL; `extract_embedded_state` resolves a bare path inside the page's only payload; a failed `scrape_with_actions` chain publishes its error screenshot as a resource instead of inline base64; and a long robots.txt Crawl-delay is named in the response. Install or upgrade with `npm install -g crawlforge-mcp-server@latest`.

- Website: https://www.crawlforge.dev
- Documentation: https://www.crawlforge.dev/docs
- API Reference: https://www.crawlforge.dev/docs/api-reference
- OpenAPI specification (3.1): https://www.crawlforge.dev/openapi.json
- TypeScript SDK (npm install crawlforge-sdk): https://github.com/mysleekdesigns/crawlforge-sdk-ts
- Python SDK (pip install crawlforge): https://github.com/mysleekdesigns/crawlforge-sdk-python
- Pricing: https://www.crawlforge.dev/pricing
- Getting Started: https://www.crawlforge.dev/docs/getting-started
- Concise Summary: https://www.crawlforge.dev/llms.txt

### Localized Versions

CrawlForge is available in English (default), Spanish, Simplified Chinese, and Malay. The site mirrors the English structure under locale path prefixes:

- English (default): https://www.crawlforge.dev/
- Español: https://www.crawlforge.dev/es
- 简体中文 (zh-Hans): https://www.crawlforge.dev/zh
- Bahasa Melayu (ms): https://www.crawlforge.dev/ms

Marketing, pricing, legal, documentation, and blog pages are translated; only the newest blog posts stay English-only until their translations land. Every page declares hreflang alternates; the XML sitemap at https://www.crawlforge.dev/sitemap.xml lists all localized URLs.

## Authentication

All API requests require an API key sent via the `X-API-Key` header.

```
curl -X POST https://www.crawlforge.dev/api/v1/tools/fetch_url \
  -H "X-API-Key: cf_live_YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"url": "https://example.com"}'
```

API key formats:
- `cf_test_...` - Test/development environment
- `cf_live_...` - Production environment

## Pricing Plans

| Plan | Monthly | Yearly (2 months free) | Monthly Credits |
|------|---------|------------------------|----------------|
| Free | $0 | - | 1,000 one-time credits |
| Hobby | $19/mo | $190/yr | 5,000 |
| Professional | $99/mo | $990/yr | 50,000 |
| Business | $399/mo | $3,990/yr | 250,000 |
| Enterprise | Custom | Custom | Unlimited |

Yearly plans receive the full year's credits when the yearly invoice is paid; unused credits roll over while the subscription is active.

One-time credit packs available from $3 (1,000 credits) with no subscription.

Credit add-ons:
- 1K credits: $3
- 5K credits: $14
- 10K credits: $27
- 25K credits: $62.50

## Tools Reference

### fetch_url
- **Cost:** 1 credit
- **Description:** Fetch and return the raw content of a webpage
- **Parameters:**
  - `url` (string, required): The URL to fetch
  - `timeout` (number, optional): Request timeout in ms (default: 30000)
  - `headers` (object, optional): Custom request headers
- **Use when:** You need the raw HTML or text content of a single page

### extract_text
- **Cost:** 1 credit
- **Description:** Extract clean, readable text from a URL, stripping HTML tags and boilerplate
- **Parameters:**
  - `url` (string, required): The URL to extract text from
  - `include_links` (boolean, optional): Include hyperlink URLs in output
- **Use when:** You need human-readable text content without HTML markup

### extract_links
- **Cost:** 1 credit
- **Description:** Discover and return all links found on a webpage
- **Parameters:**
  - `url` (string, required): The URL to extract links from
  - `filter` (string, optional): Regex pattern to filter links
  - `include_external` (boolean, optional): Include external links (default: true)
- **Use when:** You need to discover navigation structure or find specific URLs on a page

### extract_metadata
- **Cost:** 1 credit
- **Description:** Extract page metadata including title, description, Open Graph, and structured data
- **Parameters:**
  - `url` (string, required): The URL to extract metadata from
- **Use when:** You need SEO data, social media tags, or structured data from a page

### scrape_structured
- **Cost:** 2 credits
- **Description:** Extract specific data from a page using CSS selectors
- **Parameters:**
  - `url` (string, required): The URL to scrape
  - `selectors` (object, required): Map of field names to CSS selectors
  - `multiple` (boolean, optional): Return multiple matches per selector
- **Use when:** You need specific data points from a known page structure (prices, titles, etc.)

### scrape_template
- **Cost:** 1 credit
- **Description:** Run a pre-built scraper template for 18 popular sites and APIs (Shopify storefronts (product and collection), Amazon, GitHub, YouTube, Reddit, Hacker News, Product Hunt, Stack Overflow, npm, the Greenhouse, Lever, Ashby, Workable, Recruitee and Teamtailor job boards, NHTSA VIN decode and the NPI provider registry)
- **Parameters:**
  - `url` (string, required): URL on the supported site
  - `template` (string, required): Template id, hyphenated. One of: "shopify-product", "shopify-collection", "amazon-product", "github-repo", "youtube-video", "reddit-thread", "hacker-news-front-page", "producthunt-launch", "stackoverflow-question", "npm-package", plus the job-board and government list connectors (template "list" enumerates them). linkedin-profile and tweet are retired: those sites' robots.txt disallow every keyless path
  - `fields` (string[], optional): Subset of template fields to return
- **Use when:** Target site is one of the supported templates and you want clean structured data without writing selectors. reddit-thread reads the post from the Arctic Shift archive (reddit.com blocks direct scraping); use reddit_search for the comment tree

### extract_content
- **Cost:** 2 credits
- **Description:** Smart content extraction that identifies and returns the main content of a page
- **Parameters:**
  - `url` (string, required): The URL to extract content from
  - `format` (string, optional): Output format - "markdown", "text", or "html" (default: "markdown")
- **Use when:** You need the primary article/content from a page, ignoring navigation and ads

### extract_embedded_state
- **Cost:** 2 credits
- **Description:** Return the JSON state a page already ships in its own HTML - __NEXT_DATA__, RSC flight chunks, __NUXT__, __APOLLO_STATE__, __INITIAL_STATE__, __PRELOADED_STATE__ and application/json blocks. No LLM in the extraction path, so values cannot be fabricated
- **Parameters:**
  - `url` (string, required): The URL to read embedded state from
  - `path` (string, optional): Return one subtree, e.g. "next_data.props.pageProps" - dotted keys and array indexes, not JSONPath
  - `user_agent` (string, optional): Override the User-Agent sent to the target
  - `respect_robots` (boolean, optional): Honour the target's robots.txt (default: true)
  - `timeout` (number, optional): Fetch timeout in ms, 1000-60000 (default: 20000)
- **Use when:** A page renders its data client-side, or you need exact prices and IDs with no model in the loop

### map_site
- **Cost:** 2 credits
- **Description:** Map the structure of a website by crawling and analyzing its link structure
- **Parameters:**
  - `url` (string, required): The starting URL to map from
  - `max_depth` (number, optional): Maximum crawl depth (default: 3)
  - `max_pages` (number, optional): Maximum pages to visit (default: 100)
- **Use when:** You need to understand a website's structure before targeted scraping

### process_document
- **Cost:** 2 credits
- **Description:** Process PDF and other document formats, extracting text and metadata
- **Parameters:**
  - `url` (string, required): URL of the document to process
  - `extract_tables` (boolean, optional): Attempt to extract tables (default: false)
  - `extract_images` (boolean, optional): Extract image descriptions (default: false)
- **Use when:** You need to extract content from PDFs, Word docs, or other document formats

### search_web
- **Cost:** 5 credits per query that returned results. A 10-query `queries` batch is *projected* at 50 — a ceiling, not the charge — and each query that failed comes off it, so a batch in which none succeeded costs nothing. Identical on both surfaces
- **Description:** Search the web through Google Custom Search and return ranked results. One query per call, or up to 10 in a single call with `queries`
- **Parameters:**
  - `query` (string, optional): Search query. Exactly one of `query` or `queries` is required
  - `queries` (string[], optional): 1-10 queries to run in one call, as an alternative to `query`. The response carries `queries`, `count` and `results_by_query` — a list in input order, one entry per query, each with its own `query` plus that search's fields, or `query` and `error` when that one search failed. A repeated query runs and is billed twice. A single-`query` call is unchanged and carries none of these three fields
  - `limit` (number, optional): Results to return, 1-100 (default: 10). Google caps a single request at 10
  - `offset` (number, optional): Zero-based index of the first result (default: 0). Each page is a separate search
  - `lang` (string, optional): ISO 639-1 language code
  - `site` (string, optional): Limit to one domain, appended as `site:<domain>`
  - `safe_search` (boolean, optional): Enable Google SafeSearch
  - `time_range` (string, optional): "day", "week", "month", "year" or "all" (default: "all")
  - `file_type` (string, optional): Limit to one extension, appended as `filetype:<ext>`
  - `redact_pii` (boolean | object, optional, default false): Remove personal data from the titles, snippets and related searches before they are returned. See the shared `redact_pii` section below
- **Use when:** You need to find relevant web pages for a topic or query, or for several topics at once

### serp_rank
- **Cost:** 5 credits
- **Description:** Check where a domain ranks in Google's organic search results for a keyword — the real SERP position, not Custom Search order. Returns the target's organic rank, the ranking URL, and every position it holds. Powered by DataForSEO.
- **Parameters:**
  - `keyword` (string, required): The search query to check ranking for
  - `target` (string, required): Domain or URL to locate in the results
  - `depth` (number, optional): How many results to scan, 10-200 (default: 100; 100 = 1 page of cost)
  - `device` (string, optional): Device to emulate - "desktop" or "mobile" (default: "desktop")
  - `location_name` (string, optional): Location, e.g. "United States"
  - `location_code` (number, optional): Numeric DataForSEO location code (overrides location_name)
  - `language_code` (string, optional): Language code, e.g. "en"
- **Use when:** You need to track a domain's real organic Google position for target keywords

### reddit_search
- **Cost:** 5 credits
- **Description:** Search Reddit posts and comments or read a full thread — reddit.com blocks direct scraping, so this reads the Arctic Shift community archive (free, no Reddit credentials). Returns normalized posts/comments with full reddit.com permalinks, ISO dates, and text capped at 2,000 characters with truncation flags; thread mode returns the post plus its nested comment tree. Scoped searches (subreddit or author) query Arctic Shift directly (near-real-time); a Reddit-wide keyword search finds posts with a site-restricted web search and reads them from the archive by ID, since Arctic Shift cannot keyword-search across all of Reddit. PullPush stopped serving automated clients in August 2026 and is no longer used automatically. The self-hosted MCP package additionally supports source "reddit_api", reading Reddit's official Data API on your own app credentials (REDDIT_CLIENT_ID / REDDIT_CLIENT_SECRET); the hosted REST endpoint deliberately does not, because a shared server-side Reddit key would be commercial use under Reddit's terms.
- **Parameters:**
  - `query` (string, optional): Keyword search across posts or comments
  - `subreddit` (string, optional): Restrict results to one subreddit
  - `author` (string, optional): Restrict results to one Reddit user
  - `mode` (string, optional): "posts", "comments", or "thread" (default: "posts")
  - `link_id` (string, optional): Post ID to read as a full thread (mode "thread")
  - `after` (string, optional): Only results after this date
  - `before` (string, optional): Only results before this date
  - `limit` (number, optional): Maximum results to return
  - `sort` (string, optional): Sort order for results
  - `source` (string, optional): Archive backend to query
- **Use when:** You need Reddit discussions, community sentiment, or a full comment thread without Reddit API credentials

### scrape
- **Cost:** 2 credits; +1 credit when a `highlights` or `question` format is present (once per call, not per format); `mode: "model"` adds 3 more on the MCP server only — the REST API rejects it with 400 and charges nothing; +5 credits when `escalate: true` and the stealth stage actually ran, so the projected base + escalation is a ceiling, never a floor
- **Description:** Unified single-fetch, multi-format extraction. One page load returns any combination of markdown, HTML, raw HTML, text, links, metadata, screenshot, and JSON; a format that fails comes back as a per-format warning instead of failing the whole call. Two query-scoped formats, `highlights` and `question`, return only the sentences, table rows and code blocks that match a query — verbatim page text with an `offset` and `length` into the `markdown` of the same call, no model in the path. With `escalate: true` a plain fetch that comes back as a bot wall — a Cloudflare, Amazon, DataDome, PerimeterX, Akamai or Vercel challenge page, an empty shell or an error placeholder — is rendered once in the stealth browser inside the same call and every requested format is derived from that render, instead of returning a hint that costs a second call. robots.txt is respected on the escalated path too, and no bot defence is handled any differently than `stealth_mode` already does. A wall the stealth render is refused at too is still reported as a block carrying `escalated: true` and charged nothing: escalation saves the second round trip rather than promising the page, and is most reliable where the plain fetch failed only because the page needs JavaScript to render
- **Parameters:**
  - `url` (string, required): The URL to scrape
  - `formats` (array, optional): Formats to return, e.g. ["markdown", "links", "metadata"]; also the objects `{ "type": "highlights", "query": "…", "max_highlights": 1-50 (default 10), "mode": "extractive" | "model" }` — top-N units ranked by BM25 with phrase and heading boosts, each `{ text, kind: sentence | table_row | code_block, offset, length, score }` (`table_row` units come from the MCP server, whose markdown keeps tables as pipe rows; the hosted REST API flattens tables to text, so its units are `sentence` or `code_block`) — and `{ "type": "question", "question": "…", "mode": "extractive" | "model" }` — an `answer` of `{ text, grounded, evidence: [up to 5 units] }`; extractive mode joins the top evidence verbatim with `grounded: true`, model mode synthesises (Ollama first, then a server-side OpenAI or Anthropic key, then MCP sampling) and sets `grounded: false` with a warning if any number or proper noun in the answer is missing from the evidence
  - `onlyMainContent` (boolean, optional): Strip nav, ads, and boilerplate from text formats
  - `escalate` (boolean, optional, default false): Retry a walled page once through the stealth browser inside the same call, deriving every requested format from that render. The plain fetch always runs first, and the 5-credit add-on is charged only when the stealth stage ran. The response then carries `escalated`, plus `stealth: { engine, vendor_detected }` when it is `true`. On the MCP server only, a host that walled a request is remembered for 24 hours, so the next `escalate: true` call to it skips the doomed plain fetch and warns that it did. On the hosted REST API the stage needs the CrawlForge execution backend: where that is not configured it returns 503 and charges nothing. A wall the stealth render is refused at too is returned as a block with `escalated: true`, charged nothing
  - `escalate_engine` (string, optional, default "playwright"): Browser engine for the escalation stage, "playwright" or "camoufox" (only where installed on the backend); ignored when no escalation runs
- **Use when:** You need more than one representation of the same page — it replaces chaining fetch_url, extract_links, and extract_metadata with a single request; or you need only the passages that answer a question, quoted verbatim with a locator rather than summarised; or you want a bot-walled page retried in the same call instead of paying for a second, separate `stealth_mode` call

### get_batch_results
- **Cost:** 1 credit
- **Description:** Retrieve paginated results for a `batch_scrape` job by its `batchId`. Poll job status and page through completed results while the rest of the batch is still running
- **Parameters:**
  - `batchId` (string, required): Batch ID returned by batch_scrape
  - `page` (number, optional): Page number to retrieve
  - `limit` (number, optional): Results per page (1-100, default 25)
- **Use when:** You submitted an async batch_scrape and need to poll status or walk large result sets without holding the whole batch in memory

### read_result
- **Cost:** 1 credit
- **Description:** Read back a result that was too large to return inline. When a tool's JSON exceeds `max_inline_chars` (default 40,000; on scrape, fetch_url, extract_content, crawl_deep, batch_scrape, stealth_mode scrape, scrape_with_actions, process_document, deep_research and extract_embedded_state) the response carries `preview`, `result_handle`, `total_chars`, `truncated: true` and `expires_at`; read_result slices, searches, paginates by line or reads a JSON path from the stored result. Stored results are kept for 1 hour and are readable only by the account that created them
- **Parameters:**
  - `handle` (string, required): The result_handle from a truncated response
  - `operation` (string, required): slice, search, lines or json_path
  - `offset` (number, optional): slice: first character (default 0); lines: first line index
  - `length` (number, optional): slice: characters to return (default 10,000); lines: lines to return (default 200, max 5,000)
  - `query` (string, optional): search: case-insensitive literal substring, never a regex; required for search
  - `max_matches` (number, optional): search: 1-100, default 20
  - `path` (string, optional): json_path: the path to read; required for json_path
  - `max_inline_chars` (number, optional): json_path: largest value returned inline (default 40,000); a larger value returns value: null with a preview
- **Use when:** A tool response carries truncated: true and a result_handle — search for the heading you need, then slice at the match offset, instead of re-running the tool or fetching the page again

### agent
- **Cost:** 8 credits
- **Description:** Autonomous research and extraction from a natural-language prompt — no URLs required. The agent plans its own steps and finds and reads its own sources, bounded by hard caps
- **Parameters:**
  - `prompt` (string, required): What to research or extract, in plain English
  - `urls` (array, optional): Seed URLs to start from
  - `schema` (object, optional): JSON schema to shape a machine-readable answer
  - `model` (string, optional): LLM to reason with
  - `maxSteps` (number, optional): Step budget (hard cap 10)
  - `maxUrls` (number, optional): Fetch budget (hard cap 20)
- **Use when:** The question spans sites you have not identified yet and you want a synthesized answer rather than raw HTML

### crawl_deep
- **Cost:** 4 credits
- **Description:** Deep crawl a website, following links and extracting content from multiple pages
- **Parameters:**
  - `url` (string, required): Starting URL
  - `max_pages` (number, optional): Maximum pages to crawl (default: 50)
  - `max_depth` (number, optional): Maximum link depth (default: 3)
  - `include_patterns` (string[], optional): URL patterns to include
  - `exclude_patterns` (string[], optional): URL patterns to exclude
- **Use when:** You need content from multiple pages across a website

### summarize_content
- **Cost:** 4 credits
- **Description:** AI-powered content summarization with configurable length and style
- **Parameters:**
  - `url` (string, required): URL of content to summarize
  - `max_length` (number, optional): Maximum summary length in words
  - `style` (string, optional): Summary style - "brief", "detailed", "bullet_points"
- **Use when:** You need a concise summary of long-form content

### analyze_content
- **Cost:** 3 credits
- **Description:** Analyze content for sentiment, language, readability, and key topics
- **Parameters:**
  - `url` (string, required): URL of content to analyze
  - `analyses` (string[], optional): Types of analysis - "sentiment", "language", "readability", "topics"
- **Use when:** You need content intelligence (sentiment, topics, language detection)

### batch_scrape
- **Cost:** 5 credits
- **Description:** Process multiple URLs in a single request for efficient bulk scraping
- **Parameters:**
  - `urls` (string[], required): Array of URLs to scrape (max 100)
  - `extract` (string, optional): What to extract - "text", "metadata", "links", "content"
- **Use when:** You have multiple URLs to process and want to minimize API calls

### scrape_with_actions
- **Cost:** 5 credits
- **Description:** Scrape dynamic pages using browser automation (clicks, form fills, scrolling)
- **Parameters:**
  - `url` (string, required): Starting URL
  - `actions` (object[], required): Array of browser actions to perform
    - Action types: "click", "type", "scroll", "wait", "screenshot"
    - Each action has `selector` (CSS selector) and action-specific params
  - `extract_after` (boolean, optional): Extract content after actions complete
- **Use when:** Content requires interaction (login, infinite scroll, dynamic loading)

### deep_research
- **Cost:** 10 credits
- **Description:** Multi-stage AI research combining web search, content extraction, and synthesis
- **Parameters:**
  - `query` (string, required): Research query or topic
  - `depth` (string, optional): Research depth - "quick", "standard", "comprehensive"
  - `max_sources` (number, optional): Maximum sources to consult (default: 10)
  - `verify_sources` (boolean, optional): Cross-reference facts across sources
- **Use when:** You need thorough research on a topic with source verification

### track_changes
- **Cost:** 3 credits
- **Description:** Track content changes with baseline capture, scheduled monitoring, and alerts
- **Parameters:**
  - `url` (string, required): URL to track
  - `operation` (string, optional): "compare" | "create_baseline" | "monitor" | "get_history" | "get_stats"
  - `trackingOptions` (object, optional): Granularity, ignoreWhitespace, trackText, trackLinks, trackStructure
  - `monitoringOptions` (object, optional): Scheduled interval, webhook, notification thresholds
- **Use when:** You need to detect and track website changes over time

### Hosted monitors (REST API)
- **Cost:** no per-monitor fee. Each check holds 3 credits per target (the track_changes price) and keeps 3 for every target that was fetched and compared (`new`, `changed` or `unchanged`); `blocked` and `error` targets are not charged. Creating, listing, updating and deleting monitors costs nothing and works at zero credits. `estimated_credits_per_month` on the monitor = runs of the schedule in the next 30 days × targets × 3
- **Description:** A monitor is a name, up to 20 targets (`url` plus an optional CSS `selector`), a five-field cron `schedule_cron` (default `0 * * * *`, consecutive runs at least 5 minutes apart) evaluated in an IANA `timezone` (default `UTC`), and where to send results. CrawlForge's own scheduler runs every 5 minutes, so nothing has to run on the customer's side. The first check of a target captures a baseline (`new`); every later check diffs against the previous check and rolls the baseline forward. A challenge wall (Cloudflare, DataDome, PerimeterX, Akamai, Amazon, Vercel) or an unusable document is `blocked` with the vendor, never a change; robots.txt is always respected (`error` with `ROBOTS_DISALLOWED`); a monitor still running at its next slot records `skipped_overlap`; a balance that cannot cover the hold records `insufficient_credits` and fetches nothing. Baselines are kept until the monitor is deleted; checks are pruned after `retention_days` (1-365, default 30). Limit 50 monitors per account (409 `MONITOR_LIMIT_REACHED`)
- **Endpoints:** `POST /api/v1/monitors` (201 with the monitor, including `webhook_secret`), `GET /api/v1/monitors?limit&cursor` (with `last_check`, never `webhook_secret`), `GET` / `PATCH` / `DELETE /api/v1/monitors/{id}`, `POST /api/v1/monitors/{id}/run` (one check inline, 409 `MONITOR_RUNNING` while one is in progress), `GET /api/v1/monitors/{id}/checks?limit&cursor` (no pages), `GET /api/v1/monitors/{id}/checks/{check_id}` (with `pages` and webhook `deliveries`). Same `X-API-Key` authentication as the tools. `track_changes` with `operation: "monitor"` plus optional `schedule`, `notify_emails` and `webhook_url` creates a monitor from one URL for 0 credits
- **Notifications:** email to up to 5 `notify_emails` only when a check has new, changed, blocked or errored pages. Webhooks: `POST webhook_url` with `Content-Type: application/json`, `X-Webhook-Event` (`monitor.page`, one per page whose status is not `unchanged`, then `monitor.check.completed`), `X-Webhook-ID`, `X-Webhook-Timestamp` (unix ms) and `X-Webhook-Signature: sha256=<hex HMAC-SHA256 of the exact raw body keyed by webhook_secret>`; body `{ event, id, timestamp, data }`. The same header set and signature scheme as the MCP server's webhooks, so one receiver verifies both. Up to 4 attempts with 1 s / 2 s / 4 s backoff and a 10 s timeout; 2xx is delivered; a 4xx other than 408/429 is not retried
- **Docs:** https://www.crawlforge.dev/docs/api-reference/monitors
- **Use when:** You want pages checked on a schedule and to be told what changed, without running a scheduler of your own

### extract_structured
- **Cost:** 3 credits
- **Description:** LLM-powered schema-driven extraction with CSS selector fallback
- **Parameters:**
  - `url` (string, required): URL to extract from
  - `schema` (object, required): JSON Schema describing the target data shape
  - `prompt` (string, optional): Natural-language extraction instructions
  - `llmConfig` (object, optional): Provider and API key for LLM extraction
  - `selectorHints` (object, optional): CSS selectors guiding the fallback
- **Use when:** You want typed structured output without writing bespoke selectors

### generate_llms_txt
- **Cost:** 5 credits
- **Description:** Analyze a site and generate standard-compliant llms.txt and llms-full.txt
- **Parameters:**
  - `url` (string, required): Website to analyze
  - `format` (string, optional): "both" | "llms-txt" | "llms-full-txt"
  - `complianceLevel` (string, optional): "basic" | "standard" | "strict"
  - `analysisOptions` (object, optional): maxDepth, maxPages, detectAPIs, analyzeContent
  - `outputOptions` (object, optional): organizationName, contactEmail, customGuidelines, customRestrictions
- **Use when:** Publishing AI interaction guidelines for your site

### stealth_mode
- **Cost:** 5 credits
- **Description:** Anti-detection browsing with fingerprint randomization and proxy rotation
- **Parameters:**
  - `url` (string, required): URL to access in stealth mode
  - `proxy_country` (string, optional): Country code for proxy (default: "US")
  - `fingerprint` (string, optional): Browser fingerprint profile
- **Use when:** Target sites have anti-bot protection or block automated requests

### localization
- **Cost:** 2 credits
- **Description:** Access geo-targeted content from 26 countries with automatic localization
- **Parameters:**
  - `url` (string, required): URL to access
  - `country` (string, required): Target country code (e.g., "US", "GB", "JP", "DE")
  - `language` (string, optional): Preferred language code
- **Supported countries:** US, GB, CA, AU, DE, FR, ES, IT, NL, SE, NO, DK, FI, PL, CZ, AT, CH, BE, PT, IE, JP, KR, CN, IN, BR, MX
- **Use when:** You need location-specific content or pricing from different regions

### extract_with_llm
- **Cost:** 3 credits
- **Description:** LLM-powered extraction that returns structured data described in natural language. Defaults to local Ollama (no API key required); OpenAI and Anthropic are optional alternatives.
- **Parameters:**
  - `url` (string, required): URL to extract from
  - `prompt` (string, required): Natural-language description of the data to extract
  - `schema` (object, optional): JSON Schema describing the target shape
  - `provider` (string, optional): "ollama" (default) | "openai" | "anthropic"
  - `model` (string, optional): Provider-specific model name
- **Use when:** You want LLM-driven extraction without committing to a paid provider, or you need to swap providers per request

### list_ollama_models
- **Cost:** 1 credit
- **Description:** List the local Ollama models available to `extract_with_llm`. Useful for discovering installed models before running an extraction.
- **Parameters:** None
- **Use when:** You want to confirm which Ollama models are installed locally before calling `extract_with_llm`

### Shared parameter: redact_pii
- **Cost:** 0 credits in the default `fast` mode — regex over text the call has already fetched, no model. `mode: "model"` adds 3 credits once per call on the MCP server and is a 400 charged nothing on the REST API, which runs no LLM
- **Accepted by:** `scrape`, `extract_content`, `extract_text`, `batch_scrape`, `crawl_deep`, `stealth_mode` (on `operation: "scrape"` only), `scrape_with_actions`, `process_document`, `search_web`
- **Shape:** `boolean | { entities?: string[], replace_style?: "tag" | "mask" | "remove", mode?: "fast" | "model" }`. `true` is shorthand for `{ mode: "fast" }` — all four regex classes, tagged
- **Entities:** `EMAIL`, `PHONE`, `FINANCIAL` (card numbers validated with Luhn, IBANs with mod-97) and `SECRET` (API keys, bearer tokens, labelled passwords). Names are case-insensitive; omitted, empty or not an array means all four. `PERSON` and `LOCATION` are model-only and are accepted on the MCP server with `mode: "model"`; on the REST API every model-only name is rejected. A name the surface cannot redact is a 400 naming the accepted classes, never a silent no-op
- **Replace styles:** `tag` writes `<EMAIL>`, `mask` writes `[REDACTED]`, `remove` deletes the span. `SECRET` keeps its label and replaces only the value in every style, `remove` included, so a line still says which credential was there
- **Output:** `redaction: { entities, count, mode }` — at `result.redaction` on the MCP server, inside `data` on the REST API, deliberately the same access path. Present whenever redaction was requested, including `{ "entities": {}, "count": 0 }` when the page held nothing, so "found nothing" is distinguishable from "the parameter was ignored"; a class with no hits is omitted rather than reported as `0`. On the MCP server in model mode the object also carries `model_ran`, which never appears on the REST API
- **Ordering:** redaction runs before the result is stored, so a result fetched later with `read_result` is already redacted
- **Known limits:** counters derived from the text (`content_length`, `word_count`, `character_count`) describe it before redaction, and address-like fields (`url`, `link`, `href`, `canonical_url`) are deliberately never redacted

## Integration Examples

### Claude Desktop (MCP)

Add to your Claude Desktop MCP configuration:

```json
{
  "mcpServers": {
    "crawlforge": {
      "command": "npx",
      "args": ["-y", "crawlforge-mcp-server"],
      "env": {
        "CRAWLFORGE_API_KEY": "cf_live_YOUR_API_KEY"
      }
    }
  }
}
```

### Cursor IDE

Add to `.cursor/mcp.json`:

```json
{
  "mcpServers": {
    "crawlforge": {
      "command": "npx",
      "args": ["-y", "crawlforge-mcp-server"],
      "env": {
        "CRAWLFORGE_API_KEY": "cf_live_YOUR_API_KEY"
      }
    }
  }
}
```

### Claude Code

One command registers the server at user scope (docs: https://www.crawlforge.dev/docs/integration/claude-code):

```bash
claude mcp add crawlforge -s user -e CRAWLFORGE_API_KEY=cf_live_YOUR_API_KEY -- npx -y crawlforge-mcp-server@latest mcp
```

Or share it with a team through a project `.mcp.json`:

```json
{
  "mcpServers": {
    "crawlforge": {
      "command": "npx",
      "args": ["-y", "crawlforge-mcp-server@latest", "mcp"],
      "env": {
        "CRAWLFORGE_API_KEY": "cf_live_YOUR_API_KEY"
      }
    }
  }
}
```

### VS Code (GitHub Copilot agent mode)

Add to `.vscode/mcp.json` (VS Code uses `servers` and a `type` field; docs: https://www.crawlforge.dev/docs/integration/vscode):

```json
{
  "servers": {
    "crawlforge": {
      "type": "stdio",
      "command": "npx",
      "args": ["-y", "crawlforge-mcp-server@latest", "mcp"],
      "env": {
        "CRAWLFORGE_API_KEY": "cf_live_YOUR_API_KEY"
      }
    }
  }
}
```

### Zed

Add a `context_servers` block to `~/.config/zed/settings.json` (docs: https://www.crawlforge.dev/docs/integration/zed):

```json
{
  "context_servers": {
    "crawlforge": {
      "command": "npx",
      "args": ["-y", "crawlforge-mcp-server@latest", "mcp"],
      "env": {
        "CRAWLFORGE_API_KEY": "cf_live_YOUR_API_KEY"
      }
    }
  }
}
```

### Gemini CLI

Run `gemini mcp add -s user -e CRAWLFORGE_API_KEY=cf_live_YOUR_API_KEY crawlforge npx -- -y crawlforge-mcp-server@latest mcp`, or add to `~/.gemini/settings.json` (docs: https://www.crawlforge.dev/docs/integration/gemini-cli):

```json
{
  "mcpServers": {
    "crawlforge": {
      "command": "npx",
      "args": ["-y", "crawlforge-mcp-server@latest", "mcp"],
      "env": {
        "CRAWLFORGE_API_KEY": "cf_live_YOUR_API_KEY"
      }
    }
  }
}
```

### Windsurf

Add to `~/.codeium/windsurf/mcp_config.json` (docs: https://www.crawlforge.dev/docs/integration/windsurf):

```json
{
  "mcpServers": {
    "crawlforge": {
      "command": "npx",
      "args": ["-y", "crawlforge-mcp-server@latest", "mcp"],
      "env": {
        "CRAWLFORGE_API_KEY": "cf_live_YOUR_API_KEY"
      }
    }
  }
}
```

### Cline

Cline panel → MCP Servers → Configure → Configure MCP Servers opens `cline_mcp_settings.json`; add (docs: https://www.crawlforge.dev/docs/integration/cline):

```json
{
  "mcpServers": {
    "crawlforge": {
      "command": "npx",
      "args": ["-y", "crawlforge-mcp-server@latest", "mcp"],
      "env": {
        "CRAWLFORGE_API_KEY": "cf_live_YOUR_API_KEY"
      },
      "disabled": false,
      "autoApprove": []
    }
  }
}
```

### Configurator and discovery

- MCP configurator (fills in your key and emits the exact config for every client above): https://www.crawlforge.dev/docs/integration/configure
- Agent onboarding (SKILL.md): https://www.crawlforge.dev/agent-onboarding/SKILL.md
- MCP discovery file: https://www.crawlforge.dev/.well-known/mcp.json

Set `CRAWLFORGE_TOOL_GROUPS` (for example `basic,search`) in the same `env` block to load only some of the 30 tools.

### TypeScript / Node.js

```typescript
const response = await fetch('https://www.crawlforge.dev/api/v1/tools/fetch_url', {
  method: 'POST',
  headers: {
    'X-API-Key': process.env.CRAWLFORGE_API_KEY!,
    'Content-Type': 'application/json',
  },
  body: JSON.stringify({ url: 'https://example.com' }),
});

const data = await response.json();
// { success: true, data: {...}, credits_used: 1, credits_remaining: 999 }
```

### Python

```python
import requests

response = requests.post(
    'https://www.crawlforge.dev/api/v1/tools/fetch_url',
    headers={
        'X-API-Key': 'cf_live_YOUR_API_KEY',
        'Content-Type': 'application/json',
    },
    json={'url': 'https://example.com'},
)

data = response.json()
```

## Rate Limits

| Plan | Requests/sec | Requests/min |
|------|-------------|--------------|
| Free | 1 | 60 |
| Hobby | 2 | 120 |
| Professional | 4 | 240 |
| Business | 10 | 600 |

## Error Codes

| Code | Meaning |
|------|---------|
| 200 | Success |
| 400 | Bad Request - invalid parameters |
| 401 | Unauthorized - invalid API key |
| 402 | Payment Required - insufficient credits |
| 429 | Too Many Requests - rate limit exceeded |
| 500 | Internal Server Error |

## Support

- Documentation: https://www.crawlforge.dev/docs
- FAQ: https://www.crawlforge.dev/docs/support/faq
- Crawler Operating Rules (G1-G8): https://www.crawlforge.dev/docs/policy/acceptable-use
- Verifying the CrawlForge Crawler (User-Agent, Web Bot Auth, robots.txt): https://www.crawlforge.dev/docs/policy/crawler-verification
- Untrusted Content & Prompt Injection (trust model, what we fence, what we deliberately do not sanitise): https://www.crawlforge.dev/docs/policy/untrusted-content
- Security and Data Handling (data flow, retention periods, subprocessors, crawler identity, vulnerability disclosure): https://www.crawlforge.dev/security
- security.txt: https://www.crawlforge.dev/.well-known/security.txt
- Email: support@crawlforge.dev
- Discord: https://discord.gg/crawlforge

## Blog Posts
- **Competitor Monitoring API for RevOps: No Scraper to Run** (https://www.crawlforge.dev/blog/competitor-monitoring-api-for-revops): What a RevOps team gets from CrawlForge's hosted monitors instead of the usual scraper-plus-cron loop that nobody notices has died. One `POST /api/v1/monitors` takes up to 20 URLs (each an optional CSS `selector`), a five-field cron expression and a timezone, and CrawlForge's own scheduler — which wakes every five minutes — runs the checks, diffs each page against its previous check and rolls the baseline forward, so every result reads as "changed since last time". Results arrive as email or as signed webhooks: a `monitor.page` event per page that is not `unchanged` carrying `change_percent`, `added_count`, `removed_count` and up to 20 sample lines per side, then a `monitor.check.completed` summary. Covers the two headers a handler must use — `X-Webhook-Signature` (`sha256=` plus an HMAC-SHA256 of the raw body keyed with the monitor's `webhook_secret`, returned on create and single `GET` but never in a list) and `X-Webhook-ID`, stable across up to four delivery attempts, so one price change does not become four CRM tasks. Billing: no per-monitor fee, management calls free, 3 credits held per target per check (the `track_changes` price) and nothing charged for targets that come back `blocked` or `error`, with a worked table — five pages checked each morning is 450 credits a month, an hourly single-page monitor 2,160. States the limits plainly: `robots.txt` is always respected with no override (`ROBOTS_DISALLOWED`, never charged), consecutive runs must be five minutes apart, and CrawlForge holds no contact, firmographic or org-chart data — it is the monitoring layer above your sources, not a replacement for an enrichment database.
- **Competitor Monitoring API Pay As You Go: What a Check Costs** (https://www.crawlforge.dev/blog/competitor-monitoring-api-pay-as-you-go): What a pay-as-you-go competitor monitoring API meters, and why the billing unit decides what you are willing to watch. CrawlForge charges 3 credits — the `track_changes` price — per target it fetches and compares, and nothing else: creating, listing, updating, pausing and deleting monitors are zero-credit calls, there is no per-monitor or per-page fee, an account holds up to 50 monitors of 20 targets each, and each check holds credits up front then settles to what actually ran so targets returning `blocked` or `error` are never charged. `estimated_credits_per_month` is a ceiling, not a floor. At the $3-per-1,000 one-time pack rate a credit is $0.003, so one target compared once is about $0.009, with a worked 30-day table: five pricing pages daily is 450 credits (~$1.35), forty pages weekly is 480 credits (~$1.44), one page hourly is 2,160 credits (~$6.48) — frequency is what costs money, breadth is nearly free. Contrasts that with the subscription shape, using Visualping's plan grid verified live on 2026-09-09 (free: 150 checks across 5 pages hourly; Personal 1K $14/mo for 1,000 checks across 10 pages; Personal 5K $35 for 20 pages; Business 20K $140 for 200 pages and 5 seats) — two ceilings and a seat count, cheaper at low page counts (five pages daily fits their free plan) and much more expensive when breadth outruns frequency (200 pages weekly is ~800 checks, inside a $14 plan's check budget but 20x its page budget, so $140, versus 2,400 credits or about $7 metered). Names what a per-page subscription buys that an API does not — visual side-by-side diffs, Slack and Teams destinations, false-alert tuning, a login recorder — and states that CrawlForge holds no contact or firmographic data. Also covers what happens when the balance runs out (`insufficient_credits`: nothing fetched, nothing charged, the monitor stays scheduled) and the five-minute minimum between consecutive runs.
- **Shipping an MCP Server Inside Your Product: Five Decisions** (https://www.crawlforge.dev/blog/mcp-server-inside-your-product): The architecture behind embedding a web-data MCP server in a product you ship to customers, rather than adding one to your own editor — a case study on LightAudit Score, a free open-source Lighthouse console whose AI fixes cite pages it actually opened. Five decisions, each one a security-review question. (1) The research server is off by default and the feature degrades visibly without it: analyses fall back to audit data alone and carry a plain 'No web research' badge, stamped with the provider and model, and never invent a source. (2) Ten of CrawlForge's twenty-nine tools are granted — search, fetch, extract, summarise — while site crawlers, batch jobs, browser automation and `deep_research` are denied outright, because a citation pass needs to find a page and read it, not crawl a domain on the user's credits. (3) The user's API key never passes through the app: CrawlForge authenticates from its own config written by its own setup wizard, which removes credential custody from the security review entirely. (4) Pin the version — `crawlforge-mcp-server@5.6.6` rather than a bare package name with `npx -y`, which otherwise means customers run code nobody on the team has read and makes incidents unreproducible. (5) Name one server but drive any server declared in a standard `.mcp.json`, so an MCP dependency is not a vendor-lock-in question in enterprise review. Includes the granted-vs-denied tool table with per-tool credit costs and a worked unit cost of 11 credits per cited analysis (one `search_web` at 5 plus three `extract_content` reads at 2), roughly ninety analyses on the 1,000 free credits.
- **MCP Server for Due Diligence: What an Agent Can Verify** (https://www.crawlforge.dev/blog/mcp-server-for-due-diligence): What a general web-data MCP server does inside a diligence workflow, and what it deliberately does not do. CrawlForge owns no registry records, sanctions lists, filings or org charts — it reads pages you point it at, so it complements a diligence provider rather than replacing one. Maps each diligence task to a tool and its cost: `deep_research` for what has been published about a counterparty (10), `process_document` for filings and annual reports (2), `extract_structured` for named fields off a registry page (3), `map_site` then `crawl_deep` for a subsidiary's web estate (2 + 4), `track_changes` for post-close monitoring (3). The argument turns on traceability rather than accuracy claims: `deep_research` returns verbatim passages with a `source_url` on every finding instead of a synthesised summary, and `research_scope.domains` pins a run to as many as 10 trusted domains. Covers the failure mode that matters in diligence — a model returning a plausible fact that was never on the page, observed live when `extract_structured` produced a version number absent from the source — plus the two `deep_research` parameter traps (`research_query`, not `query` or `topic`; minimum 10 characters, unknown keys discarded silently) and a worked cost of 23 credits per counterparty.
- **Reddit Scraper for Claude Code: What Claude Actually Calls** (https://www.crawlforge.dev/blog/reddit-scraper-for-claude-code): What actually happens when you ask Claude Code to search Reddit. Setup is one `claude mcp add` line; from there Claude picks the tool and mode itself, typically chaining a scoped `posts` search to a `thread` read of one result — two calls, 10 credits, no Reddit credentials, returning normalized archive rows that can go straight into `summarize_content`. The core mechanic: dropping the subreddit changes the mechanism rather than the scope. Arctic Shift cannot keyword-search across all of Reddit (its API requires a subreddit, author or post scope) and PullPush stopped serving automated clients in August 2026, so an unscoped query runs discovery — a site-restricted web search finds matching posts, then those IDs are read from the archive. Three consequences ship in the response's own `notes` array: results are ordered by web-search relevance rather than score or date, discovery caps at 10 posts per call whatever `limit` was passed, and `after`/`before` are not applied at all. Also covers when to override Claude's choice (name a subreddit; say "search comments" for opinions), cost in an agent loop (5 credits a call, so a search plus three thread reads is a predictable 20), and the accuracy limit to design around: scores and comment counts on content younger than ~36 hours can read 0 or 1 because the archive captures a post on arrival and does not re-poll it.
- **Web Search MCP Server With Local LLM: Which Model Actually Runs** (https://www.crawlforge.dev/blog/web-search-mcp-server-with-local-llm): Which local model a web search MCP server actually ends up running, and why. Separates the two jobs: `search_web` is a search API call with no LLM involved, while the local model matters for `extract_with_llm`, `summarize_content`, `analyze_content` and `deep_research` — the search is remote, the reading is local. Providers register in the order openai, anthropic, ollama, but the two cloud providers only register when their API key is present, so on a machine with no keys Ollama is the only one registered and becomes the default; setting `OPENAI_API_KEY` makes OpenAI outrank it and `DISABLE_OLLAMA=true` removes Ollama entirely. Model selection is four steps: an explicit `OLLAMA_DEFAULT_MODEL` always wins (a stray one in a local .env silently defeats the whole ranking), otherwise the highest-ranked model actually installed, otherwise the first installed model, and `llama3.2` only when Ollama is unreachable so the error names a real model. The ranking is measured, not assumed: against three live product pages with verified ground truth, gemma3:4b scored 18/18 in ~1,040 ms while gpt-oss:20b scored 18/18 in 3,464 ms and llama3.2 scored 16/18 and invented a compare-at price on every run — parameter count did not predict accuracy. A separate ranking covers judging claims, where only gemma3:12b is listed as fit and a model absent from that list is never asked. Also states plainly that running locally is a data-residency change and not a billing one: `search_web` still costs 5 credits whichever model answers, because metering runs before a provider is chosen.
- **Web Search MCP Server: Self-Hosted With SearXNG** (https://www.crawlforge.dev/blog/web-search-mcp-server-self-hosted): Where a web search MCP server can and cannot be self-hosted, split into three layers: the server process (`crawlforge-mcp-server` is an MIT-licensed npm package running locally over stdio, no vendor daemon), the search backend (set `CRAWLFORGE_SEARXNG_URL` and pass `provider: "searxng"` to `search_web` and the query goes to your own SearXNG instance and nowhere else, so no third-party search vendor sees it; results are normalised so ranking, deduplication and caching still apply), and the control plane, which is not self-hosted: the credit check wraps the tool and runs before the provider is chosen, so a SearXNG-backed search still needs a CrawlForge API key and still costs 5 credits. Also covers running `extract_with_llm`, `summarize_content` and `analyze_content` against a local Ollama model, and says plainly that a SearXNG-native MCP wrapper is the better fit when the requirement is zero vendor dependency.
- **CrawlForge MCP v5.2: Every Change Across 28 Tools, Tested Against Live Sites** (https://www.crawlforge.dev/blog/crawlforge-mcp-v5-2-release-notes): Umbrella release notes for v5.2.0 through v5.2.5 plus the crawlforge-extractors package, all found by running all 28 tools of that release against live sites rather than trusting the test suite. Added: `shopify-product` template reading a store's own /products/<handle>.json for exact price, compare-at price and per-variant stock on any storefront including custom domains; `resolveUrl`/`extractRaw` hooks so any template can fetch a machine-readable endpoint instead of HTML; `responseTime` on `fetch_url`; `cached` and `crawled_at` on `crawl_deep`; remote Ollama via `OLLAMA_API_KEY`. Fixed: `amazon-product` returned nulls against every live Amazon page while six unit tests passed (fixtures written to match the selectors, not the site); `track_changes` scored price moves by page share so monitors never fired, `customSelectors` widened rather than scoped the diff, and `structuralSimilarity` could report 0 unmeasured and never fall below 0.5; `scrape` deleted Next.js framework-streamed content, counted script payload as page text, and returned LLM schema echoes as data; `markdown` returned raw HTML on layout tables (turndown-plugin-gfm's keep filter); Ollama was never registered as an LLM provider and routing now picks the best installed model (gemma3:4b beat llama3.2 18/18 vs 16/18); `scrape_with_actions` carried seven Playwright defects; `stealth_mode` leaked a Chromium renderer per call; Reddit-wide keyword search rebuilt on web discovery plus Arctic Shift hydration after PullPush began refusing all automated clients with HTTP 429; `npm-package` reads the registry document; `youtube-video` reads views from the correct InteractionCounter and adds likes; five languages (Chinese, Greek, Arabic, Norwegian, Malay) were undetectable by `analyze_content`; PDF table extraction returned [] for every document. 28 tools at v5.2, no breaking changes, only `reddit_search` moved price (2 -> 5 credits). v5.2.5 adds three more `track_changes` fixes: `customSelectors` on tags outside a fixed allowlist indexed zero elements, a scoped compare silently ran unscoped, and "Text content changed" was reported on compares with no changes.
- **CrawlForge MCP v5.2.0: Shopify Product Data Without Parsing HTML** (https://www.crawlforge.dev/blog/crawlforge-v5-2-0-shopify-product-template): Release post for v5.2.0. New `shopify-product` template reads a store's own /products/<handle>.json instead of the rendered page, returning exact price, compare-at price, per-variant stock, options, images and tags on any Shopify storefront including custom domains; `scrape_template` gains resolveUrl/extractRaw hooks so a template can read a JSON endpoint; `fetch_url` reports responseTime and `crawl_deep` reports cached/crawled_at. Documents the fixes too: an amazon-product template that passed six unit tests while returning nulls against every live Amazon page (the fixtures matched the selectors, not the site), track_changes scoring price moves by page share so monitors never fired, seven Playwright defects in scrape_with_actions, and Ollama never having been registered as an LLM provider. 28 tools, no breaking changes, scrape_template still 1 credit.

- **Web Scraping API Credits That Don't Expire** (https://www.crawlforge.dev/blog/web-scraping-api-credits-that-dont-expire): Why most scraping APIs expire unused credits (breakage) and how CrawlForge prices the alternative: one-time credit packs that never expire ($3/1K, $14/5K, $27/10K, $62.50/25K, no subscription required), paid-plan credits that roll over month to month, cancellation that never wipes a balance, and a one-time 1,000-credit free tier that never refills but never lapses. Includes what 1,000 credits buys across the tool catalog (1-10 credits per call), when a subscription beats packs, and how Firecrawl's monthly-reset free tier compares.
- **AI Crawler List 2026: Every Bot + Should You Block Them?** (https://www.crawlforge.dev/blog/ai-crawlers-list-2026): Reference table of 31 AI crawlers and their robots.txt tokens, sorted into the three classes that actually determine the cost of blocking: training (GPTBot, ClaudeBot, Applebot-Extended, CCBot), search index (OAI-SearchBot, Claude-SearchBot, PerplexityBot, DuckAssistBot), and user-triggered (ChatGPT-User, Claude-User, Perplexity-User). Covers why Google-Extended is a token rather than a crawler and does not affect Search ranking, the RFC 9309 one-group rule that breaks most robots.txt files, agentic browsers that robots.txt cannot address, and verifying bots against published IP ranges. Includes a complete copy-paste robots.txt.
- **Answer 3 Questions, Get 1,000 Free CrawlForge Credits** (https://www.crawlforge.dev/blog/survey-1000-free-credits): Survey reward announcement. Signed-in members who complete the 3-question product survey get a one-time 1,000-credit grant, added instantly. Requires at least 250 credits of prior API usage (the reward is for feedback grounded in real usage); one reward per account, non-recurring.
- **CrawlForge MCP v5.1.0: Search Reddit Without the API** (https://www.crawlforge.dev/blog/crawlforge-v5-1-0-reddit-search-tool): Release post for reddit_search, the 28th tool. Searches Reddit posts and comments and reads full nested threads via the Arctic Shift and PullPush community archives — no Reddit API key, OAuth app, or credentials. Three modes (posts, comments, thread), scoped filters, normalized LLM-friendly output, 5 credits per call. Covers archive routing, honest caveats (fresh-score lag, PullPush post-2023 gaps), and why reddit.com itself 403-blocks every scraper.
- **Reddit API Alternatives That Still Work in 2026** (https://www.crawlforge.dev/blog/reddit-api-alternatives-2026): Honest comparison of the five working routes to Reddit data since self-service API signup closed in November 2025: CrawlForge reddit_search (community archives, no Reddit credentials, 5 credits), querying Arctic Shift or PullPush directly, commercial scraper marketplaces, and the approval-gated official API (still the only route for write access).
- **How to Scrape Reddit Without the API (2026)** (https://www.crawlforge.dev/blog/scrape-reddit-without-api): Step-by-step tutorial. Why every direct path to reddit.com returns 403 (including stealth browsers and the .json trick), and how to search posts and comments and read full nested threads through the Arctic Shift community archive with reddit_search — REST and MCP examples, filters, and honest caveats. Scoped searches query Arctic Shift directly and support after/before date filters; an unscoped Reddit-wide keyword search is served by discovery, a site-restricted web search that finds posts before they are read from the archive by ID, because PullPush stopped serving automated clients in August 2026. 5 credits a call, no Reddit credentials.
- **Reddit Data for AI Agents: The MCP Route** (https://www.crawlforge.dev/blog/reddit-data-for-ai-agents): How to give an MCP agent (Claude Desktop, Claude Code, Cursor) working Reddit search via reddit_search: zero Reddit credentials, context-window-friendly normalized output, and costed agent patterns — an 11-credit research briefing, an 8-credit sentiment scan, and a 5-credit weekly subreddit monitor.
- **The Web Scraping Keyword Gap: A 2026 SERP Study** (https://www.crawlforge.dev/blog/web-scraping-keyword-gap-2026): SERP study built on real Google organic positions for the keywords this industry competes on: the legacy terms are a sealed commercial fortress, while the MCP-era terms have no commercial defender at all.
- **How to Run a Keyword Gap Analysis with an MCP Server** (https://www.crawlforge.dev/blog/keyword-gap-analysis-mcp-server): Build a reproducible keyword gap analysis through MCP with serp_rank and search_web — real organic positions, competitor page-type classification, and the traps that make most keyword research wrong.
- **Agent Scraper: What It Is and How to Build One** (https://www.crawlforge.dev/blog/agent-scraper): An agent scraper follows a goal rather than a selector. What that means in practice, the three ways to build one, working code, the real failure modes, and the credit math.
- **CrawlForge v5.0.4: 34 Fixes From Live-Testing 27 MCP Tools** (https://www.crawlforge.dev/blog/crawlforge-v5-0-4-live-testing-27-tools): Four patch releases in one day. Every MCP tool in the catalog at the time and every CLI subcommand were run against live production websites; all 34 defects found were fixed. Full-surface live testing is now part of the release gate.
- **Is Web Scraping Legal in 2026? The EDPB's New AI Rules** (https://www.crawlforge.dev/blog/is-web-scraping-legal-ai-2026): An engineering reading of EDPB Guidelines 03/2026, adopted 7 July 2026: scraping stays legal, but consent is ruled out as a workable basis at scale and everything routes through the Article 6(1)(f) legitimate-interest test. The change that reaches your code is that robots.txt, ai.txt, CAPTCHAs and login walls now count as evidence of what data subjects reasonably expect. In public consultation until the end of October 2026; final text not expected before the end of 2026.
- **CrawlForge v5.0.0: Security, Correctness, MCP Spec** (https://www.crawlforge.dev/blog/crawlforge-v5-0-0-security-hardening-mcp-spec): Major release bundling a seven-phase security and correctness remediation: zero npm audit vulnerabilities, 914 tests, full MCP spec adoption, and environment variables for client-side tool selection to cut context bloat.
- **SSRF in MCP Servers: Why Scrapers Leak Cloud Secrets** (https://www.crawlforge.dev/blog/mcp-server-ssrf-cloud-metadata-security): Why web-scraping MCP servers are unusually exposed to server-side request forgery against cloud instance-metadata endpoints, and the defenses that actually hold. Cites a July 2026 study finding 91.8% of audited MCP servers lack authentication.
- **How to Actually Use CrawlForge MCP in Claude Code** (https://www.crawlforge.dev/blog/use-crawlforge-mcp-claude-code-terminal): For when the server is installed but Claude never calls it: the prompts, permission rules, and CLAUDE.md configuration that make MCP tools fire reliably in the terminal.
- **CrawlForge v4.8.0: Claude Skills That Auto-Activate** (https://www.crawlforge.dev/blog/crawlforge-v4-8-0-claude-agent-skills): Release post for 7 auto-activating Claude Agent Skills, enforced SSRF protection, working screenshots, a design-token branding format, and built-in scheduled change monitoring.
- **CrawlForge vs Firecrawl vs Tavily vs Exa: Best Web Data API for AI Agents (2026)** (https://www.crawlforge.dev/blog/crawlforge-vs-firecrawl-tavily-exa-web-data-api): Four-way comparison for AI agents: Tavily and Exa are search-first APIs (Exa adds neural/semantic search), Firecrawl is a scrape-and-crawl engine, and CrawlForge is an MCP-native server bundling search, scraping, crawling and deep research. Includes pricing and a pick-one decision guide.
- **How to Build a Web-Scraping MCP Server in TypeScript (2026)** (https://www.crawlforge.dev/blog/build-web-scraping-mcp-server): Hands-on tutorial: build a working web-scraping MCP server in TypeScript with the official SDK — a minimal server, a real cheerio scraping tool, testing, and Claude Desktop setup.
- **How to Give ChatGPT Web Scraping with MCP Connectors (2026)** (https://www.crawlforge.dev/blog/connect-chatgpt-web-scraping-mcp): How to connect ChatGPT to web scraping using custom MCP connectors: why a stdio server such as CrawlForge needs a remote wrapper, the FastMCP bridge to build, and how to add it in ChatGPT.
- **Web Scraping with Claude: The Complete Guide (2026)** (https://www.crawlforge.dev/blog/web-scraping-with-claude-complete-guide): Complete guide to scraping with Claude: connect CrawlForge MCP to Claude Desktop, Claude Code, or the Claude API and extract data from any site without writing scraping code.
- **Best MCP Servers for Web Scraping in 2026 (Top 8 Ranked)** (https://www.crawlforge.dev/blog/best-mcp-servers-web-scraping-2026): Ranked roundup of the 8 best MCP servers for web scraping, comparing tool breadth, anti-bot handling, free tiers and pricing side by side.
- **Best Web Scraping Tools for AI Agents in 2026** (https://www.crawlforge.dev/blog/best-web-scraping-tools-ai-agents-2026): The best web scraping tools ranked by agent-readiness: MCP-native tool discovery, typed schemas, and token-efficient output.
- **How to Use Web Scraping Templates the Right Way** (https://www.crawlforge.dev/blog/how-to-use-web-scraping-templates): Copy-paste, ready-to-run MCP web scraping templates for price monitoring, lead enrichment, SEO audits and more, plus how to read, adapt and cost them.
- **How to Scrape Websites with Claude Code (2026 Guide)** (https://www.crawlforge.dev/blog/scrape-websites-with-claude-code): Scrape any website from the terminal with Claude Code and CrawlForge MCP: fetch pages, extract data and handle anti-bot protection in under two minutes.
- **How to Scrape Websites in Cursor IDE with CrawlForge MCP** (https://www.crawlforge.dev/blog/scrape-websites-in-cursor-ide): Connect CrawlForge MCP to Cursor IDE and extract structured data from any site without leaving the editor.
- **How to Scrape Websites in Zed AI with CrawlForge MCP** (https://www.crawlforge.dev/blog/scrape-websites-in-zed-ai): Configure CrawlForge MCP in Zed AI so the editor can fetch, extract and research live web data on demand.
- **How to Scrape Websites with GitHub Copilot Agents in VS Code** (https://www.crawlforge.dev/blog/scrape-websites-github-copilot-vs-code): Configure CrawlForge MCP for GitHub Copilot agents in VS Code so Copilot can fetch, extract and research live web data on demand.
- **LlamaIndex Web Scraping Guide with CrawlForge MCP** (https://www.crawlforge.dev/blog/llamaindex-web-scraping-guide): Feed LlamaIndex with live web data: use CrawlForge as a LlamaIndex reader for RAG pipelines, agent tools and real-time LLM knowledge bases.
- **CrawlForge v4.2.2 - New CLI + 3 Tools for Local AI Scraping** (https://www.crawlforge.dev/blog/crawlforge-v4-2-2-cli-and-3-new-tools): v4.2.2 launch announcement covering the new CLI bundled in the crawlforge-mcp-server package, extract_with_llm (Ollama-default local extraction), scrape_template (10 pre-built site scrapers), and list_ollama_models (local model discovery). That v4.2.2 release brought the catalog to 23 tools (now 30).
- **Web Scraping From the CLI - The CrawlForge CLI Guide** (https://www.crawlforge.dev/blog/web-scraping-cli-complete-guide): Complete reference for the crawlforge CLI (bundled in crawlforge-mcp-server) with all 15 commands, JSON-piping patterns (jq), cron scheduling, GitHub Actions integration, and a CLI-vs-MCP-vs-API decision matrix.
- **Extract Web Data With Local LLMs (Ollama + CrawlForge)** (https://www.crawlforge.dev/blog/extract-data-with-local-llms-ollama): Deep dive on extract_with_llm and list_ollama_models. Run LLM-powered structured extraction on your own machine -- no API key, no data leaving localhost. Includes setup, schema patterns, provider comparison, and cost analysis.
- **Scrape Amazon, LinkedIn & 8 More Sites With One Tool** (https://www.crawlforge.dev/blog/scrape-amazon-linkedin-github-templates): Walkthrough of scrape_template with all 10 supported sites (Amazon, LinkedIn, GitHub, YouTube, Reddit, Hacker News, Stack Overflow, npm, Product Hunt, Twitter/X). One call, structured JSON, 1 credit. (Historical post — the catalogue is 18 templates today, and the LinkedIn and X/Twitter templates were retired in 5.5.0.)
- **CrawlForge MCP Is Now Live** (https://www.crawlforge.dev/blog/crawlforge-mcp-is-now-live): Official launch announcement covering the 28 tools CrawlForge MCP shipped with, credit-based pricing, and native MCP integration with Claude and Cursor. (Historical post — the catalog is 30 tools today.)
- **CrawlForge MCP Quickstart** (https://www.crawlforge.dev/blog/crawlforge-mcp-quickstart): Step-by-step guide to get your first CrawlForge scrape running in under 5 minutes with Claude Desktop or the REST API.
- **Install CrawlForge MCP in Claude Code - Beginner's Guide** (https://www.crawlforge.dev/blog/install-crawlforge-mcp-claude-code-beginners-guide): Complete walkthrough for installing and configuring CrawlForge as an MCP server in Claude Code for terminal-based AI workflows.
- **18 Web Scraping Tools in One MCP Server** (https://www.crawlforge.dev/blog/18-web-scraping-tools-one-mcp-server): Deep dive into the CrawlForge tool catalog, explaining when to use each tool and how they work together for complex scraping tasks. (Historical post — the catalog has since grown to 30 tools.)
- **5 Ways to Use CrawlForge with LangChain** (https://www.crawlforge.dev/blog/5-ways-use-crawlforge-langchain): Practical examples of integrating CrawlForge tools into LangChain agents for RAG pipelines, research automation, and data enrichment.
- **CrawlForge vs Apify vs ScrapingBee Comparison** (https://www.crawlforge.dev/blog/crawlforge-vs-apify-vs-scrapingbee-comparison): Feature-by-feature comparison of CrawlForge against Apify and ScrapingBee, covering pricing, AI integration, and developer experience.
- **How to Add Web Scraping to Claude Desktop** (https://www.crawlforge.dev/blog/how-to-add-web-scraping-to-claude-desktop): Tutorial for enabling web scraping capabilities in Claude Desktop using CrawlForge MCP with zero coding required.
- **Building an AI Research Assistant with Claude MCP** (https://www.crawlforge.dev/blog/building-ai-research-assistant-claude-mcp): Build a multi-source research assistant that uses deep_research and summarize_content to synthesize information from across the web.
- **MCP vs REST - Why Native MCP Beats REST for Scraping** (https://www.crawlforge.dev/blog/mcp-vs-rest-native-mcp-scraping-server): Technical comparison showing why MCP-native scraping tools outperform REST API wrappers for AI agent workflows.
- **Introducing Deep Research - AI-Powered Analysis** (https://www.crawlforge.dev/blog/introducing-deep-research-ai-powered-analysis): Introduction to the deep_research tool that combines multi-stage web search, content extraction, and AI synthesis for comprehensive analysis.
- **Web Scraping for AI Training Data - 2025 Guide** (https://www.crawlforge.dev/blog/web-scraping-ai-training-data-2025-guide): Best practices for collecting, cleaning, and structuring web data for AI model training, including ethical and legal considerations.
- **Welcome to CrawlForge** (https://www.crawlforge.dev/blog/welcome-to-crawlforge): Introduction to the CrawlForge platform, our mission to make web data accessible to AI agents, and what to expect from the product.
- **CrawlForge vs Firecrawl - MCP Web Scraper Comparison** (https://www.crawlforge.dev/blog/crawlforge-vs-firecrawl-mcp-web-scraper-comparison): Head-to-head comparison of CrawlForge and Firecrawl covering tool breadth, pricing models, MCP support, and unique features like stealth mode.
- **Build a Competitive Intelligence Agent with Claude and CrawlForge** (https://www.crawlforge.dev/blog/build-competitive-intelligence-agent-claude-crawlforge): Tutorial for building an automated competitive intelligence agent that monitors competitor websites, tracks pricing changes, and generates reports.
- **Stealth Mode Scraping - Bypass Anti-Bot Detection** (https://www.crawlforge.dev/blog/stealth-mode-scraping-bypass-anti-bot-detection): How CrawlForge's stealth_mode tool uses fingerprint randomization and proxy rotation to access sites with aggressive anti-bot protection.
- **Deep Research Automation - 10 Hours to 10 Minutes** (https://www.crawlforge.dev/blog/deep-research-automation-10-hours-to-10-minutes): Case study showing how deep_research reduces manual research workflows from hours to minutes with automated source discovery and synthesis.
- **Complete Guide to MCP Web Scraping** (https://www.crawlforge.dev/blog/complete-guide-mcp-web-scraping): Comprehensive guide to web scraping via the Model Context Protocol, covering setup, tool selection, error handling, and production best practices.

## Frequently Asked Questions

### How do credits work?
Each tool costs a set number of credits per use. Simple tools like fetch_url cost 1 credit, while advanced tools like deep_research cost 10 credits per query. Your credit balance is deducted atomically with each API call, and you can monitor usage in your dashboard.

### Do unused credits roll over?
Yes! On paid plans, all unused credits roll over to the next month and never expire while your subscription is active. The Free tier grants 1,000 credits once at signup; they do not refill.

### Can I change plans anytime?
Yes, you can upgrade or downgrade your plan at any time from your dashboard. Upgrades take effect immediately with prorated billing. Downgrades take effect at the start of your next billing cycle.

### What payment methods do you accept?
We accept all major credit and debit cards (Visa, Mastercard, American Express) through Stripe. Annual plans and Enterprise contracts may also pay via invoice.

### Is there a free trial?
The Free tier gives you 1,000 credits at no cost with no credit card required. This lets you fully test all 30 tools before committing to a paid plan.

### What happens when I run out of credits?
When your credits reach zero, API requests will return a 402 Payment Required error. You can purchase credit add-ons starting at $3 for 1,000 credits, or upgrade your plan for a larger monthly allocation.

## Getting Started Quick Guide

1. **Sign up** at https://www.crawlforge.dev/signup to create your free account (no credit card required, 1,000 free credits).
2. **Generate an API key** from your dashboard at https://www.crawlforge.dev/dashboard. Your key will start with `cf_live_`.
3. **Install the MCP server** by adding CrawlForge to your Claude Desktop or Cursor IDE config: `npx -y crawlforge-mcp-server` with your API key as an environment variable.
4. **Make your first request** by asking Claude to fetch a webpage or run a search. CrawlForge tools are automatically available as MCP tools.
5. **Monitor usage** in your dashboard to track credit consumption, view request history, and manage your subscription.

## Use Cases

### Lead Generation
Use extract_content and scrape_structured to pull contact information, company details, and prospect data from business directories, company websites, and professional networks. Combine with batch_scrape to process hundreds of prospect pages in a single request.

### Price Monitoring
Combine track_changes with scrape_structured to monitor competitor pricing across e-commerce sites. Set up automated change detection with significance scoring to get alerted only when prices change meaningfully, filtering out minor page updates.

### Data Extraction
Use scrape_structured with CSS selectors for known page layouts, or extract_content with AI-powered extraction for unstructured pages. Process PDFs and documents with process_document. Scale to thousands of pages using batch_scrape and crawl_deep.

### Competitive Intelligence
Build automated monitoring workflows with track_changes to watch competitor websites for product launches, feature updates, and messaging changes. Use deep_research to synthesize competitive landscapes from multiple sources into actionable reports.

### AI Research Workflows
The deep_research tool automates multi-stage research by searching the web, extracting content from top sources, cross-referencing facts, and synthesizing findings into structured reports. Combined with summarize_content and analyze_content, it replaces hours of manual research.

## Comparison: CrawlForge vs Firecrawl vs Apify

### CrawlForge
- MCP-native architecture purpose-built for AI agents
- 30 specialized tools covering scraping, research, and monitoring
- Credit-based pricing starting at $0 with 1,000 free credits
- Built-in stealth mode with anti-detection and proxy rotation
- Deep research tool for multi-stage AI-powered analysis
- Single npm package install for Claude, Cursor, and any MCP client

### Firecrawl
- REST API with MCP wrapper available
- Focused on web scraping and crawling
- Page-based pricing model
- Good markdown conversion capabilities
- Requires separate MCP adapter configuration

### Apify
- Actor-based platform with extensive marketplace
- Hundreds of pre-built scrapers for specific sites
- Usage-based pricing with compute units
- Steeper learning curve for custom actors
- REST API without native MCP support

## Additional Blog Posts

### Integrations and Framework Guides
- **How to Use CrawlForge with n8n** (https://www.crawlforge.dev/blog/crawlforge-n8n-integration-guide): Connect CrawlForge MCP to n8n for automated web scraping workflows with no-code pipelines that extract, transform, and load web data on a schedule.
- **How to Use CrawlForge with Vercel AI SDK** (https://www.crawlforge.dev/blog/crawlforge-vercel-ai-sdk-guide): Build AI apps with live web data using CrawlForge and the Vercel AI SDK. Register 28 web scraping tools with generateText and streamText in under 10 minutes.
- **How to Use CrawlForge with Windsurf IDE** (https://www.crawlforge.dev/blog/crawlforge-windsurf-integration-guide): Add 28 web scraping tools to Windsurf IDE with CrawlForge MCP. Fetch docs, scrape references, and research APIs without leaving your editor.
- **How to Use CrawlForge with Cline (VS Code)** (https://www.crawlforge.dev/blog/crawlforge-cline-integration-guide): Configure CrawlForge MCP in Cline for VS Code and let your AI coding assistant access live web data from inside your editor.
- **How to Use CrawlForge with OpenAI Agents SDK** (https://www.crawlforge.dev/blog/crawlforge-openai-agents-integration): Connect CrawlForge to the OpenAI Agents SDK for live data retrieval, research, and structured extraction inside GPT-powered agents.
- **How to Use CrawlForge with Anthropic Claude API** (https://www.crawlforge.dev/blog/crawlforge-anthropic-claude-api-guide): Connect CrawlForge web scraping tools to the Claude API via tool_use. TypeScript examples with Claude Sonnet for live web data.
- **How to Use CrawlForge with Smithery** (https://www.crawlforge.dev/blog/crawlforge-smithery-integration-guide): Discover, install, and configure CrawlForge via the Smithery MCP marketplace with CLI commands and tool usage examples.
- **How to Use CrawlForge with Cursor Rules** (https://www.crawlforge.dev/blog/crawlforge-cursor-rules-guide): Create .cursorrules files that teach Cursor AI to use CrawlForge tools effectively, with ready-to-use rules for web research and data extraction.
- **How to Use CrawlForge with Mastra AI Agents** (https://www.crawlforge.dev/blog/crawlforge-mastra-integration-guide): Build AI agents with web scraping capabilities using Mastra and CrawlForge. TypeScript setup, tool integration, workflows, and agent examples.
- **How to Use CrawlForge with Dify Workflows** (https://www.crawlforge.dev/blog/crawlforge-dify-integration-guide): Add CrawlForge as a custom tool in Dify for web scraping in LLM app workflows. No-code and API integration guides.
- **How to Use CrawlForge with Make and Zapier** (https://www.crawlforge.dev/blog/crawlforge-make-zapier-automation-guide): Connect CrawlForge to Make (Integromat) and Zapier for automated web scraping with HTTP modules, webhooks, and workflow examples.
- **How to Use CrawlForge with LangGraph Agents** (https://www.crawlforge.dev/blog/crawlforge-langgraph-integration-guide): Build stateful web scraping agents with LangGraph and CrawlForge. TypeScript guide covering graph nodes, state management, and conditional scraping flows.

### Tutorials and Use Cases
- **Build an AI-Powered Price Monitoring System** (https://www.crawlforge.dev/blog/ai-price-monitoring-system-guide): Track competitor prices automatically with CrawlForge and Claude. Extract, compare, and alert on pricing changes across thousands of product pages.
- **Automate SEO Audits with CrawlForge MCP** (https://www.crawlforge.dev/blog/automating-seo-audits-crawlforge): Run comprehensive technical SEO audits automatically. Crawl your site, check metadata, find broken links, and generate actionable reports.
- **Real-Time Competitive Intelligence with AI Agents** (https://www.crawlforge.dev/blog/competitive-intelligence-ai-agents): Build an AI-powered competitive intelligence system using CrawlForge and Claude. Monitor competitors, track changes, and generate strategic insights weekly.
- **Web Scraping for AI Training Data Pipelines** (https://www.crawlforge.dev/blog/web-scraping-ai-training-data-pipelines): Build production-ready AI training data pipelines with CrawlForge. Extract, clean, and structure web content for fine-tuning LLMs and training ML models.
- **Content Migration Automation with CrawlForge** (https://www.crawlforge.dev/blog/content-migration-automation-guide): Migrate website content between CMS platforms automatically. Extract pages, preserve structure, and rebuild content in your new system.
- **Build a Lead Enrichment Engine with CrawlForge** (https://www.crawlforge.dev/blog/building-lead-enrichment-engine): Enrich sales leads with company data, tech stacks, and contact details automatically. Scrape public business data to qualify leads.
- **Build a Research Agent with CrawlForge Deep Research** (https://www.crawlforge.dev/blog/real-time-research-agent-deep-research): Create an AI research agent that gathers, verifies, and synthesizes information from dozens of sources in minutes using CrawlForge deep_research.
- **E-commerce Product Data Extraction at Scale** (https://www.crawlforge.dev/blog/ecommerce-product-data-extraction-scale): Extract product data from thousands of e-commerce pages with CrawlForge. Build catalogs, monitor inventory, and power comparison engines.

### Deep Dives and Reference Guides
- **Best Web Scraping Tools in 2026** (https://www.crawlforge.dev/blog/best-web-scraping-tools-2026): Compare 12 web scraping tools for 2026 including CrawlForge, Firecrawl, Apify, and Scrapy. Features, pricing, and recommendations for every use case.
- **MCP Protocol Explained: A Developer Guide for 2026** (https://www.crawlforge.dev/blog/mcp-protocol-explained-developers): Learn how the Model Context Protocol works, why it matters for AI agents, and how to build MCP servers and clients with architecture diagrams and code.
- **Web Scraping: Python vs MCP in 2026** (https://www.crawlforge.dev/blog/web-scraping-python-vs-mcp): Compare Python scraping (requests, BeautifulSoup, Scrapy) with MCP-based scraping. Side-by-side code, performance benchmarks, and when to use each.
- **How to Build a RAG Pipeline with Web Data** (https://www.crawlforge.dev/blog/build-rag-pipeline-web-data): Build a production RAG pipeline that crawls websites, extracts content, chunks text, generates embeddings, and serves retrieval-augmented answers.
- **Web Scraping by Industry: 2026 Playbook** (https://www.crawlforge.dev/blog/industry-web-scraping-guide-2026): Industry-specific web scraping strategies for real estate, finance, e-commerce, healthcare, and travel. Data targets, CrawlForge tools, and compliance rules.

## Alternatives and Head-to-Head Comparisons

CrawlForge maintains detailed comparison pages that explain when to choose CrawlForge versus each competitor. Each comparison covers architecture, pricing, AI integration, stealth capabilities, and includes a verdict plus a "Pick CrawlForge when / Pick competitor when" decision guide.

- **CrawlForge vs ScrapingBee** (https://www.crawlforge.dev/alternatives/crawlforge-vs-scrapingbee): MCP-native tools with built-in structured extraction versus a traditional proxy-based scraping API. CrawlForge adds research, monitoring, and 30 tools ScrapingBee does not offer.
- **CrawlForge vs Scrapy** (https://www.crawlforge.dev/alternatives/crawlforge-vs-scrapy): Managed MCP web scraping service versus a self-hosted Python framework. Zero infrastructure versus full control - pick based on whether you want to own or outsource ops.
- **Best Bright Data Alternative: CrawlForge** (https://www.crawlforge.dev/alternatives/bright-data-alternative): Developer-focused MCP tools with transparent pricing versus an enterprise-grade data platform. CrawlForge wins on simplicity; Bright Data wins on enterprise volume and proxy networks.
- **CrawlForge vs Playwright** (https://www.crawlforge.dev/alternatives/crawlforge-vs-playwright): Managed web extraction service versus a browser automation library. Skip the infrastructure and get structured data via MCP tool calls instead of writing browser scripts.
- **CrawlForge vs Puppeteer** (https://www.crawlforge.dev/alternatives/crawlforge-vs-puppeteer): Managed MCP web scraping versus a Node.js browser automation library. CrawlForge removes Chrome instance management, proxy rotation, and scaling concerns.
- **Best Firecrawl Alternative: CrawlForge** (https://www.crawlforge.dev/alternatives/firecrawl-alternative): CrawlForge offers MCP-native web scraping with 30 tools, deep research with conflict detection, and 1,000 free credits as a direct Firecrawl alternative.
- **Best Apify Alternative: CrawlForge** (https://www.crawlforge.dev/alternatives/apify-alternative): CrawlForge replaces complex actor configurations with 30 MCP-native tools and simple per-tool credit pricing. Get started without the Apify learning curve.
- **Web Scraping API Comparison 2026** (https://www.crawlforge.dev/alternatives/web-scraping-api-comparison): Side-by-side comparison of CrawlForge, Firecrawl, Apify, ScrapingBee, and Bright Data for web scraping and AI data extraction.

## Use Case Deep Dives

Each use case page includes a problem statement, the CrawlForge solution, specific tools used, a working TypeScript code example, and credit cost estimates.

- **AI Agent Data Pipelines** (https://www.crawlforge.dev/use-cases/ai-agent-data-pipelines): Feed AI agents live web data with deep_research and extract_content. Agents get structured, real-time information without custom scrapers. ~12 credits per research task.
- **E-commerce Price Monitoring** (https://www.crawlforge.dev/use-cases/ecommerce-price-monitoring): Track competitor prices across hundreds of product pages using batch_scrape and track_changes. Get alerts when competitors adjust pricing. ~8 credits per batch.
- **SEO Analysis and Auditing** (https://www.crawlforge.dev/use-cases/seo-analysis-auditing): Audit sites for metadata, broken links, and content gaps using map_site, extract_metadata, and analyze_content. ~8 credits per site audit.
- **Competitive Intelligence** (https://www.crawlforge.dev/use-cases/competitive-intelligence): Gather competitor data through anti-bot protections using stealth_mode and scrape_structured with schema-based extraction. ~8 credits per protected page.
- **Content Migration** (https://www.crawlforge.dev/use-cases/content-migration): Crawl legacy sites and extract all content for CMS migration using crawl_deep and extract_text. Move thousands of pages in a single run. ~6 credits per page.
- **Lead Enrichment** (https://www.crawlforge.dev/use-cases/lead-enrichment): Enrich sales leads with company data using search_web and extract_metadata. Build on-demand enrichment pipelines. ~5 credits per lead.
- **AI Training Data Collection** (https://www.crawlforge.dev/use-cases/ai-training-data): Collect large-scale web datasets for fine-tuning using batch_scrape and extract_content. Clean, structured text ready for training pipelines. ~7 credits per document.
- **Real-Time Research Agents** (https://www.crawlforge.dev/use-cases/real-time-research-agents): Build agents that search the web and synthesize findings using deep_research and search_web with conflict detection. ~13 credits per research session.
- **RAG Knowledge Base Ingestion** (https://www.crawlforge.dev/use-cases/rag-knowledge-base): Turn documentation sites into clean, chunk-ready markdown for RAG using map_site, scrape, and summarize_content. ~6 credits per document.
- **PDF and Document Data Extraction** (https://www.crawlforge.dev/use-cases/pdf-document-extraction): Extract typed JSON fields from PDFs, reports, and filings using process_document and extract_with_llm, with local Ollama by default. ~5 credits per document.
- **Job Listing Aggregation** (https://www.crawlforge.dev/use-cases/job-listing-aggregation): Collect postings from boards that paginate, lazy-load, or sit behind a login using scrape_with_actions and extract_structured. ~8 credits per board page.
- **Keyword Rank Tracking** (https://www.crawlforge.dev/use-cases/keyword-rank-tracking): Check a domain's real Google organic position by keyword, location, and device using serp_rank, with search_web for the competing results. ~5 credits per keyword.
- **llms.txt and AI Search Visibility** (https://www.crawlforge.dev/use-cases/llms-txt-ai-visibility): Generate llms.txt and llms-full.txt with generate_llms_txt, then verify crawler reach with map_site. ~7 credits per site.
- **Review and Sentiment Monitoring** (https://www.crawlforge.dev/use-cases/review-sentiment-monitoring): Track customer sentiment across marketplaces and forums using scrape_template, reddit_search, analyze_content, and summarize_content. ~13 credits per product.
- **Geo Pricing and Localization Testing** (https://www.crawlforge.dev/use-cases/geo-pricing-localization): See the prices and content served to visitors in another country using localization and stealth_mode. ~7 credits per region check.

## Glossary Guides

The 50 terms used across this documentation are defined in four category guides. Each term is a section with a stable `#slug` anchor, so a definition can be cited directly.

- **Web Scraping Terms** (https://www.crawlforge.dev/glossary/web-scraping-terms): 15 terms covering how pages are fetched, rendered and parsed — web scraping, web crawler, DOM parsing, CSS selector, XPath, headless browser, proxy rotation, rate limiting, CAPTCHA solving, robots.txt, sitemap, user agent, HTTP headers, pagination, dynamic content.
- **AI and MCP Terms** (https://www.crawlforge.dev/glossary/ai-and-mcp-terms): 15 terms covering how language models reach live web data — token, embeddings, context window, large language model, prompt engineering, fine-tuning, function calling, tool use, structured output, retrieval-augmented generation, vector database, Model Context Protocol, MCP server, MCP client, AI agent.
- **Data and API Terms** (https://www.crawlforge.dev/glossary/data-and-api-terms): 10 terms covering the formats and protocols that move scraped data — REST API, GraphQL, API endpoint, JSON, JSON-LD, webhook, Markdown, HTML parsing, schema markup, structured data.
- **Web Data Industry Terms** (https://www.crawlforge.dev/glossary/web-data-industry-terms): 10 terms covering commercial web data work — ETL, data pipeline, data quality, data governance, web data, competitive intelligence, price monitoring, lead enrichment, SEO audit, content migration.

Glossary index: https://www.crawlforge.dev/glossary

## Complete Tool Documentation Reference

Every CrawlForge tool has a dedicated documentation page with parameters, response formats, code examples, and credit costs.

- fetch_url: https://www.crawlforge.dev/docs/api-reference/tools/fetch-url
- extract_text: https://www.crawlforge.dev/docs/api-reference/tools/extract-text
- extract_links: https://www.crawlforge.dev/docs/api-reference/tools/extract-links
- extract_metadata: https://www.crawlforge.dev/docs/api-reference/tools/extract-metadata
- scrape_structured: https://www.crawlforge.dev/docs/api-reference/tools/scrape-structured
- extract_content: https://www.crawlforge.dev/docs/api-reference/tools/extract-content
- map_site: https://www.crawlforge.dev/docs/api-reference/tools/map-site
- process_document: https://www.crawlforge.dev/docs/api-reference/tools/process-document
- localization: https://www.crawlforge.dev/docs/api-reference/tools/localization
- track_changes: https://www.crawlforge.dev/docs/api-reference/tools/track-changes
- Hosted monitors API: https://www.crawlforge.dev/docs/api-reference/monitors
- analyze_content: https://www.crawlforge.dev/docs/api-reference/tools/analyze-content
- summarize_content: https://www.crawlforge.dev/docs/api-reference/tools/summarize-content
- crawl_deep: https://www.crawlforge.dev/docs/api-reference/tools/crawl-deep
- stealth_mode: https://www.crawlforge.dev/docs/api-reference/tools/stealth-mode
- scrape_with_actions: https://www.crawlforge.dev/docs/api-reference/tools/scrape-with-actions
- batch_scrape: https://www.crawlforge.dev/docs/api-reference/tools/batch-scrape
- search_web: https://www.crawlforge.dev/docs/api-reference/tools/search-web
- serp_rank: https://www.crawlforge.dev/docs/api-reference/tools/serp-rank
- reddit_search: https://www.crawlforge.dev/docs/api-reference/tools/reddit-search
- deep_research: https://www.crawlforge.dev/docs/api-reference/tools/deep-research
- scrape_template: https://www.crawlforge.dev/docs/api-reference/tools/scrape-template
- extract_with_llm: https://www.crawlforge.dev/docs/api-reference/tools/extract-with-llm
- list_ollama_models: https://www.crawlforge.dev/docs/api-reference/tools/list-ollama-models
- scrape: https://www.crawlforge.dev/docs/api-reference/tools/scrape
- agent: https://www.crawlforge.dev/docs/api-reference/tools/agent
- get_batch_results: https://www.crawlforge.dev/docs/api-reference/tools/get-batch-results
- Tools index: https://www.crawlforge.dev/docs/api-reference/tools
- OpenAPI specification (3.1): https://www.crawlforge.dev/openapi.json
- TypeScript SDK (npm install crawlforge-sdk): https://github.com/mysleekdesigns/crawlforge-sdk-ts
- Python SDK (pip install crawlforge): https://github.com/mysleekdesigns/crawlforge-sdk-python