CrawlForge CLI
The crawlforge command ships with the crawlforge-mcp-server package, so a single global install gives you both the MCP server and a standalone CLI for one-off commands and scripting—no MCP client required. Set an API key and you're scraping in seconds.
Install & Authenticate
Install crawlforge-mcp-server globally—the crawlforge command is included. The recommended way to authenticate is crawlforge login, which approves a new key in your browser and stores it in ~/.crawlforge/config.json for the CLI to read automatically; npx crawlforge-setup does the same with a key you paste. Get a key from your API Keys dashboard.
npm install -g crawlforge-mcp-server
crawlforge login # browser sign-in (recommended)
npx crawlforge-setup # or paste a key by hand
crawlforge --helpDependencies, including crawlforge-extractors, install automatically.
Sign in with crawlforge login
crawlforge login prints an approval URL. Open it in a browser, sign in to CrawlForge and approve the request; the CLI waits for the approval, validates the new key and saves it to ~/.crawlforge/config.json (readable only by you). It gives up after 10 minutes if nobody approves; --wait <seconds> changes that limit.
crawlforge login # prints an approval URL, then waits for you to approve it
crawlforge login --name ci-box # label the key as it will appear in the dashboard
crawlforge login --wait 120 # stop waiting after 120 s (default 600)
crawlforge login --json # machine-readable URL and result for scripts--name sets the key's label in the API Keys dashboard; --json prints the URL and result as JSON for scripts. login only stores the key—run crawlforge init --client <name> afterwards to register the MCP server with Claude Code, Claude Desktop or Cursor, or use the MCP configurator for other clients.
CRAWLFORGE_API_KEY=cf_live_... in your ~/.zshrc / ~/.bashrc, or pass --api-key cf_live_... on any command. Priority: --api-key > environment variable > stored config.Commands
The 16 tool commands below cover 28 of the 31 CrawlForge tools (some commands expose multiple tools via flags; reddit_search, extract_embedded_state and read_result are available via MCP and the REST API only). Run crawlforge <command> --help to see flags and options for any command.
One crawlforge browser invocation is one whole session, and a session cannot span two CLI processes: the command opens the page, snapshots it, runs the steps in --steps, optionally reads, then closes. Use the browser_session MCP tool or the REST API when a session has to outlive the call that opened it.
| Command | Purpose |
|---|---|
scrape | Fetch a URL (fetch_url); add --extract for cleaned text/markdown |
search | Web search (--provider crawlforge or searxng) |
crawl | Deep multi-page crawl with depth/page/concurrency controls |
map | Generate a sitemap (--format json or xml) |
extract | Structured extraction via --schema (CSS) or --prompt (LLM) |
track | Track content changes (--selector, --threshold) |
analyze | Sentiment, entities, and readability analysis |
research | Multi-source deep research with source verification |
stealth | Anti-bot browser mode (--engine playwright or camoufox) |
batch | Scrape many URLs from a newline-delimited file |
actions | Browser automation from a JSON action script (--script) |
browser | Browser session run end to end: open, snapshot, act on the refs it returns, read, close (--steps, --read) |
localize | Locale/geo-aware fetch (--locale, --country, --currency) |
llmstxt | Generate llms.txt and llms-full.txt (--include-full) |
template | Run a pre-built site template (--list shows all 10) |
monitor | Scheduled change monitoring (--interval, --webhook) |
Skills installers
Two helper commands copy CrawlForge skill files into your AI coding tool so it knows how to use every CrawlForge command.
| Command | Purpose |
|---|---|
install-skills | Install CrawlForge skills into Claude Code, Cursor, or VS Code (--target, --dry-run) |
uninstall-skills | Remove installed CrawlForge skill files (--target) |
crawlforge mcp (alias crawlforge serve). MCP clients normally launch it for you—see the Claude Desktop and Cursor guides.Examples
1. Scrape a page as clean markdown
crawlforge scrape https://example.com --extract --format markdownWithout --extract the command runs fetch_url (raw headers + body); with it, extract_content returns cleaned text or markdown. Add --pretty to print indented JSON.
2. Run deep research with multiple sources
crawlforge research "AI agent frameworks 2026" --depth deep --max-urls 10Analyzes up to 10 sources, verifies citations, and emits a structured research report. Depth is basic, standard, or deep. Costs 10 credits.
3. Use a pre-built site template
crawlforge template github-repo https://github.com/anthropics/claude-code --pretty
crawlforge template --list # show all 10 templatesTemplates extract a known schema for popular sites (GitHub repos, Amazon products, YouTube videos, and more) without writing selectors yourself. The target URL is a positional argument.
4. Pipe JSON into other tools
crawlforge search "nodejs MCP server" --json | jq '.results[].url'--json emits single-line JSON for shell pipelines and CI; --quiet returns only an exit code.
Global Flags
These flags work with any command and shape how output is delivered.
| Flag | Description |
|---|---|
--json | Output compact JSON (pipe-friendly) |
--pretty | Output pretty-printed JSON |
--quiet | Suppress all stdout output (exit code only) |
--api-key <key> | Override the CRAWLFORGE_API_KEY env var |
--timeout <ms> | Global request timeout in ms (default: 30000) |