CrawlForge CLI
The crawlforge command ships with the crawlforge-mcp-server package, so a single global install gives you both the MCP server and a standalone CLI for one-off commands and scripting—no MCP client required. Set an API key and you're scraping in seconds.
Install & Authenticate
Install crawlforge-mcp-server globally—the crawlforge command is included. The recommended way to authenticate is crawlforge login, which approves a new key in your browser and stores it in ~/.crawlforge/config.json for the CLI to read automatically; npx crawlforge-setup does the same with a key you paste. Get a key from your API Keys dashboard.
npm install -g crawlforge-mcp-server
crawlforge login # browser sign-in (recommended)
npx crawlforge-setup # or paste a key by hand
crawlforge --helpDependencies, including crawlforge-extractors, install automatically.
Sign in with crawlforge login
crawlforge login prints an approval URL. Open it in a browser, sign in to CrawlForge and approve the request; the CLI waits for the approval, validates the new key and saves it to ~/.crawlforge/config.json (readable only by you). It gives up after 10 minutes if nobody approves; --wait <seconds> changes that limit.
crawlforge login # prints an approval URL, then waits for you to approve it
crawlforge login --name ci-box # label the key as it will appear in the dashboard
crawlforge login --wait 120 # stop waiting after 120 s (default 600)
crawlforge login --json # machine-readable URL and result for scripts--name sets the key's label in the API Keys dashboard; --json prints the URL and result as JSON for scripts. login only stores the key—run crawlforge init --client <name> afterwards to register the MCP server with Claude Code, Claude Desktop or Cursor, or use the MCP configurator for other clients.
export CRAWLFORGE_API_KEY=cf_live_... in your ~/.zshrc / ~/.bashrc, or pass --api-key cf_live_... on any command. Priority: --api-key > environment variable > stored config.
Commands
The 16 tool commands below cover 28 of the 31 CrawlForge tools (some commands expose multiple tools via flags; reddit_search, extract_embedded_state and read_result are available via MCP and the REST API only). Run crawlforge <command> --help to see flags and options for any command.
One crawlforge browser invocation is one whole session, and a session cannot span two CLI processes: the command opens the page, snapshots it, runs the steps in --steps, optionally reads, then closes. Use the browser_session MCP tool or the REST API when a session has to outlive the call that opened it.
| Command | Purpose |
|---|---|
scrape | Fetch a URL (fetch_url); add --extract for cleaned text/markdown |
search | Web search (--provider crawlforge or searxng) |
crawl | Deep multi-page crawl with depth/page/concurrency controls |
map | Generate a sitemap (--format json or xml) |
extract | Structured extraction via --schema (CSS) or --prompt (LLM) |
track | Track content changes (--selector, --threshold) |
analyze | Sentiment, entities, and readability analysis |
research | Multi-source deep research with source verification |
stealth | Anti-bot browser mode (--engine playwright or camoufox) |
batch | Scrape many URLs from a newline-delimited file |
actions | Browser automation from a JSON action script (--script) |
browser | Browser session run end to end: open, snapshot, act on the refs it returns, read, close (--steps, --read) |
localize | Locale/geo-aware fetch (--locale, --country, --currency) |
llmstxt | Generate llms.txt and llms-full.txt (--include-full) |
template | Run a pre-built site template (--list shows all 10) |
monitor | Scheduled change monitoring (--interval, --webhook) |
Skills installers
Two helper commands copy CrawlForge skill files into your AI coding tool so it knows how to use every CrawlForge command.
| Command | Purpose |
|---|---|
install-skills | Install CrawlForge skills into Claude Code, Cursor, or VS Code (--target, --dry-run) |
uninstall-skills | Remove installed CrawlForge skill files (--target) |
the same binary also starts the server over stdio with crawlforge mcp (alias crawlforge serve). MCP clients normally launch it for you—see the Claude Desktop and Cursor guides.
Examples
1. Scrape a page as clean markdown
crawlforge scrape https://example.com --extract --format markdownWithout --extract the command runs fetch_url (raw headers + body); with it, extract_content returns cleaned text or markdown. Add --pretty to print indented JSON.
2. Run deep research with multiple sources
crawlforge research "AI agent frameworks 2026" --depth deep --max-urls 10Analyzes up to 10 sources, verifies citations, and emits a structured research report. Depth is basic, standard, or deep. Costs 10 credits.
3. Use a pre-built site template
crawlforge template github-repo https://github.com/anthropics/claude-code --pretty
crawlforge template --list # show all 10 templatesTemplates extract a known schema for popular sites (GitHub repos, Amazon products, YouTube videos, and more) without writing selectors yourself. The target URL is a positional argument.
4. Pipe JSON into other tools
crawlforge search "nodejs MCP server" --json | jq '.results[].url'--json emits single-line JSON for shell pipelines and CI; --quiet returns only an exit code.
Global Flags
These flags work with any command and shape how output is delivered.
| Flag | Description |
|---|---|
--json | Output compact JSON (pipe-friendly) |
--pretty | Output pretty-printed JSON |
--quiet | Suppress all stdout output (exit code only) |
--api-key <key> | Override the CRAWLFORGE_API_KEY env var |
--timeout <ms> | Global request timeout in ms (default: 30000) |
Next Steps
Ready to scrape from the terminal?
Sign up for free credits and start running commands in minutes.