Web Scraping From the CLI: The CrawlForge CLI Guide

Most AI tools love to be agents. The CrawlForge CLI is built for the opposite: scriptable, terminal-first, predictable. You install it, set an environment variable, and 17 of CrawlForge's 31 tools become shell commands. JSON in, JSON out. Pipe to jq, schedule with cron, run in CI -- it works the same way everywhere.

Table of Contents

• What Is the CrawlForge CLI?
• Install in 30 Seconds
• The 16 Commands at a Glance
• Your First Scrape
• Piping JSON Output to jq
• Scheduling With Cron
• CLI vs MCP vs Raw API
• Three Real-World Workflows
• Global Flags Reference
• What It Costs

What Is the CrawlForge CLI?

The CrawlForge CLI ships inside the crawlforge-mcp-server package as the crawlforge command and exposes 17 of CrawlForge's 31 tools as terminal commands. A single global install gives you both the MCP server and the CLI. It does not need a long-running process or an MCP client: you type crawlforge scrape <url>, it makes an HTTPS call to CrawlForge's API, and prints JSON to stdout. That is the entire story.

It exists because half the scraping work people do is not agent-shaped. Cron jobs, CI steps, one-off research, ad-hoc pulls from a shell -- those want plain old commands, not a JSON-RPC handshake.

Install in 30 Seconds

npm install -g crawlforge-mcp-server
export CRAWLFORGE_API_KEY="cf_live_your_key_here"
crawlforge --help

That is it. No config file, no auth flow, no service to start. If you do not have an API key yet, grab one at crawlforge.dev/signup -- you get 1,000 free credits on signup.

To make the env var permanent on macOS or Linux:

echo 'export CRAWLFORGE_API_KEY="cf_live_..."' >> ~/.zshrc
source ~/.zshrc

On Windows (PowerShell):

[Environment]::SetEnvironmentVariable("CRAWLFORGE_API_KEY", "cf_live_...", "User")

The 16 Commands at a Glance

Every command maps to one or more CrawlForge tools:

Your First Scrape

The simplest possible call:

crawlforge scrape https://news.ycombinator.com

What comes back is the page's main content as JSON:

{
  "url": "https://news.ycombinator.com",
  "title": "Hacker News",
  "content": "Hacker News new | past | comments | ask...",
  "links": ["https://news.ycombinator.com/from?site=...", "..."],
  "fetched_at": "2026-05-21T10:14:33Z",
  "credits_used": 1
}

Want just the URLs? Pipe to jq:

crawlforge scrape https://news.ycombinator.com --json | jq '.links[]'

Want it in a file? Redirect stdout:

crawlforge scrape https://news.ycombinator.com --pretty > hn.json

Piping JSON Output to jq

This is the workflow that makes the CLI worth installing. Everything outputs JSON, and JSON pipes into anything.

Get the HN front-page story titles:

crawlforge template hacker-news-front-page https://news.ycombinator.com --json \
  | jq -r '.stories[] | .title'

Search the web and extract URLs:

crawlforge search "best web scraping libraries 2026" --json \
  | jq '.results[] | .url'

Scrape a page and count words:

crawlforge scrape https://example.com --json \
  | jq -r '.content' \
  | wc -w

Batch scrape, then filter for error responses:

crawlforge batch urls.txt --json \
  | jq '.results[] | select(.status_code >= 400)'

The pattern: --json gives you machine-readable output, then jq slices and dices.

Scheduling With Cron

A daily check on a competitor's pricing page:

# crontab -e
0 9 * * * /usr/local/bin/crawlforge track https://competitor.com/pricing --json > /var/log/pricing.json

A nightly research run:

0 2 * * * /usr/local/bin/crawlforge research "AI tooling news" --depth standard --pretty > /var/log/research.json

A weekly llms.txt regeneration for your own site:

0 3 * * 0 /usr/local/bin/crawlforge llmstxt https://yoursite.com --include-full > /var/www/yoursite.com/llms.txt

In CI? Use the same commands in your GitHub Actions YAML. The CLI checks CRAWLFORGE_API_KEY first, so just set it as a repository secret.

# .github/workflows/daily-research.yml
- name: Run weekly research
  env:
    CRAWLFORGE_API_KEY: ${{ secrets.CRAWLFORGE_API_KEY }}
  run: |
    npm install -g crawlforge-mcp-server
    crawlforge research "industry news" --depth standard --pretty > report.json

CLI vs MCP vs Raw API: When to Use Each

Rule of thumb: if a human is typing the command, use the CLI. If an LLM is selecting the tool, use MCP. If a server is calling it in a loop, use the raw API.

Three Real-World Workflows

1. Competitive Pricing Monitor

A shell script that runs daily, scrapes three competitor pricing pages, diffs against yesterday's snapshot, and posts to Slack if anything changed.

#!/bin/bash
for url in $(cat competitors.txt); do
  crawlforge track "$url" --json \
    > "snapshots/$(date +%F)-$(basename $url).json"
done

# Diff against yesterday's snapshot
diff "snapshots/$(date -v-1d +%F)-pricing.json" \
     "snapshots/$(date +%F)-pricing.json" \
  || curl -X POST $SLACK_WEBHOOK -d '{"text": "Pricing changed"}'

Cost: ~9 credits per day (3 competitors × 3 credits for track).

2. Lead Enrichment From a CSV

Read a CSV of company domains, scrape each homepage for contact info, write enriched data back.

while IFS=, read -r company domain; do
  data=$(crawlforge scrape "https://$domain" --json)
  email=$(echo "$data" | jq -r '.metadata.contact_email // empty')
  echo "$company,$domain,$email" >> enriched.csv
done < companies.csv

Cost: 1 credit per company.

3. Research Report Pipeline

A weekly Sunday cron that runs a research query and emails the synthesized summary to the team.

crawlforge research "AI agent frameworks news this week" --depth deep --pretty > report.json
jq -r '.summary' report.json \
  | mail -s "Weekly AI report" team@example.com

Cost: 10 credits per run (research includes the synthesized summary).

Global Flags Reference

These work on every command:

• --json -- compact, machine-readable JSON (pipe-friendly)
• --pretty -- pretty-printed JSON
• --quiet -- suppress all stdout output (exit code only)
• --api-key <key> -- override the CRAWLFORGE_API_KEY env var
• --timeout <ms> -- override the default 30s timeout

To write results to a file, redirect stdout: crawlforge scrape <url> --pretty > out.json.

What It Costs

The CLI itself is free. You pay only for the underlying tool calls, billed against your existing credit balance. No extra subscription, no per-invocation fee. A daily cron that runs track against three URLs and research once a week costs roughly 100 credits per month -- months of runway even on the one-time 1,000-credit free tier.

Ready to install? Get your free API key at crawlforge.dev/signup and run npm install -g crawlforge-mcp-server. New here? Read the v4.2.2 launch announcement for everything new, or the original MCP quickstart for the MCP version instead.