CrawlForge MCP
API Reference

Hosted monitors

Give CrawlForge a set of pages and a cron schedule. It fetches and diffs them on its own scheduler, records every check, and tells you what changed by email or signed webhook. There is no per-monitor fee: each check bills the track_changes price per compared target.

Overview

A monitor is a name, up to 20 targets (a url plus an optional CSS selector), a five-field cron schedule with an IANA timezone, and where to send the results. CrawlForge's scheduler runs every 5 minutes and starts each due check itself, so nothing has to be running on your side.

The first check of a target captures a baseline and reports it as new. Every later check diffs against the previous check and rolls the baseline forward, so each check reports the change since the previous one. Baselines live until the monitor is deleted; checks are pruned after retention_days.

Every endpoint takes the same X-API-Key header as the tools (or Authorization: Bearer cf_…). Management calls charge nothing and work at zero credits. Responses use the tools' envelope, { success, data }, and errors carry error.code and error.message.

Endpoints

MethodPathWhat it does
POST/api/v1/monitorsCreate a monitor. Returns 201 with the monitor, including its webhook_secret.
GET/api/v1/monitorsList monitors, each with its newest check as last_check. limit and cursor query parameters; the body carries next_cursor. Never returns webhook_secret.
GET/api/v1/monitors/{id}One monitor, including webhook_secret.
PATCH/api/v1/monitors/{id}Update any subset of the create fields. Setting status to paused sets next_run_at to null.
DELETE/api/v1/monitors/{id}Delete the monitor. Returns { id, deleted: true }.
POST/api/v1/monitors/{id}/runRun one check now, inline, and return it with pages. 409 MONITOR_RUNNING while a check is already in progress.
GET/api/v1/monitors/{id}/checksCheck history without pages. limit and cursor query parameters.
GET/api/v1/monitors/{id}/checks/{check_id}One check with pages and webhook deliveries.

Create a monitor

Only name and targets are required. The defaults give you an hourly check in UTC with 30 days of history.

terminalBash
curl -X POST https://crawlforge.dev/api/v1/monitors \
  -H "X-API-Key: cf_test_YOUR_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "competitor pricing",
    "targets": [{ "url": "https://competitor.com/pricing", "selector": ".pricing-table" }],
    "schedule_cron": "0 * * * *",
    "webhook_url": "https://example.com/hooks/crawlforge"
  }'

Every field the body accepts. PATCH takes any subset of the same fields.

FieldTypeDefaultDescription
namestringrequired1 to 80 characters.
targetsarrayrequired1 to 20 objects of { url, selector? }. URLs must be public http(s) addresses; a private or local address is refused with 400.
schedule_cronstring0 * * * *Five-field cron expression. Consecutive runs must be at least 5 minutes apart.
timezonestringUTCIANA timezone the schedule is evaluated in, e.g. Europe/Madrid.
notify_emailsstring[]—Up to 5 addresses emailed when a check finds something. See Notifications.
webhook_urlstring—Public https URL that receives the signed events. See Notifications.
webhook_secretstringgenerated16 to 128 characters used to sign every delivery. Generated when omitted and webhook_url is set. Returned on create and on GET one, never on list.
retention_daysnumber301 to 365. Checks older than this are pruned; baselines are kept until the monitor is deleted.
statusstringactiveactive or paused. A paused monitor keeps its baselines and has no next_run_at.

Full body

Two targets, weekday mornings in Madrid, email plus webhook with your own secret, 90 days of history:

POST /api/v1/monitorsJson
{
  "name": "competitor pricing",
  "targets": [
    { "url": "https://competitor.com/pricing", "selector": ".pricing-table" },
    { "url": "https://competitor.com/changelog" }
  ],
  "schedule_cron": "0 9 * * 1-5",
  "timezone": "Europe/Madrid",
  "notify_emails": ["alerts@example.com"],
  "webhook_url": "https://example.com/hooks/crawlforge",
  "webhook_secret": "3f9a1c7e5b2d48f0a6c1e9d7b5a3f2c4",
  "retention_days": 90,
  "status": "active"
}

The monitor object

Create, GET one and PATCH return the monitor. next_run_at is the next slot, last_check_at the time of the newest check, and estimated_credits_per_month is explained under Billing. List responses add last_check, the newest check without its pages.

201 CreatedJson
{
  "success": true,
  "data": {
    "id": "cmf9k2x1a0001s8h4b7d3q2wz",
    "name": "competitor pricing",
    "targets": [
      { "url": "https://competitor.com/pricing", "selector": ".pricing-table" }
    ],
    "schedule_cron": "0 * * * *",
    "timezone": "UTC",
    "notify_emails": [],
    "webhook_url": "https://example.com/hooks/crawlforge",
    "webhook_secret": "3f9a1c7e5b2d48f0a6c1e9d7b5a3f2c4",
    "status": "active",
    "next_run_at": "2026-09-07T15:00:00.000Z",
    "last_check_at": null,
    "retention_days": 30,
    "estimated_credits_per_month": 2160,
    "created_at": "2026-09-07T14:12:08.000Z",
    "updated_at": "2026-09-07T14:12:08.000Z"
  }
}

List, run now, and read checks

List

terminalBash
curl "https://crawlforge.dev/api/v1/monitors?limit=20" \
  -H "X-API-Key: cf_test_YOUR_KEY"

# Next page: pass the next_cursor from the previous response
curl "https://crawlforge.dev/api/v1/monitors?limit=20&cursor=NEXT_CURSOR" \
  -H "X-API-Key: cf_test_YOUR_KEY"

Run now

Runs one check immediately, outside the schedule, and returns it with its pages. The same billing rules apply. A second call while that check is still running answers 409 MONITOR_RUNNING.

terminalBash
# Run one check now and get it back with its pages
curl -X POST https://crawlforge.dev/api/v1/monitors/MONITOR_ID/run \
  -H "X-API-Key: cf_test_YOUR_KEY"

Checks

terminalBash
# Check history (no pages)
curl "https://crawlforge.dev/api/v1/monitors/MONITOR_ID/checks?limit=20" \
  -H "X-API-Key: cf_test_YOUR_KEY"

# One check, with its pages and webhook deliveries
curl https://crawlforge.dev/api/v1/monitors/MONITOR_ID/checks/CHECK_ID \
  -H "X-API-Key: cf_test_YOUR_KEY"

The check object

One check with two targets: the pricing table changed and was charged, the docs page hit a Cloudflare challenge and was not. deliveries is the webhook log for the check.

GET /api/v1/monitors/{id}/checks/{check_id}Json
{
  "success": true,
  "data": {
    "id": "cmf9k5r8t0003s8h4m1n6p4vx",
    "monitor_id": "cmf9k2x1a0001s8h4b7d3q2wz",
    "started_at": "2026-09-07T15:00:02.000Z",
    "finished_at": "2026-09-07T15:00:05.000Z",
    "status": "completed",
    "summary": { "total": 2, "new": 0, "changed": 1, "unchanged": 0, "blocked": 1, "errored": 0 },
    "credits_reserved": 6,
    "credits_charged": 3,
    "error": null,
    "pages": [
      {
        "url": "https://competitor.com/pricing",
        "selector": ".pricing-table",
        "status": "changed",
        "change_percent": 12.5,
        "structural_similarity": 0.9713,
        "added_count": 4,
        "removed_count": 2,
        "added_samples": ["Pro $79 / month", "Includes 50,000 credits"],
        "removed_samples": ["Pro $99 / month", "Includes 25,000 credits"],
        "content_hash": "9f2c0b7e4d1a8c6f3e5b2a9d7c4f1e8b0a6d3c2f5e9b1a7d4c8f2e6b3a9d1c7f",
        "error_code": null,
        "error": null,
        "blocked_vendor": null,
        "duration_ms": 1180,
        "charged": true
      },
      {
        "url": "https://competitor.com/docs",
        "selector": null,
        "status": "blocked",
        "change_percent": null,
        "structural_similarity": null,
        "added_count": null,
        "removed_count": null,
        "added_samples": [],
        "removed_samples": [],
        "content_hash": null,
        "error_code": "BLOCKED",
        "error": "Challenge page served by cloudflare",
        "blocked_vendor": "cloudflare",
        "duration_ms": 640,
        "charged": false
      }
    ],
    "deliveries": [
      {
        "id": "cmf9k5rb20005s8h4q7w2e9rt",
        "event": "monitor.page",
        "url": "https://example.com/hooks/crawlforge",
        "attempts": 1,
        "status": "delivered",
        "http_status": 200,
        "error": null,
        "delivered_at": "2026-09-07T15:00:05.100Z"
      },
      {
        "id": "cmf9k5rb20006s8h4z3x8c5vb",
        "event": "monitor.check.completed",
        "url": "https://example.com/hooks/crawlforge",
        "attempts": 1,
        "status": "delivered",
        "http_status": 200,
        "error": null,
        "delivered_at": "2026-09-07T15:00:05.400Z"
      }
    ]
  }
}

Billing and the monthly estimate

There is no per-monitor fee and creating, listing, updating or deleting a monitor costs nothing. Each check holds 3 credits per target, the track_changes price, and keeps 3 for every target whose status is new, changed or unchanged. Targets that came back blocked or error are not charged. Every check writes one track_changes row to the usage log.

If your balance cannot cover the hold, the check is recorded as insufficient_credits and nothing is fetched.

estimated_credits_per_month on the monitor is the number of times the schedule fires in the next 30 days × targets × 3. It is the ceiling for a month in which every target is fetched and compared every time.

schedule_cronTargetsRuns in 30 daysestimated_credits_per_month
0 * * * *17202,160
*/15 * * * *12,8808,640
0 9 * * *530450

Check and page statuses

Each check carries one status, a summary with the count per page status, and credits_reserved / credits_charged.

Check statusMeaning
runningThe check is in progress. POST …/run answers 409 MONITOR_RUNNING meanwhile.
completedEvery target was processed. Read summary and pages.
failedThe check could not finish; error says why.
skipped_overlapThe previous check was still running when this slot arrived. Nothing was fetched.
insufficient_creditsThe balance could not cover the hold. Nothing was fetched.

Each entry in pages carries one status. A changed page includes change_percent, structural_similarity, added_count, removed_count and up to 20 sample lines of up to 500 characters in added_samples and removed_samples; the counts are the real totals.

Page statusMeaningCharged
newFirst capture of this target. The baseline was written; there is nothing to diff yet.Yes
changedThe content differs from the previous check. The baseline rolls forward to this capture.Yes
unchangedSame content hash as the previous check.Yes
blockedThe target answered with a challenge wall (Cloudflare, DataDome, PerimeterX, Akamai, Amazon, Vercel) or an unusable document. blocked_vendor names it. Never reported as a change.No
errorThe fetch failed, the selector matched nothing, or robots.txt disallows the path. error_code carries the code: FETCH_FAILED, SELECTOR_NOT_FOUND, ROBOTS_DISALLOWED, and so on.No
robots.txt is always respected A monitor has no respect_robots override. A target whose path robots.txt disallows for CrawlForge is reported as error with ROBOTS_DISALLOWED on every check and is never charged.

Schedule rules

schedule_cron is a standard five-field cron expression — minute, hour, day of month, month, day of week — evaluated in timezone. Two consecutive runs must be at least 5 minutes apart, so * * * * * and */2 * * * * are refused with 400.

CrawlForge's scheduler wakes every 5 minutes and starts every check whose slot has passed, so a check begins within 5 minutes of its slot. next_run_at on the monitor is the next slot; pausing the monitor sets it to null.

A monitor whose previous check is still running when its next slot arrives records a skipped_overlap check instead of starting a second one.

ExpressionRuns
0 * * * *Every hour, on the hour (the default).
*/15 * * * *Every 15 minutes.
0 9 * * 1-509:00 Monday to Friday, in timezone.
0 6,18 * * *06:00 and 18:00 every day.
30 2 1 * *02:30 on the first of every month.

Notifications

Email. When notify_emails is set, a message goes out only for a check that has at least one new, changed, blocked or error page. A check whose every target is unchanged sends nothing.

Webhooks. When webhook_url is set, every delivery is a POST with a JSON body of { event, id, timestamp, data } and these headers:

HeaderValue
Content-Typeapplication/json
X-Webhook-Eventmonitor.page or monitor.check.completed
X-Webhook-IDDelivery id, stable across retries, so a receiver can drop a duplicate.
X-Webhook-TimestampUnix time in milliseconds when the delivery was sent.
X-Webhook-Signaturesha256= followed by the hex HMAC-SHA256 of the exact raw body, keyed by webhook_secret.

Two events. Page events go out first, then the check summary.

eventWhendata
monitor.pageOne per page whose status is not unchanged.{ monitor: { id, name }, check_id, page, dashboard_url } — page is the full page result, samples included.
monitor.check.completedEvery completed check.{ monitor, check: { id, started_at, finished_at, status, summary, credits_charged }, pages, dashboard_url } — each entry in pages has url, selector, status, change_percent, added_count, removed_count, error_code, without samples.
POST webhook_url — X-Webhook-Event: monitor.check.completedJson
{
  "event": "monitor.check.completed",
  "id": "cmf9k5rb20006s8h4z3x8c5vb",
  "timestamp": 1788793205400,
  "data": {
    "monitor": { "id": "cmf9k2x1a0001s8h4b7d3q2wz", "name": "competitor pricing" },
    "check": {
      "id": "cmf9k5r8t0003s8h4m1n6p4vx",
      "started_at": "2026-09-07T15:00:02.000Z",
      "finished_at": "2026-09-07T15:00:05.000Z",
      "status": "completed",
      "summary": { "total": 2, "new": 0, "changed": 1, "unchanged": 0, "blocked": 1, "errored": 0 },
      "credits_charged": 3
    },
    "pages": [
      {
        "url": "https://competitor.com/pricing",
        "selector": ".pricing-table",
        "status": "changed",
        "change_percent": 12.5,
        "added_count": 4,
        "removed_count": 2,
        "error_code": null
      },
      {
        "url": "https://competitor.com/docs",
        "selector": null,
        "status": "blocked",
        "change_percent": null,
        "added_count": null,
        "removed_count": null,
        "error_code": "BLOCKED"
      }
    ],
    "dashboard_url": "https://www.crawlforge.dev/dashboard/monitors/cmf9k2x1a0001s8h4b7d3q2wz"
  }
}

Delivery. Up to 4 attempts — the first plus three retries — with 1 s, 2 s and 4 s between them and a 10 s timeout each. A 2xx response counts as delivered. A 4xx other than 408 or 429 is not retried. The delivery log rides on the check: GET …/checks/{check_id} returns deliveries with the attempt count, HTTP status and error of each event.

Verifying the signature

Compute the HMAC over the raw request body, before any JSON parsing, and compare it in constant time. The header set and signature scheme are the same ones the CrawlForge MCP server's webhooks use, so one receiver verifies both.

verifyWebhook.tsTypescript
import { createHmac, timingSafeEqual } from 'node:crypto';
import express from 'express';

// Sign the RAW body exactly as received. Parsing and re-serialising the JSON
// first changes the bytes and the signature no longer matches.
function verifyCrawlForgeWebhook(rawBody: Buffer, signatureHeader: string, secret: string): boolean {
  const expected = Buffer.from('sha256=' + createHmac('sha256', secret).update(rawBody).digest('hex'));
  const received = Buffer.from(signatureHeader);
  return expected.length === received.length && timingSafeEqual(expected, received);
}

const app = express();

// express.raw keeps the body as a Buffer; express.json would have parsed it.
app.post('/hooks/crawlforge', express.raw({ type: 'application/json' }), (req, res) => {
  const secret = process.env.CRAWLFORGE_WEBHOOK_SECRET!;
  if (!verifyCrawlForgeWebhook(req.body, req.header('X-Webhook-Signature') ?? '', secret)) {
    return res.status(401).end();
  }

  const { event, data } = JSON.parse(req.body.toString('utf8'));
  if (event === 'monitor.page') {
    console.log(data.page.status, data.page.url, data.page.change_percent);
  } else if (event === 'monitor.check.completed') {
    console.log('check', data.check.id, data.check.summary);
  }
  res.status(204).end();
});

Limits

  • 50 monitors per account. The 51st create answers 409 MONITOR_LIMIT_REACHED.
  • 20 targets per monitor and 5 addresses in notify_emails.
  • At least 5 minutes between consecutive runs.
  • retention_days from 1 to 365, default 30. Baselines are kept until the monitor is deleted.
  • Diff samples: up to 20 lines of up to 500 characters in each of added_samples and removed_samples.

Errors

StatusMeaning
400 VALIDATION_ERRORA field is out of range, the cron expression is invalid or fires more often than every 5 minutes, or a target URL is not a public http(s) address.
401Missing or invalid API key.
404No monitor or check with that id on your account.
409 MONITOR_LIMIT_REACHEDYou already have 50 monitors.
409 MONITOR_RUNNINGPOST …/run while a check is in progress.
Related
track_changes
One-off baseline and compare, and operation: "monitor" to create a monitor from a single URL; the MCP server's create_scheduled_monitor with hosted: true creates the same kind of monitor.
Dashboard
Create, pause and inspect monitors and their checks without writing a request.

Footer

CrawlForge MCP

Enterprise web scraping for AI Agents. 31 specialized MCP tools designed for modern developers building intelligent systems.

Product

  • Features
  • Playground
  • Pricing
  • Use Cases
  • Integrations
  • Alternatives
  • Changelog

Resources

  • Getting Started
  • API Reference
  • Templates
  • Guides
  • Blog
  • Glossary
  • FAQ
  • Sitemap

Developers

  • MCP Protocol
  • Claude Desktop
  • Cursor IDE
  • LangChain
  • LlamaIndex

Company

  • About
  • Contact
  • Privacy
  • Terms
  • Acceptable Use
  • Security
  • Cookies

Stay updated

Get the latest updates on new tools and features.

Built with Next.js and MCP protocol

© 2025-2026 CrawlForge. All rights reserved.