Competitor Monitoring API for Consultants: One Per Client

Every retainer has a sentence in it about keeping an eye on the market. Nobody bills for that sentence separately, and it is the first thing to slip in a busy month.

The manual version is a browser tab per client and a good memory. The built version is a scraper, which holds up until the third engagement. By then the thing you wrote for client one is quietly diffing a page that was redesigned in June, and nobody has noticed, because a monitor finding nothing and a monitor that is dead look identical from the outside.

A competitor monitoring api for consultants carries a constraint an in-house one does not. You are watching several markets at once, on other people's behalf, for exactly as long as each engagement lasts.

How do you keep one client's watch separate from another's?

One monitor per client. A monitor is a name, up to 20 URLs, a cron schedule and a timezone, and an account holds 50 of them. That is fifty engagements of twenty pages each before any cap is worth thinking about.

curl -X POST https://crawlforge.dev/api/v1/monitors \
  -H "X-API-Key: cf_live_YOUR_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "Northwind - competitor pricing",
    "targets": [
      { "url": "https://rival-a.com/pricing", "selector": ".pricing-table" },
      { "url": "https://rival-b.com/careers", "selector": "main" }
    ],
    "schedule_cron": "0 7 * * 1",
    "timezone": "Europe/London",
    "retention_days": 365,
    "notify_emails": ["you@yourfirm.com"]
  }'

Name it after the client. That is not housekeeping: every check writes one row to your request log carrying the monitor's name, its id and the credits it actually charged, so what did we spend watching Northwind's market in August becomes a question the log answers rather than one you reconstruct the night before invoicing.

Set timezone per monitor as well. The client's Monday morning is not necessarily yours, and a 7am check in the wrong hemisphere lands after the call it was meant to inform.

Can you drive it from Claude Code instead of curl?

Yes, and for the first week of an engagement that is the better tool. CrawlForge's MCP server exposes the same machinery through track_changes, so the discovery pass — what is even worth watching at this client? — happens in the editor you are already drafting the deck in.

{
  "url": "https://rival-a.com/pricing",
  "operation": "create_scheduled_monitor",
  "trackingOptions": { "customSelectors": [".pricing-table"] },
  "scheduledMonitorOptions": {
    "hosted": true,
    "name": "Northwind - rival A pricing",
    "schedule": "0 7 * * 1"
  }
}

Two things about that call are worth knowing before you lean on it, and neither is obvious from the schema.

hosted defaults to false. A local scheduled monitor runs inside the MCP server process, so it stops the moment you quit your editor. Fine for a discovery week, useless for a retainer. Pass hosted: true and the monitor moves to CrawlForge's own scheduler, where it fires whether or not your laptop is open.

A plain-English goal does not survive the move. Locally you can write goal: "tell me when the Pro plan price changes" and a model judges each diff against it. Hosted monitors ignore goal and notify on every changed, new, blocked or errored page. The tool does return a warning saying so, which is easy to scroll past. Scope a hosted monitor with a CSS selector instead of a sentence.

There is also an arithmetic problem rather than a trap. The MCP path creates one monitor per URL, fanning customSelectors into targets on that single page, so twenty client pages that way costs twenty of your fifty monitors. The REST POST above puts all twenty in one. Use MCP for the week you are working out what matters, then one REST call for the monitor you intend to keep.

What happens between engagement phases?

Pause it. PATCH the monitor to status: "paused" and it keeps every baseline and gets no next run, so it costs nothing while it waits. Resume, and the first check diffs against the last capture from before the pause. The quiet months come back as a single change rather than as a hole in the record.

When a phase needs an answer today rather than at the next slot, POST /api/v1/monitors/{id}/run runs one check inline and hands back the pages.

Where does the deliverable actually come from?

The check history, not your memory of it. GET /api/v1/monitors/{id}/checks lists checks newest first, and reading one back returns its pages with change_percent, added and removed counts, and up to 20 sample lines from each side of the diff. That is quotable material for the what changed this quarter slide, instead of a screenshot somebody took in July.

Set retention_days on day one. It runs from 1 to 365 and defaults to 30, and checks older than the window are pruned at the end of each run and do not come back. A twelve-month engagement that wants its own history at close-out has to say 365 at the start; there is no way to ask for it in month eleven. The endpoint reference is in the docs.

What does watching a client's market cost?

Three credits per target fetched and compared, the track_changes price, whether the call came from REST or MCP. Blocked and errored targets are not charged. Creating, updating, pausing and deleting monitors are free.

Twenty pages on a Monday schedule is four or five checks in a 30-day window, so 240 to 300 credits. Ten clients on that pattern stay under 3,000 credits a month between them. There are no seats and no ceiling on how many pages you may watch, which is a different post.

The consequence worth acting on is that breadth is cheap and frequency is not, so watch more pages less often. A rival's careers page quietly posting an account executive role in Munich tells you more about their next two quarters than their pricing page will, and it changes maybe twice a year.

What it won't do

• No client-facing view. One account, one key, no per-client login and no access boundary between monitors. What the client sees is what you send them.
• No contact or firmographic data. It reads pages you point it at, above whatever database your research already runs on. It is not an Apollo or a ZoomInfo and does not pretend to be.
• robots.txt is always respected, with no override. A disallowed path reports ROBOTS_DISALLOWED on every check and is never charged for it.
• Five minutes minimum between runs, which makes this a daily and weekly instrument. That is the rhythm of a report anyway.

If you want the change routed into a CRM or a channel rather than written into a document, the webhook side is here.

Start free with 1,000 credits, about two months of one client's five-page daily watch, or open the monitors dashboard and read a real check before you write any code.