Market Research API for Agencies: One Setup, Every Client

An agency's research problem is rarely one hard question. It is the same handful of questions, for every client, every month, plus a fresh landscape whenever new business wants a pitch deck by Thursday.

A market research api for agencies has to be built for that repetition: one setup that runs the whole roster, a cost you can split by client, and a fast first pass on a prospect you have not won yet. CrawlForge is the web layer for that. It reads the public pages and search results you point it at and returns what they say. It is not a market-size or contact database.

What does the monthly client report need?

For most agencies, two things per client: where the client ranks against its competitors for the searches that matter, and what those competitors changed on their own sites. Both are public, both move monthly, and neither sits in a dataset you can license by the row.

Two calls cover them:

• serp_rank (5 credits) returns a domain's organic Google position for one keyword, checked from the location_name, language_code and device you set per client. The response carries found, rank, the ranking url and checked_at.
• scrape with a question format (3 credits) returns the five sentences on a competitor's page that best answer your question, verbatim, each with an offset. The RevOps post covers how that format behaves; here it is one row in a template.

How do you run one template across every client?

Keep everything client-specific in data, not in code. Each client is a small config: its domain, its market, its keywords, its competitors and the questions you ask of their pages.

{
  "client": "Northwind",
  "domain": "northwind.co.uk",
  "location_name": "United Kingdom",
  "device": "mobile",
  "keywords": ["office cleaning contracts", "commercial cleaning manchester"],
  "competitors": ["rival-a.co.uk", "rival-b.co.uk", "rival-c.co.uk"],
  "pages": ["https://rival-a.co.uk/pricing", "https://rival-b.co.uk/services"],
  "questions": ["What is the minimum contract length?"]
}

One workflow then loops over the roster. For each keyword it calls serp_rank once for the client and once per competitor, since each call locates one domain. For each competitor page it runs one scrape per question.

curl -X POST https://crawlforge.dev/api/v1/tools/serp_rank \
  -H "X-API-Key: cf_live_NORTHWIND_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "keyword": "office cleaning contracts",
    "target": "northwind.co.uk",
    "location_name": "United Kingdom",
    "device": "mobile"
  }'

In n8n that loop is a node that reads the configs feeding two HTTP Request nodes (the n8n setup is in the docs); in a script it is two nested loops. Client thirteen is a new config file, not a new workflow.

Give each client its own named API key and the request log splits the month's spend by client, which the consultants post walks through. An account holds ten keys, so past ten clients, group them by retainer.

How do you research a prospect before the pitch?

Start with one deep_research run for 10 credits. Give it the prospect's category and market, and it searches, fetches up to ten sources and returns the best-matching passages verbatim with a sources list of URLs. Then run serp_rank on the prospect's three most obvious keywords for 15 more.

That is 25 credits per prospect, under a dime at pack rates, and it puts a "where you rank today" slide in the deck. That slide tends to start the conversation, and it costs almost nothing when the pitch is lost.

What does a roster cost each month?

Twelve clients, each with five keywords, three competitors, six competitor pages and two questions per page:

That is about $5 a month at the one-time 5,000-credit pack rate ($14), with no seats and no monthly minimum. Rank checks are the expensive line, because they multiply keywords by domains, so trim competitors from the rank loop before you trim keywords.

What won't it do?

• No market sizing. No TAM, panels or survey data. If the deck needs a market-size figure, it comes from a report.
• No contact or firmographic data. It reads pages and search results. It is not a company database.
• No client-facing portal. There is no white-label dashboard; the report is whatever you build from the rows.
• robots.txt is respected by default on the pages you scrape. A disallowed page is refused before it is fetched and costs nothing.

If a client wants to hear when a competitor's page changes rather than reading about it at month end, that is a monitor per client, not a more frequent run of this.

Start free with 1,000 credits. That is enough to run this template for seven clients for a month before you wire it into the whole roster.