How to Run a Keyword Gap Analysis with an MCP Server

Most keyword research tools answer the wrong question. They tell you how hard a term is on a scale of 1 to 100, which is a summary of a summary. What you actually need to know is who holds the top ten and what kind of pages they are, because that determines whether the position can be taken at all.

This guide builds that analysis as a reproducible pipeline over an MCP connection. It is the exact method behind our 2026 web scraping SERP study, including the mistake that invalidates most DIY keyword research.

Table of contents

• The three stages
• The mistake that makes most keyword research fiction
• Stage 1: Discover candidate terms
• Stage 2: Pull real organic positions
• Stage 3: Classify the page types
• Scoring the gap
• Checking intent before you commit
• Credit cost and runtime
• Running it from the REST API
• Frequently asked questions

The three stages

Stage 3 is usually judgement rather than a tool call — you are labelling nine URLs, which is faster by eye than by API.

The mistake that makes most keyword research fiction

Start here, because it invalidates everything downstream.

A search API's result order is not Google's ranking. Search tools retrieve results and then re-rank them. CrawlForge's search_web re-orders what it retrieves using BM25, semantic similarity, authority, and freshness weights, and it reports exactly that in its response:

{
  "processing": {
    "ranking": {
      "algorithmsUsed": ["bm25", "semantic", "authority", "freshness"],
      "weightsApplied": { "bm25": 0.4, "semantic": 0.3, "authority": 0.2, "freshness": 0.1 }
    }
  }
}

That re-ranking makes results better for research and useless as a rank measurement. If you read position 1 out of a search API and record it as a Google ranking, every conclusion built on it is wrong.

Use search_web to learn which domains are in play. Use serp_rank — which returns Google's own rank_group value — for anything you intend to call a position.

Stage 1: Discover candidate terms

Cast wide first. You are looking for the vocabulary of the category, not rankings yet.

import { Client } from '@modelcontextprotocol/sdk/client/index.js';

const client = new Client({ name: 'keyword-gap', version: '1.0.0' });

interface Candidate {
  keyword: string;
  indexedPages: number;
  domains: string[];
}

async function discover(keyword: string): Promise<Candidate> {
  const res = await client.callTool({
    name: 'search_web',
    arguments: { query: keyword, limit: 20 },
  });

  const data = JSON.parse(res.content[0].text);

  return {
    keyword,
    // Google's rough index estimate -- a crowding signal, NOT search volume.
    indexedPages: parseInt(data.total_results ?? '0', 10),
    // Presence only. This order is re-ranked and is not a Google ranking.
    domains: [...new Set(data.results.map((r) => r.displayLink))],
  };
}

Two fields matter. total_results is Google's approximate count of competing pages — a crowding signal, not volume. The domain list tells you who is in the conversation.

Stage 2: Pull real organic positions

Now measure. serp_rank runs a live Google query and returns true organic positions, plus the competitor listing.

interface RankResult {
  keyword: string;
  found: boolean;
  position: number | null;
  competitors: Array<{ position: number; domain: string; url: string }>;
}

async function measure(keyword: string, target: string): Promise<RankResult> {
  const res = await client.callTool({
    name: 'serp_rank',
    arguments: {
      keyword,
      target,
      location_name: 'United States',
      device: 'desktop',
      depth: 10,
    },
  });

  const data = JSON.parse(res.content[0].text);

  // Unconfigured deployments return configured:false rather than a fake rank.
  if (data.configured === false) {
    throw new Error('serp_rank is not configured on this server');
  }

  return {
    keyword,
    found: data.found,
    position: data.position,
    competitors: (data.results ?? []).map((r) => ({
      position: r.position,
      domain: r.domain,
      url: r.url,
    })),
  };
}

Run these sequentially, not in parallel. The live SERP endpoint performs a real-time Google query; firing several at once reliably produces timeouts rather than faster results.

depth: 10 is the right default. Deeper scans cost more and, for gap analysis, positions 11 and beyond rarely change the decision.

Stage 3: Classify the page types

This is the stage that produces the actual insight, and it is mostly judgement.

For each of the nine or ten results, label what kind of page it is:

• Commercial — homepage, product page, pricing page
• Vendor content — a company's blog post or documentation
• Independent — directory, forum thread, code repository, news article

Then count the commercial results. That single number tells you more than any difficulty score:

type PageType = 'commercial' | 'vendor-content' | 'independent';

function gapScore(labels: PageType[]): 'closed' | 'contested' | 'open' {
  const commercial = labels.filter((l) => l === 'commercial').length;
  if (commercial >= 4) return 'closed';
  if (commercial >= 1) return 'contested';
  return 'open';
}

Applied to real data from our study:

"web scraping api" is closed — six vendor homepages and product pages, each backed by years of link building. No amount of content quality moves that.

"firecrawl alternative" lands in the middle. Only two results are commercial landing pages, Apify's alternatives page and a Context comparison page, but the rest is dense with vendor blog posts all chasing the same departing customers. Winnable, crowded, and worth the effort only if you have something specific to say.

"mcp scraping" scores open, and the composition explains why: two directories, a Reddit thread, a GitHub repo, three third-party blogs, and a docs page. Nobody has published a commercial page for it.

Scoring the gap

Combine the two signals. A term is worth pursuing when the SERP is open and the crowding is manageable:

interface Scored {
  keyword: string;
  verdict: 'closed' | 'contested' | 'open';
  indexedPages: number;
  priority: 'high' | 'medium' | 'skip';
}

function prioritise(c: Candidate, verdict: Scored['verdict']): Scored['priority'] {
  if (verdict === 'closed') return 'skip';
  if (verdict === 'open' && c.indexedPages < 2_000_000) return 'high';
  return 'medium';
}

The threshold is a judgement call, not a law. The ordering is what matters: an open SERP with moderate crowding beats a contested SERP with low crowding, every time.

Checking intent before you commit

One last gate, and it cannot be automated away.

Read the titles of the top ten and ask whether those searchers want to buy what you sell. In our study, "llm scraper" passed every numeric filter — 329,000 competing pages, clearly on-topic — and failed on contact with the results. A large share were people trying to block scrapers: a YunoHost thread called "Prevent LLM scrapers/trawlers?", an r/sysadmin thread on fighting them, Akamai on bot management.

Same words, opposite intent. Ranking there earns traffic that never converts and a bounce rate that teaches Google your page is a bad answer.

Thirty seconds of reading catches what no score can.

Credit cost and runtime

A ten-keyword study:

That is a tenth of the free tier's 1,000 one-time credits.

Runtime is the real constraint. Live SERP lookups took between 10 and 41 seconds each in our measurements — the same keyword varied from 11s to 28s across consecutive runs, because the endpoint runs a live Google query whose latency depends on upstream capacity. Ten keywords run sequentially is five to ten minutes. Set a client timeout of at least 60 seconds and run the job in the background.

Running it from the REST API

If you are not inside an MCP client, the same lookup works over HTTP:

curl -X POST https://www.crawlforge.dev/api/v1/tools/serp_rank \
  -H "X-API-Key: $CRAWLFORGE_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "keyword": "mcp scraping",
    "target": "crawlforge.dev",
    "location_name": "United States",
    "depth": 10
  }'

The REST response uses rank and all_positions where the MCP tool returns position; both report the same underlying Google organic position. Failed lookups — an unreachable provider, a timeout — return an error and are not charged.

Where to take it next

Once you know which terms are open, the follow-on work is technical: making sure the pages you publish for them are actually crawlable and correctly marked up. Our guide to automating SEO audits with CrawlForge covers that half, and the serp_rank API reference documents every parameter used above.

Start free with 1,000 credits — enough for roughly 200 rank checks.