Agent Scraper: What It Is and How to Build One

A traditional scraper is a set of instructions: fetch this URL, read .product-price, return the text. An agent scraper is a goal: find the current price, whatever the page looks like today.

That difference sounds academic until the site ships a redesign. The instruction-follower returns null and your pipeline goes quiet. The goal-seeker looks at the page, notices the price moved into a modal behind a "See pricing" button, clicks it, and returns the number.

It also sounds like a free lunch, and it is not. Agent scrapers are slower, more expensive per page, and occasionally confidently wrong in ways a broken selector never is. This guide covers what an agent scraper actually is, the three ways to build one, working code for two of them, and — the part most write-ups skip — the specific conditions under which you should not use one at all.

Table of contents

• What is an agent scraper?
• How does an agent scraper work?
• Agent scraper vs traditional scraper
• The three ways to build an agent scraper
• Build one in a single call
• Build your own agent loop
• Why do agent scrapers fail in production?
• What does an agent scraper cost?
• When should you not use an agent scraper?
• Five guardrails before you ship
• Frequently asked questions

What is an agent scraper?

An agent scraper is a web scraper driven by an LLM reasoning loop rather than a fixed script. You give it an objective in natural language — "find every pricing tier on this site and what each one includes" — and it decides which pages to fetch, when to escalate to a browser, when it has enough information, and how to shape the answer. No CSS selectors, no XPath, no per-site maintenance.

Three properties separate it from everything that came before:

1. It plans. The sequence of requests is chosen at runtime from the goal, not hardcoded ahead of time.
2. It adapts. A 403, an empty result, or an unexpected layout is an input to the next decision rather than a fatal error.
3. It terminates on a condition, not a count. It stops when the objective is satisfied — which is exactly why you must impose hard stops from the outside.

The term gets used loosely, so it is worth being precise about what an agent scraper is not. A scraper that calls an LLM once to clean up extracted HTML is AI-assisted scraping — the control flow is still your script. A browser automation script with an LLM picking selectors at build time is AI-generated scraping — the LLM wrote code that then runs deterministically. Only when the model is inside the loop, choosing the next action each turn, do you have an agent scraper.

How does an agent scraper work?

Every agent scraper, regardless of framework, runs the same six-stage loop:

1. Goal. A natural-language objective, plus optional seed URLs.
2. Plan. The model decides the first action — usually a search or a fetch.
3. Act. It calls a tool: fetch a URL, run a search, drive a browser, escalate past a bot wall.
4. Observe. The tool returns content, ideally as clean markdown rather than raw HTML, because every byte lands in the context window.
5. Evaluate. Is the objective met? If not, what is missing, and which action closes the gap?
6. Emit. Return prose or, better, JSON validated against a schema you supplied.

Stages 2 through 5 repeat. And that loop is unbounded by default — which is the single most important thing to understand about running one in production.

The failure mode is not that the agent gets stuck. It is that the agent stays productive: it keeps finding one more page that seems relevant, and burns your budget doing genuinely reasonable work you did not ask for. So the hard stops belong in the orchestrator, never in the prompt. CrawlForge's agent tool enforces three of them at the infrastructure layer, where the model cannot negotiate them away:

Asking a model nicely to "only check a few pages" is not a limit. A number the runtime enforces is.

Agent scraper vs traditional scraper

Neither approach dominates. They fail in opposite directions, which is what makes the choice tractable.

That last row of failure modes is the one to internalise. A broken selector announces itself — your row count drops to zero and monitoring fires. A confabulated field does not. It arrives correctly typed, plausibly formatted, and wrong. Any agent scraper feeding a decision that matters needs validation downstream, which we come back to in the guardrails section.

The three ways to build an agent scraper

Most teams should start with the first, graduate to the second when they need custom control flow, and reach for the third only when the data genuinely sits behind an interaction a plain fetch cannot reach. The third is also the slowest and most expensive by a wide margin — a browser step costs seconds where a fetch costs milliseconds.

The middle option is where the Model Context Protocol earns its keep. Because MCP servers expose typed tool schemas the agent reads at runtime, your loop does not need a hand-written wrapper per endpoint — the agent discovers that stealth_mode exists and what it takes. We unpack that architecture in MCP vs REST for scraping, and compare the field in the best web scraping tools for AI agents.

Build one in a single call

The fastest agent scraper is one you do not write. CrawlForge ships agent as one of its 27 MCP tools: it plans its own searches, fetches and filters pages, and synthesises an answer — with no URLs required and the hard stops above already enforced.

Connected to Claude Code, Claude Desktop, or Cursor, the natural-language version is the whole interface:

Find the current pricing tiers for Vercel, Netlify, and Railway,
and tell me what each tier includes.

Under the hood that resolves to a single tool call:

{
  "name": "agent",
  "arguments": {
    "prompt": "Find the current pricing tiers for Vercel, Netlify and Railway, and what each tier includes.",
    "maxUrls": 12,
    "maxSteps": 6
  }
}

Prose is fine for a human reading the answer. For a pipeline, pass a schema and the agent returns validated JSON instead — the difference between a paragraph you have to parse and a record you can insert:

{
  "name": "agent",
  "arguments": {
    "prompt": "Extract every pricing tier for the three vendors below.",
    "urls": [
      "https://vercel.com/pricing",
      "https://www.netlify.com/pricing/",
      "https://railway.com/pricing"
    ],
    "maxUrls": 12,
    "schema": {
      "type": "object",
      "properties": {
        "tiers": {
          "type": "array",
          "items": {
            "type": "object",
            "properties": {
              "vendor": { "type": "string" },
              "tier": { "type": "string" },
              "monthlyUsd": { "type": "number" },
              "includes": { "type": "array", "items": { "type": "string" } }
            },
            "required": ["vendor", "tier"]
          }
        }
      }
    }
  }
}

Two details worth knowing. Seed urls narrow the search space when you already know where the data lives — the agent still decides how to read those pages, but stops guessing where to start. And model: "pro" swaps the default loop for full multi-source research; it costs more time and credits, so reach for it when breadth matters more than latency.

One agent call is 8 credits, flat, whether it reads three pages or twenty.

Build your own agent loop

When you need custom control flow — your own retry policy, your own stopping condition, results written to your own store — you write the loop and treat CrawlForge as the tool layer. The pattern that matters here is the escalation ladder: start cheap, escalate only on failure.

const BASE = 'https://www.crawlforge.dev/api/v1/tools';

type Rung = { tool: string; credits: number; body: (url: string) => object };

// Cheapest first. Each rung costs more and handles more hostile pages.
const LADDER: Rung[] = [
  { tool: 'scrape', credits: 2, body: (url) => ({ url, formats: ['markdown'] }) },
  {
    tool: 'scrape_with_actions',
    credits: 5,
    body: (url) => ({ url, actions: [{ type: 'scroll', delay: 1500 }] }),
  },
  {
    tool: 'stealth_mode',
    credits: 5,
    body: (url) => ({ url, stealth_config: { anti_detection_level: 'advanced' } }),
  },
];

async function call(tool: string, body: object) {
  const res = await fetch(`${BASE}/${tool}`, {
    method: 'POST',
    headers: {
      'X-API-Key': process.env.CRAWLFORGE_API_KEY ?? '',
      'Content-Type': 'application/json',
    },
    body: JSON.stringify(body),
  });
  return res.json();
}

async function fetchWithEscalation(url: string) {
  let spent = 0;

  for (const rung of LADDER) {
    const result = await call(rung.tool, rung.body(url));
    spent += rung.credits;

    // A 200 with 200 characters of "enable JavaScript" is still a failure.
    const markdown = result?.data?.markdown ?? '';
    if (result.success && markdown.length > 500) {
      return { markdown, spent, via: rung.tool };
    }
  }

  return { markdown: null, spent, via: null };
}

The markdown.length > 500 check is the part people leave out, and it is where most homemade agent scrapers quietly break. Anti-bot pages return HTTP 200. A challenge interstitial is a successful response containing no data, so res.ok tells you nothing — you have to inspect what came back before deciding the rung worked. (For the full picture on what escalation is up against, see bypassing anti-bot detection with stealth mode.)

With the fetch layer solved, the agent loop itself is small: give the model the goal, let it pick URLs, feed each one through fetchWithEscalation, and stop when the schema is satisfied or your step budget runs out. The escalation ladder is what makes it affordable — most pages resolve on rung one at 2 credits, and you only pay 5 for the ones that fight back.

Why do agent scrapers fail in production?

Four failure modes, in rough order of how often they bite.

Intelligence does not defeat fingerprinting. This is the misconception that costs teams the most money. Bot protection evaluates TLS fingerprints, browser API consistency, request timing, and header entropy — none of which an LLM influences. An agent driving a stock headless Chromium is detected at the same rate as any other headless Chromium, sometimes higher, because agent browsing patterns (straight to the target element, no scrolling, irregular timing) look less human than a script's. ScrapeWise, which sells managed scraping infrastructure, reports block rates of 61–74% for unassisted agents against Cloudflare- and Akamai-protected e-commerce targets in its own April 2026 testing. Treat vendor-run benchmarks as directional rather than definitive, but the mechanism is sound: you pay inference costs on jobs that die at the WAF, before extraction ever starts.

The economics invert at scale. Agent scraping is cheap for hundreds of pages and ruinous for millions, because the LLM inference cost is per-page and does not amortise. A deterministic pipeline built once and maintained occasionally beats a reasoning loop by an order of magnitude at volume. The crossover point is lower than most teams expect — see the cost table below.

Plausible wrong answers. An agent that cannot find a field is capable of inferring one from surrounding text. The output is schema-valid, correctly typed, and false. Selector-based scrapers cannot make this mistake; they simply return nothing.

Latency rules out high-frequency work. Inference adds seconds per page. Anything that needs sub-minute refresh across many URLs — live pricing surveillance, stock availability — belongs in a deterministic pipeline with an agent nowhere near the hot path.

There is a fifth consideration that is not a failure mode but a shift in the landscape: sites increasingly distinguish agent traffic from crawler traffic and treat it differently. HUMAN Security's 2026 State of AI Traffic report puts year-over-year AI agent traffic growth at 7,851%, and notes that agents interact with sites where crawlers only read them. Expect the access policies you scrape under to keep moving. Our guide to whether web scraping is legal in 2026 covers the compliance side.

What does an agent scraper cost?

Here is the honest arithmetic, using CrawlForge's flat per-call credit costs. Dollar figures assume the Hobby plan at $19 for 5,000 credits ($0.0038/credit); on Professional ($99 for 50,000) every figure drops roughly 48%.

Read the last two rows together, because they contain the whole lesson. Identical output; a tenth of the cost. Once you know which URLs you want, an agent is the wrong tool — batch extraction is 10× cheaper for the same 100 pages.

Which gives a clean rule: use the agent to find out what to fetch, use batch tools to fetch it. Discovery is where reasoning pays for itself. Volume is where it bankrupts you. A run that uses agent to identify 100 relevant URLs and then batch_scrape to pull them costs 28 credits — about eleven cents — while an agent-only version of the same job costs 40 and takes far longer.

When should you not use an agent scraper?

Skip the agent entirely when any of these hold:

• The site's structure is stable and known. A selector you write once and touch twice a year is cheaper and faster than reasoning about the same layout every run.
• You need more than a few thousand pages a day. Inference cost and latency both scale linearly and neither amortises.
• Determinism is a requirement. Financial reporting, compliance records, anything audited. "Usually the same answer" is not an acceptable property.
• The refresh interval is under a minute. Use track_changes with a deterministic diff instead.
• The output feeds an automated decision with no human review. Not never — but only with the validation described below.

The strongest production pattern is not agent-or-script. It is an agent that runs once to discover structure and generate a deterministic extraction spec, and a scheduled deterministic job that executes it, with the agent re-invoked only when the deterministic job starts failing. You pay for reasoning when the site changes, not on every row.

Five guardrails before you ship

1. Enforce budgets in the orchestrator. Steps, URLs, and wall clock, all outside the prompt. A limit the model can talk itself out of is not a limit.
2. Always pass an output schema. Structured output turns "did it work?" into a validation you can automate. Reject and retry on schema failure rather than storing prose.
3. Validate against a known field. Include one value in the schema you can verify independently — a product SKU, a page title, a currency symbol. If that field is wrong, discard the whole record; the agent was reading something other than what you think.
4. Log every URL the agent fetched. When an answer is wrong, the fetch trace is the only way to tell a bad page from bad reasoning. Without it you are debugging a black box.
5. Cache aggressively. Agent runs repeat work across invocations. Caching by URL with a short TTL routinely halves credit spend on iterative research.

Try it yourself

You do not need a framework, a browser farm, or a proxy contract to run your first agent scraper. CrawlForge exposes agent alongside 26 other tools through one MCP connection, with hard safety stops enforced by the runtime and flat per-call credits so you can price a run before you launch it.

Start free with 1,000 credits — enough for 125 agent runs — and connect it to Claude, Cursor, or your own loop in about two minutes.