scrape_with_actions

Execute browser action chains including click, scroll, type, and form auto-fill with screenshot capture. Perfect for login flows, infinite scroll, modal dialogs, and complex JavaScript-heavy sites.

Use Cases

Automate login forms and access authenticated content behind login walls

Infinite Scroll

Scrape content from infinite scroll pages like social media feeds and product listings

Modal Dialogs

Interact with popups, modals, and dynamic overlays

JavaScript-Heavy Sites

Handle SPAs and sites with dynamic content loading via AJAX

Multi-Step Forms

Navigate through multi-step wizards and complex form submissions

Visual Testing

Capture screenshots at each step for debugging and visual regression testing

Endpoint

POST/api/v1/tools/scrape_with_actions

Auth Required

1 req/s on Free plan

5 credits

Parameters

Name	Type	Required	Default	Description
url	string	Required	-	The URL to load before running the action chain Example: https://app.example.com/dashboard
actions	array	Required	-	Ordered list of 1-20 browser actions to run before scraping. Each item is an object with a `type` of `wait`, `click`, `type`, `press`, `scroll`, `screenshot`, `executeJavaScript`, `select`, `hover`, `navigate`, or `snapshot`, plus type-specific fields: `selector` (CSS target, or an `@e1` ref from an earlier `snapshot`), `text` (for `type`), `key` (for `press`), `script` (for `executeJavaScript`), `duration`/`condition` (for `wait`), `button`/`clickCount`/`delay` (for `click`), `direction`/`distance`/`toElement` (for `scroll`), `fullPage`/`quality`/`format` (for `screenshot`), `value` or `values` (for `select`), `url`/`waitUntil` (for `navigate`), and `interactiveOnly`/`maxNodes` (for `snapshot`). `select` takes either a single `value` or a `values` array, and a plain string matches an option by its value or its visible label; `hover` needs `selector`, optionally with `force` and `position`. A `snapshot` lists the page's interactive elements, including those inside open shadow roots and iframes, with a stable ref on each — `@e1`, `@e2`, … in document order — and any later action in the same chain can target one of those refs instead of a CSS selector; navigation invalidates them, so snapshot again once the page has changed. Optional on any action: `timeout` (per action, default 10000ms — distinct from `browserOptions.timeout`, which budgets the whole chain), `description`, `continueOnError` (default false), `retries` (0-5, default 1), and `captureAfter` (default false). Example: [{"type": "click", "selector": "#login"}, {"type": "type", "selector": "#email", "text": "user@example.com"}, {"type": "wait", "duration": 1000}, {"type": "screenshot"}]
formats	array	Optional	["json"]	Output formats to return: `markdown`, `html`, `json` or `text`. For an image, add a `{"type": "screenshot"}` action where you want it taken. Example: ["markdown"]
captureIntermediateStates	boolean	Optional	false	Capture page state after each action: `intermediateStates[]` gets one entry per action (`capturePoint` is the action's 1-based position) with the `url`, `title` and the requested formats at that point Example: true
formAutoFill	object	Optional	-	Fill and submit a form in one step. Shape: `{ fields: [{ selector, value, type: text\|select\|checkbox\|radio\|file, waitAfter }], submitSelector, waitAfterSubmit }` — `waitAfterSubmit` defaults to 2000ms. Each field's `type` picks how it is filled: `text` (the default) types the value, `select` picks the option by value or label, and `checkbox` and `radio` check the input — a selector matching a whole group, such as `input[name=size]`, checks the member whose `value` attribute equals `value`, and a checkbox with value `"false"` is unchecked. `file` is not supported and fails that action. Example: {"fields": [{"selector": "#email", "value": "user@example.com", "type": "text"}], "submitSelector": "#login"}
browserOptions	object	Optional	-	Browser configuration: `headless` (default true), `userAgent`, `viewportWidth` (default 1280, range 800-1920), `viewportHeight` (default 720, range 600-1080), `timeout` (default 30000ms, range 10000-120000), and `stealth` (default false). Keep `timeout` within the ~25s REST window. Set `stealth` to true to run the action chain in the stealth Chromium engine at its `medium` profile instead of the standard browser pool — the boolean is the only knob, so the level, fingerprint randomization and engine choice are not settable from here. It is slower to start, and it renders JavaScript rather than solving challenges. `consent` (default `"off"`) handles cookie and consent banners: `"reject"` runs DuckDuckGo autoconsent's opt-out on the banner once after the initial navigation and after each `navigate` action, capped at 2 s, and `"accept"` opts in instead. When it is not `"off"` the result carries `consent: { cmp, action, ms }` for the initial navigation, and each `navigate` action's result carries its own `result.consent`. `cmp` is the consent-management platform that matched, or `null` when none did; `action` is `optOut`, `optIn`, `hide` (a cosmetic rule hid the banner), `timeout` (a banner was detected but not answered within 2 s), `error`, or `none` (nothing matched); `ms` is how long it took. A chain never fails because no platform matched. The default is off so a chain that clicks the banner itself keeps working. `proxyRotation` routes the stealth browser through proxies you supply: `{ enabled, proxies, rotationInterval }`, where `enabled` defaults to false and `rotationInterval` (default 300000) is the minimum time in ms on one proxy before the list advances. It requires `stealth: true` and is refused otherwise. Proxies run per context on the Chromium engine only; a per-call `proxyRotation` is refused on Camoufox, which applies its proxy at browser launch and shares that browser across calls. Each entry in `proxies` is a proxy URL — `"http://user:pass@host:port"` (percent-encode a password containing `@`, `:` or `/`), or a bare `"host:port"` for an unauthenticated HTTP proxy; `http`, `https`, `socks4` and `socks5` are accepted. Proxies on loopback, private, link-local or cloud-metadata addresses are refused, and credentials are never echoed back — `metadata.browserOptions` in the result shows them redacted. CrawlForge supplies no proxies: the hosted stealth browser runs from a datacenter address, and a residential proxy is what gets past an IP-reputation block that no fingerprint fixes. Example: {"viewportWidth": 1440, "viewportHeight": 900, "timeout": 20000}
extractionOptions	object	Optional	-	Content extraction options: `selectors` (a CSS key→value map of data to pull out), `includeMetadata` (default true), `includeLinks` (default true), and `includeImages` (default true). Example: {"selectors": {"title": "h1", "price": ".price"}}
continueOnActionError	boolean	Optional	false	Keep running the remaining actions when one fails instead of aborting the chain Example: false
maxRetries	number	Optional	0	Whole-chain retries on failure (0-3). A retry re-navigates to the starting URL and replays every action, so each one costs as much as the original run; every attempt is reported under `attempts[]` Example: 0
screenshotOnError	boolean	Optional	true	Capture screenshot when an error occurs Example: true
respect_robots	boolean	Optional	true	Respect the target site's robots.txt. Omit it and the compliant default (`true`) applies: a URL disallowed for `CrawlForge` is refused before the browser opens and this endpoint charges nothing for it, and every `navigate` action is checked the same way. The page is checked again wherever it ends up — after a redirect, a click or a form submission — and landing on a disallowed path fails the call with nothing from that page returned. The site's `Crawl-delay` is honoured whatever this says: a `navigate`, and a `click`, `press`, `select` or `check` that could send the page somewhere, waits that long after the last request to the page's host before it runs. Set it to `false` only for a target you have your own agreement with — the override is recorded against your API key. Example: true
max_inline_chars	number	Optional	40000	Largest result to return inline, in characters of its JSON (1,000-10,000,000). Over it the response carries `preview` (the first `max_inline_chars` characters of the result's main text field), `result_handle`, `total_chars`, `truncated: true` and `expires_at`, and [read_result](/docs/api-reference/tools/read-result) reads the rest for 1 credit per call. Stored results are kept for 1 hour. Example: 40000
redact_pii	boolean \| object	Optional	false	Remove personal data from the text this call returns, before the result is stored or sent back. `true` is shorthand for `{ mode: "fast" }` — all four regex classes, tagged. Whenever you ask for redaction the response carries `redaction: { entities, count, mode }` inside `data`, even when nothing matched (`count: 0`), so "found nothing" is never mistaken for "the parameter was ignored"; a class with no hits is left out rather than reported as `0`. Redaction runs before the result is stored, so an oversized result read back later with [read_result](/docs/api-reference/tools/read-result) is already redacted. Two deliberate limits: addresses (`url`, `link`, `href`, `canonical_url`) are never redacted, and counters derived from the text (`content_length`, `word_count`, `character_count`) describe it as extracted, before redaction. Example: true

Available Action Types

wait

Wait for specified milliseconds

action.jsonJson

{"type": "wait", "duration": 2000}

click

Click an element by CSS selector

action.jsonJson

{"type": "click", "selector": "#button"}

type

Type text into an input field

action.jsonJson

{"type": "type", "selector": "#search", "text": "query"}

press

Press a keyboard key

action.jsonJson

{"type": "press", "key": "Enter"}

scroll

Scroll to element or position

action.jsonJson

{"type": "scroll", "toElement": "#content"}

screenshot

Capture screenshot

action.jsonJson

{"type": "screenshot"}

executeJavaScript

Execute custom JavaScript. Disabled unless the self-hosted server sets ALLOW_JAVASCRIPT_EXECUTION=true; refused on the hosted API.

action.jsonJson

{"type": "executeJavaScript", "script": "window.scrollTo(0, 0)"}

select

Choose options in a <select>: selector names the dropdown, then either value (one option) or values (several). A plain string matches an option by its value or its visible label.

hover

Move the pointer over the element named by selector — for menus and tooltips that only appear on hover. force and position behave as they do for click.

navigate

Load another url in the same browser session, so cookies, localStorage and login state from earlier actions carry over. Optional waitUntil is load, domcontentloaded (the default), networkidle, or commit. The new URL passes the same robots.txt and SSRF checks as the initial one, so it is not a way around the gate. Each navigation is checked for a bot wall: the action's result carries httpStatus and, when a wall was named, blocked: { vendor, evidence }. The action itself still succeeds — the wall is reported, not failed.

snapshot

List what the page offers the pointer and keyboard as an indented accessibility tree, every interactive element, including those inside open shadow roots and iframes, carrying a stable ref in document order — [document] "Example Domain", then @e1 [textbox] "Search..." and @e2 [button] "Search". An element inside an iframe or a shadow root gets an ordinary @eN ref like any other. Set interactiveOnly (default true) to false to list headings and landmarks as well, which never get a ref; maxNodes (default 200, max 1000) caps how many nodes the tree lists — under the default that is one ref each — and the result reports truncated: true when that cap stopped the walk. The action's entry in actionResults carries the tree as result.tree, alongside snapshotId, url, title, refCount, nodeCount, truncated, interactiveOnly and source — aria when the tree came from the browser's native accessibility snapshot, walk when the fallback walker produced it.

action.jsonJson

{"type": "snapshot", "interactiveOnly": true, "maxNodes": 200}

Targeting elements by ref

A snapshot puts a stable ref on each interactive element it finds, so a chain reads snapshot, act on the refs, snapshot again: [{"type": "snapshot"}, {"type": "type", "selector": "@e1", "text": "crawlforge"}, {"type": "click", "selector": "@e2"}]. A ref is valid only on the document its snapshot was taken from: any navigation — a navigate action, or a click that loads a new page — invalidates every ref, and acting on a stale one fails with an error telling you to take a new snapshot. Elements inside open shadow roots and iframes get ordinary @eN refs and are targeted the same way.

Bot walls on every navigation

Every navigation in a chain — the initial page load and each navigate action — is checked for a bot wall, an HTTP error page or a challenge interstitial, including an HTTP 202 interstitial such as an AWS WAF or Amazon challenge. The result lists them in order under navigations, index 0 being the initial load, each as { url, finalUrl, httpStatus, blocked }: httpStatus is null when it is unknown, and blocked: { vendor, evidence } appears only when a wall was named (cloudflare, akamai, datadome, perimeterx, amazon, vercel, …). A click or a form submit that loads another page is a navigation too: it gets its own entry, with trigger naming the action type (click, press, …), and the top-level finalUrl is where the chain ended, while url stays the one you requested. A wall partway through does not stop the chain, so a later action can navigate on. Whether the call succeeds is decided by the document the chain ends on: a chain that ends on a wall is never reported as success. Over this API it fails with 502 BLOCKED, carrying blocked and navigations, and nothing is charged; an HTTP error page with no named wall fails with 502 TOOL_ERROR. The tool reports walls; it does not solve them — when the block is on the network address, pass your own proxy in browserOptions.proxyRotation (with stealth: true).

Request Examples

terminalBash

# 1. Snapshot the page to learn its interactive elements and their refs.
curl -X POST https://crawlforge.dev/api/v1/tools/scrape_with_actions \
  -H "X-API-Key: cf_test_YOUR_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://app.example.com/login",
    "actions": [{"type": "snapshot"}],
    "formats": ["json"]
  }'

# actionResults[0].result.tree comes back as:
#   [document] "Sign in"
#     @e1 [textbox] "Email"
#     @e2 [textbox] "Password"
#     @e3 [button] "Sign in"

# 2. Target those refs instead of guessing selectors. The chain snapshots
#    again first, which stamps the refs onto this fresh load.
curl -X POST https://crawlforge.dev/api/v1/tools/scrape_with_actions \
  -H "X-API-Key: cf_test_YOUR_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://app.example.com/login",
    "actions": [
      {"type": "snapshot"},
      {"type": "type", "selector": "@e1", "text": "user@example.com"},
      {"type": "type", "selector": "@e2", "text": "secret123"},
      {"type": "click", "selector": "@e3"},
      {"type": "wait", "duration": 1000},
      {"type": "screenshot"}
    ],
    "formats": ["markdown"]
  }'

Response Example

200 OK9000ms

{
  "success": true,
  "data": {
    "success": true,
    "url": "https://app.example.com/login",
    "finalUrl": "https://app.example.com/dashboard",
    "httpStatus": 200,
    "engine": "chromium",
    "navigations": [
      {
        "url": "https://app.example.com/login",
        "finalUrl": "https://app.example.com/login",
        "httpStatus": 200
      },
      {
        "url": "https://app.example.com/dashboard",
        "finalUrl": "https://app.example.com/dashboard",
        "httpStatus": 200,
        "trigger": "click"
      }
    ],
    "actionResults": [
      {
        "id": "action_1759480001102_k2m8x1q0z",
        "type": "click",
        "success": true,
        "executionTime": 640
      },
      {
        "id": "action_1759480001790_p7c4n9d2a",
        "type": "wait",
        "success": true,
        "executionTime": 1001
      },
      {
        "id": "action_1759480002805_h5t1v8r3e",
        "type": "screenshot",
        "success": true,
        "executionTime": 212,
        "result": {
          "format": "png",
          "fullPage": false,
          "resourceUri": "crawlforge://screenshot/action_1759480002805_h5t1v8r3e"
        }
      }
    ],
    "totalActions": 3,
    "successfulActions": 3,
    "failedActions": 0,
    "actionsExecuted": 3,
    "content": {
      "markdown": "# Dashboard\n\nWelcome back — you are now signed in..."
    },
    "screenshots": [
      {
        "actionId": "action_1759480002805_h5t1v8r3e",
        "format": "png",
        "fullPage": false,
        "timestamp": 1759480002805,
        "resourceUri": "crawlforge://screenshot/action_1759480002805_h5t1v8r3e"
      }
    ]
  },
  "credits_used": 5,
  "credits_remaining": 995,
  "processing_time": 9000
}

Field Descriptions

data.urlThe URL you requested. It stays the same however far the chain moved

data.finalUrlWhere the chain ended, after any redirect, click or form submit

data.httpStatusStatus of the last main-frame document the chain loaded

data.navigationsEvery navigation in order, index 0 being the initial load. An entry caused by a click or a form submit carries `trigger`, the type of the action that moved the page

data.actionResultsOne entry per action, with its `type`, `success`, timing and the action's own `result`

data.actionsExecutedNumber of actions that ran; `successfulActions` and `failedActions` split it

data.content.markdownPage content in each requested format after all actions completed

data.screenshotsAlways present: one entry per `screenshot` action, plus one taken on error, and empty when there were none. Each is a `crawlforge://screenshot/{actionId}` resource URI rather than inline base64, and this REST endpoint passes that URI through without resolving it — so for now the bytes are readable only from the CrawlForge MCP server

credits_usedCredits deducted for this request (5 per scrape)

processing_timeTotal time in ms including all actions and waits

Error Handling

Browser runtime not configured (503 TOOL_NOT_AVAILABLE)

This tool needs a browser automation runtime. When the hosted execution backend is not configured the call returns 503 immediately and no credits are charged.

Execution backend timed out (504 MCP_UPSTREAM_TIMEOUT)

Navigation plus the action chain exceeded the execution backend's time budget. Shorten waits, reduce the number of actions, or retry. Failures are not charged.

Blocked by robots.txt (502 TOOL_ERROR)

The target's robots.txt disallows this URL for CrawlForge, so no browser was launched and nothing was charged. Every navigate action is checked the same way, not just the initial URL. So is wherever the page ends up: when a redirect, a click or a form submission lands on a disallowed path, the message names both URLs (“The page moved from … to …”), the chain stops there and nothing is charged. Login and search pages are commonly disallowed, so a flow that passes through one needs the override. Set respect_robots: false to override if you have your own agreement with the target — the override is recorded against your API key.

Ended on a bot wall (502 BLOCKED)

The document the chain ended on is a bot wall or a challenge interstitial, so the call fails rather than returning the wall as the page, and nothing is charged. The body carries blocked: { vendor, evidence } naming the wall, the chain's navigations and, when known, the final httpStatus, so you can see which navigation hit it. An HTTP error page or a placeholder the tool cannot attribute to a vendor fails with 502 TOOL_ERROR instead, also uncharged. A wall on an earlier navigation does not fail the call; it is reported under navigations instead. The tool reports walls and does not solve them; when the block is on the network address, route the chain through your own proxy with browserOptions.proxyRotation.

Invalid Action (400 Bad Request)

One or more actions have invalid parameters. Check action type and required fields.

Insufficient Credits (402 Payment Required)

Your account doesn't have enough credits (need 5). Purchase more credits or upgrade your plan.

Rate Limit Exceeded (429 Too Many Requests)

You've exceeded your plan's rate limit. Wait a moment or upgrade your plan for higher limits.

Pro Tip:

Use wait actions between interactions to ensure JavaScript has time to execute. For debugging, add a screenshot action after each step to see what the browser sees.

Credit Cost

5 credits

5 credits per request

Each successful scrape_with_actions request costs 5 credits, regardless of the number of actions.

Free Plan: 1,000 one-time trial credits = 200 action chains

Hobby Plan: 5,000 credits/month = 1,000 action chains ($19/mo)

Professional Plan: 100,000 credits/month = 20,000 action chains ($99/mo)

Business Plan: 500,000 credits/month = 100,000 action chains ($399/mo)

Related Tools

form_submit

Simplified form submission without action chains (3 credits)

screenshot

Capture screenshots without actions (2 credits)

Ready to try scrape_with_actions? Sign up for free and get 1,000 credits to start building.