Crawler Operating Rules

These bound what we build, not only what we ask customers to do. A feature that breaks one of them does not ship, however much demand there is for it.

No credential-based access to a platform whose terms prohibit it. We do not accept your credentials for a target site, we hold no accounts on third-party platforms, and no tool is built to sign in on your behalf.

Why: authentication circumvention is the line where scraping stops being a contract question and becomes a Computer Fraud and Abuse Act question. A breach of terms is a dispute between two parties; access without authorisation is a different category of risk, and not one we take on for a customer.

How it is enforced: no tool takes credentials for a target site as a parameter, and CrawlForge stores none. Browser contexts opened by stealth_mode are ephemeral — the execution backend recycles them — so there is no durable signed-in session to reuse. scrape_with_actions runs the action list you supply: using it to sign in to a platform whose terms prohibit automated access breaches the Acceptable Use Policy and is grounds for suspension.

We do not integrate CAPTCHA-solving services, do not forge or replay bot-challenge tokens, and do not build paywall bypasses.

Why: these are technical access controls. Defeating one is not a compatibility fix — it is a decision to override the site owner's stated answer. That is not a feature we ship, and shipping it would put every other rule here in doubt.

How it is enforced: no CAPTCHA-solving provider is a dependency of the API or of the MCP server. stealth_mode exists to make a real browser render like a real browser — realistic headers, viewport and timing — on sites that break under a plain HTTP fetch. It does not solve challenges: a page that returns a challenge returns that challenge to you.

Where a target publishes an API, the tool that covers that target uses it.

Why: an API is cheaper per record, exact instead of best-effort, stable against markup changes, and permitted by construction — the publisher decided what it exposes. Scraping a source that offers an API is the worse engineering choice before it is an ethical one.

How it is enforced: reddit_search reads Reddit content through public JSON data APIs rather than scraping reddit.com, which blocks automated access. search_web and serp_rank call search APIs instead of fetching result pages. New coverage is specified API-first: a scraping path is written only where no API covers the data.

Every outbound request carries one honest User-Agent: real product name, version, and a contact URL.

Why: a site that wants to block us must be able to. An identity that rotates or disguises itself removes the site owner's ability to make a decision about us, and it makes G5 and G7 unverifiable from the outside — a blocklist you cannot recognise is not a blocklist.

How it is enforced: every fetching tool routes through one shared fetch helper, which sends the identity below — product name, version and contact URL. If you have your own agreement with a target, a per-request userAgent override is available: that is your declaration to that target, made explicitly, not a product default.

Taking the header at its word is optional: Verifying CrawlForge shows a site owner how to check a request against our published signing key, how to block us in one robots.txt rule, and where to write for a permanent opt-out.

Every fetching tool checks robots.txt before it fetches. The override is per request, returns a warning in the response, and is recorded against the API key that asked for it.

Why: the default has to be respect, because a default is what most traffic runs on. Making the override explicit and logged turns it into your documented decision about a target you know — not a silent product-wide choice we made for you.

How it is enforced: each origin's rules are fetched once and cached, then matched against our product token. A missing, unreachable, oversized or non-200 robots.txt is treated as no restriction — a site that cannot serve the file has not expressed one. A disallowed URL is skipped, and where the entry point itself is disallowed the request fails before anything is fetched, so a blocked crawl costs no credits.

Concurrency and delays default to conservative values, and a host's own pacing signals — Crawl-delay in its robots.txt, Retry-After on a 429 or 503 — are honoured on both surfaces.

Why: load we impose is load someone else pays for — in bandwidth, in origin CPU, and in the pager of whoever is on call. A crawler that ignores a Retry-After it was handed is not fast, it is just rude at the same throughput.

`Retry-After`: it is recorded from every response, and further requests to that host are refused while the window is open — error HOST_BACKOFF, HTTP 429, naming the host and the seconds left. You are not charged for it, and respect_robots: false does not reach it: that flag is a statement about robots.txt, whereas a 429 is the host answering us directly and not something a caller gets to overrule on its behalf. Both forms the spec allows are accepted, and the wait is capped at five minutes so one mistaken header cannot wedge a host.

This is the one place the two surfaces behave differently, and the reason is the runtime rather than the policy. The MCP server sleeps the interval out and carries on. The REST API refuses straight away: inside a serverless function with a 30-second ceiling, sleeping on a two-minute wait would burn the request and hand you nothing — having waited, been killed, and still not fetched the page — so it gives you the host's answer immediately instead.

`Crawl-delay`: the five tools that fetch more than one page — crawl_deep, map_site, generate_llms_txt, batch_scrape and deep_research — space their requests to a host by whatever its robots.txt asks for. Spacing is per host, so a slow site never holds up an unrelated one. In batch_scrape a host that declares a delay is effectively serialised while every other host keeps full concurrency — which is what the directive asks for, and worth knowing if you send fifty URLs from one delayed domain. Where a delay will not fit inside the route's time budget the URL is skipped rather than hung on: the result is marked skipped with the reason, a summary line lands in notes, and you are not charged for it. Single-fetch tools fetch once and have nothing to space.

crawl_deep also takes its own crawl_delay parameter. That is a pause you ask for, which is a different thing from the one the site asks for; where both apply they resolve to one wait and the longer wins. Your plan rate limit (1–10 requests per second) is a separate ceiling on top of all of it — see the FAQ.

A site owner who asks not to be fetched is added to a platform-level blocklist. No customer flag overrides it.

Why: a blocklist any customer can switch off is not a blocklist. If the opt-out lives in per-request options, honouring it depends on every customer choosing to — which means it does not hold. It has to sit below the request layer.

How it is enforced: the blocklist is checked before a request is dispatched and applies across all tools, all API keys and all plans. There is no parameter that disables it, and a robots.txt override does not reach it. To request an opt-out or file a takedown, email support@crawlforge.dev with the domain and the pages concerned.

Tools return what the source publishes publicly. We do not build features that assemble scattered public facts into a profile of an individual.

Why: GDPR and CCPA apply to personal data regardless of where the HTML came from, and "it was public" is not a lawful basis. Under GDPR, aggregation is itself a processing purpose that has to be justified: publishing a fact about yourself in one place is not consent to being compiled.

How it is enforced: no tool cross-references identities across sources, and no person-centred index is retained. Where an extraction schema collects personal data, it is your schema and your controllership — our retention is the usage log, which records the request and not the extracted content. Establishing a lawful basis for personal data you extract is your obligation under the Acceptable Use Policy.

These are named because they get asked for. None of them is on a roadmap.

If you run a site and do not want CrawlForge fetching it, you have two routes. A robots.txt rule is honoured by default on every tool and takes effect on the next fetch. A platform-level opt-out is permanent and cannot be overridden by any customer: email support@crawlforge.dev with the domain and the pages concerned.