Verifying CrawlForge
A request in your logs says it is CrawlForge. This page is how you check that it really is, how you block it if you would rather not be fetched, and who to email.
CrawlForge product token is stable. The version after it changes with every release and is not the same on both of our surfaces, so a rule written against a complete User-Agent string will stop matching. Match the token — and if you want proof rather than a claim, verify the signature.How CrawlForge identifies itself
Every page CrawlForge fetches is fetched with one honest User-Agent: the product token CrawlForge, a version, and a contact URL. There is no second identity, no rotation, and no browser-lookalike default — see the crawler operating rules for why that is a rule rather than a preference.
CrawlForge runs on two surfaces and both send this token: the hosted REST API, and the crawlforge-mcp-server package that customers run themselves. Each sends its own package version, so the version segment varies by surface and by release while the token does not.
The User-Agent pattern
This is the supported thing to match. Only the version segment varies.
CrawlForge/<version> (+https://crawlforge.dev)What the hosted API sends today
Rendered from the same constant the crawler sends, so this page cannot fall out of step with the header. It is shown as an example, not as a string to match: the next release changes it. The MCP server sends the same shape with its own package version.
User-Agent: CrawlForge/1.0.0 (+https://crawlforge.dev)Matching us in your logs
A prefix match on the token survives every release we ship.
# Match the product token. Anchored, and stopping at the slash, so the
# version cannot break the match on our next release.
^CrawlForge/The retired CrawlForge-Bot token
Some of our tools used to identify as CrawlForge-Bot. Nothing sends that token any more — everything sends CrawlForge.
It is still honoured as a source of disallow, though. Unifying the token would otherwise have silently un-blocked every site owner who had already written the old name into robots.txt, and that was a decision made about us that a rename has no business discarding. Both tokens are consulted and a disallow from either one wins.
If your robots.txt already names the retired token, you do not need to change anything.
# Already in your file? Leave it — it still blocks us.
User-agent: CrawlForge-Bot
Disallow: /Cryptographic verification (Web Bot Auth)
A User-Agent is a claim, and anyone can send ours. So CrawlForge signs its outbound requests with Web Bot Auth: HTTP Message Signatures (RFC 9421) over an Ed25519 key whose public half we publish. You can verify a request without contacting us and without an allowlist.
Both surfaces sign. Each signed request carries Signature-Input (the covered components and the signature parameters), Signature (the Ed25519 signature, base64), and Signature-Agent (the URL of the directory that publishes our key).
The signature covers the authority the request was sent to — plus the Signature-Agent value when it is advertised. It proves the request came from the holder of our private key. It does not cover the path, the body, or the User-Agent, so treat it as proof of *who*, not of *what was asked for*.
Our signature directory
A JSON key set, served as application/http-message-signatures-directory+json and cacheable for a day. Note the plural in signatures.
curl -sS https://www.crawlforge.dev/.well-known/http-message-signatures-directoryThe headers we send
The parameter list is serialised by the same function that signs our requests. Values in angle brackets vary per request. Resolve the `keyid` against the directory rather than pinning it — it is a thumbprint of the key and changes when we rotate.
Signature-Input: sig1=("@authority" "signature-agent");created=1787788800;keyid="<keyid — look it up in the directory>";alg="ed25519";expires=1787789100;nonce="<64 random bytes, base64>";tag="web-bot-auth"
Signature: sig1=:<base64 Ed25519 signature>:
Signature-Agent: "https://www.crawlforge.dev"Verifying a request
Rebuild the signature base from the request you received, then check it against the published key. Node's built-in crypto is enough; any RFC 9421 verifier will do the same thing.
import { createPublicKey, verify } from 'node:crypto';
// `request` is the request you received; `signatureInput` and `signature`
// are its Signature-Input and Signature header values.
// 1. Fetch the directory and index the published keys by their thumbprint.
const { keys } = await fetch(
'https://www.crawlforge.dev/.well-known/http-message-signatures-directory'
).then((r) => r.json());
// 2. Read the parameters off Signature-Input, dropping the "sig1=" label.
const params = signatureInput.replace(/^[^=]+=/, '');
const keyid = /keyid="([^"]+)"/.exec(params)?.[1];
const published = keys.find((k) => k.kid === keyid);
if (!published) throw new Error('keyid is not in the directory — not us');
// 3. Rebuild the signature base from the request you received. The covered
// components are listed in the parentheses at the start of params.
const covered = /^\(([^)]*)\)/.exec(params)[1];
const lines = [`"@authority": ${request.headers.host}`];
if (covered.includes('"signature-agent"')) {
lines.push(`"signature-agent": ${request.headers['signature-agent']}`);
}
lines.push(`"@signature-params": ${params}`);
// 4. Verify the Ed25519 signature (base64, between the colons).
const key = createPublicKey({
key: { kty: 'OKP', crv: 'Ed25519', x: published.x },
format: 'jwk',
});
const bytes = Buffer.from(
signature.replace(/^[^=]+=:/, '').replace(/:$/, ''),
'base64'
);
const ok = verify(null, Buffer.from(lines.join('\n'), 'utf8'), key, bytes);
// 5. Finally, reject a stale signature: params carries created= and expires=,
// and we sign with a five-minute window.Blocking CrawlForge
One robots.txt rule covers everything. The group name is the product token, which both surfaces match against, so this blocks every CrawlForge tool on both.
Add this to your robots.txt. It takes effect on the next fetch — we read the file per origin at the start of each crawl.
User-agent: CrawlForge
Disallow: /What we will not hide from you
robots.txt is respected by default on every fetching tool. A customer can also pass respect_robots: false on a single request — that override exists, it is documented, and it is deliberate: some customers have their own agreement with a site they are fetching, and forcing that decision underground would make it invisible rather than rare.
It is not silent. The override applies to one request, never to an account or to the product. The response carries a warning saying it was used, and the request is recorded server-side as a robots_override entry with the URL, the tool and the internal id of the API key that asked for it — never the key itself — so a disputed fetch can be traced back to the customer who chose it.
A platform-level opt-out sits below all of this and no flag reaches it. If a robots.txt rule is not enough for you, use the opt-out route instead.
Egress IP addresses
We do not publish a list of egress IP addresses today. Our hosted API and our browser backend run on third-party infrastructure, and until we can commit to a range that will still be accurate next month, publishing one would be worse than publishing nothing — an allowlist built on a stale range fails closed against real traffic and open against everybody else. When there is a list we can stand behind, it will appear in this section.
Signature verification is the stronger check regardless. An address range proves only where a packet came from, and anyone who can rent an address inside that range inherits the claim; the usual patch for that — a reverse-DNS lookup on top — proves control of a hostname, not of the request. An Ed25519 signature proves possession of the private key whose public half is in our directory, it survives us changing hosts, and it cannot be replayed onto a different authority.
If a signature is not enough for a commercial arrangement on your side, email support@crawlforge.dev and say what you need.
Opt-outs and takedowns
To be removed at the platform level — permanently, across every tool, every API key and every plan, with no customer flag able to override it — email support@crawlforge.dev with the domain and the pages concerned. The same address handles takedown requests and any question about traffic you believe came from us.