Basic Tool1 credit

extract_links

Discover and extract all links from a webpage. Perfect for sitemap generation, link analysis, and crawling workflows.

Use Cases

Sitemap Generation

Build comprehensive sitemaps by extracting all internal links

Link Discovery for Crawling

Find all URLs to visit in a web crawling workflow

Internal Link Analysis

Analyze internal linking structure for SEO optimization

Broken Link Detection

Find and validate all links on a page

Endpoint

POST/api/v1/tools/extract_links

Auth Required

1 req/s on Free plan

1 credit

Parameters

Name	Type	Required	Default	Description
url	string	Optional	-	Page to fetch and extract links from. Either `url` or `html` is required. Example: https://example.com
html	string	Optional	-	Raw HTML to parse instead of fetching. Pair it with `base_url` so relative hrefs resolve. Example: <a href='/about'>About</a>
base_url	string	Optional	-	Base for resolving relative links; overrides the page's own `<base href>`. When fetching, the page's `<base href>` is used, else `url`; supply it when passing `html`. Example: https://example.com
include_internal	boolean	Optional	true	Include links pointing at the same host. Example: true
include_external	boolean	Optional	true	Include links pointing at other hosts. Set false to keep only internal links. Example: true
filter_external	boolean	Optional	false	The MCP tool's name for outbound-only links: `true` is the same as `include_internal: false`, and overrides it. An alias, accepted so one request body works on both surfaces. Links typed `other`, `anchor` or `relative` are not removed by it. Example: true
filter_domains	array	Optional	-	Keep only links whose host matches one of these domains. Example: ["docs.example.com"]
include_anchors	boolean	Optional	false	Include pure fragment links such as `#section`. Example: false
deduplicate	boolean	Optional	true	Collapse repeated URLs to a single entry, compared without fragment or trailing slash (`/a`, `/a/` and `/a#x` are one link); the first occurrence is kept. Example: true
include_metadata	boolean	Optional	true	Include each link's `title` attribute and the `domains` count map. Every link record carries `href`, `text`, `type`, `domain`, `rel` and `original_href` either way — the shape the MCP tool returns. Example: true
respect_robots	boolean	Optional	true	Respect the target site's robots.txt. Left at `true`, a path disallowed for `CrawlForge` is refused with 403 before anything is fetched and no credits are charged. Set it to `false` only for a target you have your own agreement with — the response then carries a `warnings` entry and the override is recorded against your API key. Example: true
escalate	boolean	Optional	false	Opt in to one automatic retry through the stealth browser when the plain fetch of `url` meets a wall instead of the page — a Cloudflare, Amazon, DataDome, PerimeterX, Akamai or Vercel challenge page, a bare 403, 429 or 444, or an empty shell or error placeholder served as 2xx. The plain fetch always runs first; a 404, a 5xx, a binary such as a PDF, or a host whose `Retry-After` window is still open never escalates, and `html` input is never fetched. The links are read from the rendered HTML, so the response has the same shape, plus `escalated` (whenever this is set) and `stealth: { engine, vendor_detected }` when the render ran. Escalation reuses the same stealth path as [stealth_mode](/docs/api-reference/tools/stealth-mode) and the same robots.txt gate, and adds no new handling of a bot defence; a wall the render is refused at too still comes back as a blocked page carrying `escalated: true`, charged nothing. Adds 5 credits, charged only when the render actually runs. Example: true
escalate_engine	string	Optional	playwright	Browser engine for the escalation stage: "playwright" (default) or "camoufox" (Firefox-based, stronger anti-detection; available only where installed on the backend). Ignored unless `escalate` is `true` and the plain fetch was walled. Example: camoufox
max_inline_chars	number	Optional	40000	Largest result to return inline, in characters of its JSON (1,000-10,000,000). Over it the response carries `preview` (the first `max_inline_chars` characters of the pretty-printed JSON), `result_handle`, `total_chars`, `truncated: true` and `expires_at`, and [read_result](/docs/api-reference/tools/read-result) reads the rest for 1 credit per call. Stored results are kept for 1 hour. Example: 40000

Fetching a URL

The page is fetched once with a 15-second timeout. A 429 or 503 whose Retry-After is 10 seconds or less is waited out and fetched once more; a longer or missing Retry-After is not retried.
HTML, XHTML, XML and every text/* type are read, and a response with no Content-Type is read as HTML. A PDF, an image or another binary returns 415 UNSUPPORTED_CONTENT_TYPE with next_step: "process_document" and charges nothing: read it with process_document.
A JSON body (application/json or *+json) returns no links, with the warning the target returned application/json; it has no HTML links.
A page that renders client-side — under 200 characters of visible text and either an empty <div id="root">, "app" or "__next" or text saying the page needs JavaScript ("Enable JavaScript…") — succeeds with rendered: false in data and the warning page renders client-side; use scrape in warnings when no stealth render ran, so an empty shell is not mistaken for a short page. html input is checked the same way.
A call costs 1 credit. With escalate: true the projection is the ceiling at 6 credits; the escalation stage's 5 credits are charged only when the plain fetch was walled and the stage actually ran. Every failure is charged nothing.

Request Examples

cURL - Extract All Links

terminalBash

curl -X POST https://crawlforge.dev/api/v1/tools/extract_links \
  -H "X-API-Key: cf_test_YOUR_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://example.com",
    "include_internal": true,
    "include_external": false,
    "deduplicate": true
  }'

TypeScript - Internal Links Only

extractLinks.tsTypescript

// npm install crawlforge-sdk
import { CrawlForge } from 'crawlforge-sdk';

const client = new CrawlForge({ apiKey: process.env.CRAWLFORGE_API_KEY });

const result = await client.extractLinks({
  url: 'https://example.com',
  // Internal-only: keep include_internal, switch include_external off.
  include_internal: true,
  include_external: false,
  deduplicate: true,
});

// result.data is untyped in crawlforge-sdk 0.1 — its shape is the Response Example below.
const { base_url, extracted_from, links } = result.data as {
  base_url: string; extracted_from: string; links: unknown[];
};
console.log('Base URL:', base_url);
console.log('Extracted from:', extracted_from);

for (const link of links) console.log(link);

// Narrow to specific hosts instead of a blanket internal/external split.
const docsOnly = await client.extractLinks({
  url: 'https://example.com',
  filter_domains: ['docs.example.com'],
});

Response Example

200 OK210ms

{
  "success": true,
  "data": {
    "links": [
      {
        "href": "https://example.com/about",
        "text": "About",
        "type": "internal",
        "domain": "example.com",
        "rel": null,
        "original_href": "/about",
        "title": "About us"
      },
      {
        "href": "https://example.com/products",
        "text": "Products",
        "type": "internal",
        "domain": "example.com",
        "rel": null,
        "original_href": "/products"
      },
      {
        "href": "https://other-site.com/",
        "text": "External Resource",
        "type": "external",
        "domain": "other-site.com",
        "rel": "nofollow",
        "original_href": "https://other-site.com"
      },
      {
        "href": "mailto:hello@example.com",
        "text": "Email us",
        "type": "other",
        "domain": null,
        "rel": null,
        "original_href": "mailto:hello@example.com"
      }
    ],
    "statistics": {
      "total_links": 4,
      "internal_links": 2,
      "external_links": 1,
      "anchor_links": 0,
      "relative_links": 0,
      "other_links": 1,
      "unique_domains": 2
    },
    "domains": {
      "example.com": 2,
      "other-site.com": 1
    },
    "base_url": "https://example.com",
    "extracted_from": "https://example.com/"
  },
  "credits_used": 1,
  "credits_remaining": 999,
  "processing_time": 210
}

Field Descriptions

data.linksEvery link as { href, text, type, domain, rel, original_href } (+ title when include_metadata is on) — the record shape the MCP tool returns

data.links[].typeinternal (same host as the page), external, anchor (#fragment, with include_anchors), other (mailto:, tel:, javascript: and other non-web schemes; href is null except for mailto: and tel:) or relative (html input with no base_url)

data.links[].original_hrefThe href as written on the page; href is the resolved absolute URL

data.statisticsCounts by type: total_links, internal_links, external_links, anchor_links, relative_links, other_links, unique_domains

data.domainsLinks per domain (only with include_metadata)

Error Handling

Blocked by a Bot Wall (403, 444 or 502)

The target answered with a bot-defence challenge page instead of the content, whatever its HTTP status. The response is success: false with error.code BLOCKED, a blocked object naming the vendor (for example cloudflare or akamai) and the evidence it was recognised by, and a next_step: extract_links with escalate:true, or stealth_mode operation:"scrape". The status is the target's own when it is 403, 404, 410, 429, 444 or 451, and 502 otherwise — a wall served as 200 included. No credits are charged. With escalate: true the wall is rendered once in the stealth browser instead; if the render is refused too, the same error carries escalated: true and no next_step.

Target Answered with an Error (502 Bad Gateway)

The target answered with an HTTP error and no recognised bot wall. error.code is UPSTREAM_HTTP and the message is Target answered HTTP <n>. A 403, 404, 410, 429, 444 or 451 is passed through as the response status; every other error — a 5xx, or the target's own 400 or 401 — is a 502, so it cannot be mistaken for a problem with your request or API key. A 403, 429 or 444 also carries next_step. No credits are charged.

Blocked by robots.txt (403 Forbidden)

The target site's robots.txt disallows this path for CrawlForge. Set respect_robots: false to override if you have your own agreement with the target — the override is recorded against your API key. The override does not reach a host on CrawlForge's permanent opt-out list, which is refused whatever respect_robots is set to.

Unsupported Content Type (415 Unsupported Media Type)

The target served a PDF, an image or another binary. error.code is UNSUPPORTED_CONTENT_TYPE, the message is Unsupported content type <type>: extract_links reads HTML, and next_step is process_document. The body is not read, nothing is escalated and no credits are charged.

Escalation Backend Not Configured (503 TOOL_NOT_AVAILABLE)

escalate: true runs its stealth stage on the CrawlForge execution backend. When that backend is not configured the stage returns 503 and no credits are charged.

Related Tools

fetch_url

Fetch pages before extracting links (1 credit)

smart_crawl

Intelligent crawling with link discovery (8 credits)

Ready to extract links? Sign up for free and get 1,000 credits.