Basic Tool1 credit

extract_links

Discover and extract all links from a webpage. Perfect for sitemap generation, link analysis, and crawling workflows.

Use Cases

Sitemap Generation

Build comprehensive sitemaps by extracting all internal links

Link Discovery for Crawling

Find all URLs to visit in a web crawling workflow

Internal Link Analysis

Analyze internal linking structure for SEO optimization

Broken Link Detection

Find and validate all links on a page

Endpoint

POST/api/v1/tools/extract_links

Auth Required

1 req/s on Free plan

1 credit

Parameters

Name	Type	Required	Default	Description
url	string	Optional	-	Page to fetch and extract links from. Either `url` or `html` is required. Example: https://example.com
html	string	Optional	-	Raw HTML to parse instead of fetching. Pair it with `base_url` so relative hrefs resolve. Example: <a href='/about'>About</a>
base_url	string	Optional	-	Base for resolving relative links. Defaults to `url` when fetching; supply it when passing `html`. Example: https://example.com
include_internal	boolean	Optional	true	Include links pointing at the same host. Example: true
include_external	boolean	Optional	true	Include links pointing at other hosts. Set false to keep only internal links. Example: true
filter_domains	array	Optional	-	Keep only links whose host matches one of these domains. Example: ["docs.example.com"]
include_anchors	boolean	Optional	false	Include pure fragment links such as `#section`. Example: false
deduplicate	boolean	Optional	true	Collapse repeated URLs to a single entry. Example: true
include_metadata	boolean	Optional	true	Include each link's anchor text and attributes alongside the URL. Example: true
respect_robots	boolean	Optional	true	Respect the target site's robots.txt. Left at `true`, a path disallowed for `CrawlForge` is refused with 403 before anything is fetched and no credits are charged. Set it to `false` only for a target you have your own agreement with — the response then carries a `warnings` entry and the override is recorded against your API key. Example: true

Request Examples

cURL - Extract All Links

terminalBash

curl -X POST https://crawlforge.dev/api/v1/tools/extract_links \
  -H "X-API-Key: cf_test_YOUR_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://example.com",
    "include_internal": true,
    "include_external": false,
    "deduplicate": true
  }'

TypeScript - Internal Links Only

extractLinks.tsTypescript

const response = await fetch('https://crawlforge.dev/api/v1/tools/extract_links', {
  method: 'POST',
  headers: {
    'X-API-Key': process.env.CRAWLFORGE_API_KEY!,
    'Content-Type': 'application/json',
  },
  body: JSON.stringify({
    url: 'https://example.com',
    // Internal-only: keep include_internal, switch include_external off.
    include_internal: true,
    include_external: false,
    deduplicate: true,
  }),
});

const payload = await response.json();
if (!response.ok) {
  throw new Error(payload.error.code + ': ' + payload.error.message);
}

const result = payload.data;
console.log('Base URL:', result.base_url);
console.log('Extracted from:', result.extracted_from);

for (const link of result.links) console.log(link);

// Narrow to specific hosts instead of a blanket internal/external split.
const docsOnly = await fetch('https://crawlforge.dev/api/v1/tools/extract_links', {
  method: 'POST',
  headers: {
    'X-API-Key': process.env.CRAWLFORGE_API_KEY!,
    'Content-Type': 'application/json',
  },
  body: JSON.stringify({
    url: 'https://example.com',
    filter_domains: ['docs.example.com'],
  }),
});

Response Example

200 OK210ms

{
  "success": true,
  "data": {
    "url": "https://example.com",
    "links": [
      {
        "text": "Home",
        "url": "https://example.com/",
        "internal": true
      },
      {
        "text": "About",
        "url": "https://example.com/about",
        "internal": true
      },
      {
        "text": "Products",
        "url": "https://example.com/products",
        "internal": true
      },
      {
        "text": "External Resource",
        "url": "https://other-site.com",
        "internal": false
      }
    ],
    "total_links": 4,
    "internal_links": 3,
    "external_links": 1
  },
  "credits_used": 1,
  "credits_remaining": 999,
  "processing_time": 210
}

Field Descriptions

data.linksArray of all discovered links with text and URL

data.total_linksTotal number of links found

data.internal_linksNumber of links to the same domain

data.external_linksNumber of links to external domains

Error Handling

Blocked by robots.txt (403 Forbidden)

The target site's robots.txt disallows this path for CrawlForge. Set respect_robots: false to override if you have your own agreement with the target — the override is recorded against your API key. The override does not reach a host on CrawlForge's permanent opt-out list, which is refused whatever respect_robots is set to.

Related Tools

fetch_url

Fetch pages before extracting links (1 credit)

smart_crawl

Intelligent crawling with link discovery (8 credits)

Ready to extract links? Sign up for free and get 1,000 credits.