fetch_url

Fetch and parse web pages with automatic redirect handling, timeout control, and custom headers. Perfect for retrieving HTML content from any publicly accessible URL.

Use Cases

Page Content Retrieval

Fetch HTML content from any webpage for further processing or analysis

API Data Fetching

Make GET requests to REST APIs and retrieve JSON responses

Health Checks

Monitor website availability and response times

Simple Downloads

Download static assets, documents, or page content

Endpoint

POST/api/v1/tools/fetch_url

Auth Required

1 req/s on Free plan

1 credit

Parameters

Name	Type	Required	Default	Description
url	string	Required	-	The URL to fetch (must include protocol: http:// or https://) Example: https://example.com
headers	object	Optional	-	Custom HTTP headers to include in the request Example: {"Accept": "text/html", "User-Agent": "MyBot/1.0"}
timeout	number	Optional	10000	Request timeout in milliseconds (1000-30000) Example: 15000
follow_redirects	boolean	Optional	true	Whether to follow HTTP redirects automatically Example: true
user_agent	string	Optional	-	Custom User-Agent header (overrides default) Example: Mozilla/5.0 (compatible; CrawlBot/1.0)
respect_robots	boolean	Optional	true	Respect the target site's robots.txt. Left at `true`, a path disallowed for `CrawlForge` is refused with 403 before anything is fetched and no credits are charged. Set it to `false` only for a target you have your own agreement with — the response then carries a `warnings` entry and the override is recorded against your API key. Example: true
max_inline_chars	number	Optional	40000	Largest result to return inline, in characters of its JSON (1,000-10,000,000). Over it the response carries `preview` (the first `max_inline_chars` characters of the body), `result_handle`, `total_chars`, `truncated: true` and `expires_at`, and [read_result](/docs/api-reference/tools/read-result) reads the rest for 1 credit per call. Stored results are kept for 1 hour. Example: 40000

Request Examples

cURL

terminalBash

curl -X POST https://crawlforge.dev/api/v1/tools/fetch_url \
  -H "X-API-Key: cf_test_YOUR_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://example.com",
    "timeout": 15000,
    "follow_redirects": true
  }'

TypeScript

fetchUrl.tsTypescript

// npm install crawlforge-sdk
import { CrawlForge } from 'crawlforge-sdk';

const client = new CrawlForge({ apiKey: process.env.CRAWLFORGE_API_KEY });

const result = await client.fetchUrl({
  url: 'https://example.com',
  timeout: 15000,
  headers: {
    'Accept': 'text/html',
    'Accept-Language': 'en-US',
  },
});

// result.data is untyped in crawlforge-sdk 0.1 — its shape is the Response Example below.
const page = result.data as { url: string; status: number; content_length: number };
console.log('Page fetched:', page.url);
console.log('Status:', page.status);
console.log('Content length:', page.content_length);
console.log('Credits used:', result.creditsUsed);
console.log('Credits remaining:', result.creditsRemaining);

Python

fetch_url.pyPython

# pip install crawlforge
from crawlforge import CrawlForge

client = CrawlForge()  # reads CRAWLFORGE_API_KEY

result = client.fetch_url(
    url='https://example.com',
    timeout=15000,
    headers={
        'Accept': 'text/html',
        'Accept-Language': 'en-US',
    },
)

# result.data is a plain dict — its shape is the Response Example below.
page = result.data
print(f"Page fetched: {page['url']}")
print(f"Status: {page['status']}")
print(f"Content length: {page['content_length']}")
print(f"Credits used: {result.credits_used}")
print(f"Credits remaining: {result.credits_remaining}")

Response Example

200 OK245ms

{
  "success": true,
  "data": {
    "url": "https://example.com",
    "status": 200,
    "status_text": "OK",
    "headers": {
      "content-type": "text/html; charset=UTF-8",
      "content-length": "1256",
      "server": "nginx"
    },
    "content": "Example Domain......",
    "content_length": 1256,
    "content_type": "text/html; charset=UTF-8",
    "redirected": false,
    "final_url": "https://example.com",
    "response_time_ms": 245
  },
  "credits_used": 1,
  "credits_remaining": 999,
  "processing_time": 245
}

Field Descriptions

data.urlThe original URL that was requested

data.statusHTTP status code of the response

data.contentThe full HTML content of the page

data.content_lengthSize of the content in bytes

data.final_urlFinal URL after following redirects

data.response_time_msHow long the target took to respond and deliver its body, in milliseconds

credits_usedCredits deducted for this request (1 per fetch)

credits_remainingYour remaining credit balance

Error Handling

Invalid URL (400 Bad Request)

The URL format is invalid. Ensure it includes the protocol (http:// or https://)

Timeout Error (500 Internal Server Error)

The request took longer than the specified timeout. Try increasing the timeout parameter.

Insufficient Credits (402 Payment Required)

Your account doesn't have enough credits. Purchase more credits or upgrade your plan.

Rate Limit Exceeded (429 Too Many Requests)

You've exceeded your plan's rate limit. Wait a moment or upgrade your plan for higher limits.

Blocked by robots.txt (403 Forbidden)

The target site's robots.txt disallows this path for CrawlForge. Set respect_robots: false to override if you have your own agreement with the target — the override is recorded against your API key. The override does not reach a host on CrawlForge's permanent opt-out list, which is refused whatever respect_robots is set to.

Pro Tip:

Always implement retry logic with exponential backoff for production applications. See our Error Handling Guide for best practices.