extract_content

Fetch a page, drop the chrome, and return what is actually on it. Navigation, headers, footers, sidebars, scripts, styles and ad containers are removed, then the article body is located and returned as clean text or as HTML — with the title, byline and publish date the page declares.

Use Cases

RAG Ingestion

Turn an article into clean text before chunking and embedding, without the nav menu polluting every chunk.

Reader Views

Render a distraction-free version of a page, keeping the HTML structure by setting clean_html to false.

Byline Extraction

Collect author and publish date from the page's own meta tags for attribution or freshness checks.

Image Harvesting

Get every in-article image as an absolute URL, already resolved against the page's own address.

Citation Graphs

Enable include_links to capture the outbound links inside the article body — not the ones in the site's navigation.

Content Migration

Lift article HTML off an old site and load it somewhere else, without carrying the old template with it.

Endpoint

POST/api/v1/tools/extract_content

Auth Required

1 req/s on Free plan

2 credits

Parameters

Name	Type	Required	Default	Description
url	string	Required	-	The page to extract from. Must be a valid absolute http or https URL. Example: https://example.com/blog/crawl-budgets
include_images	boolean	Optional	true	Return every `<img>` inside the extracted region as an absolute URL, plus `image_count`. Sources that will not resolve against the page URL are skipped. Set false and both keys are absent. Example: true
include_links	boolean	Optional	false	Return every `<a href>` inside the extracted region as `{ href, text }` with absolute hrefs, plus `link_count`. Off by default — turn it on and both keys appear. Example: true
clean_html	boolean	Optional	true	True returns collapsed plain text and sets `content_type` to `text`. False returns the region's inner HTML and sets `content_type` to `html` — still with the boilerplate elements removed. Example: true
extract_main_content	boolean	Optional	true	Narrow to the article body, trying `<article>`, then `<main>`, then `.content`, then `#content`, and falling back to `<body>` when none match. Set false to keep the whole body. Example: true
timeout	number	Optional	10000	Fetch timeout in milliseconds, between 1000 and 30000. Example: 10000
respect_robots	boolean	Optional	true	Respect the target site's robots.txt. Left at `true`, a path disallowed for `CrawlForge` is refused with 403 before anything is fetched and no credits are charged. Set it to `false` only for a target you have your own agreement with — the response then carries a `warnings` entry and the override is recorded against your API key. Example: true

Request Examples

terminalBash

curl -X POST https://crawlforge.dev/api/v1/tools/extract_content \
  -H "X-API-Key: cf_test_YOUR_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://example.com/blog/crawl-budgets",
    "include_links": true
  }'

Response Example

200 OK680ms

{
  "success": true,
  "data": {
    "url": "https://example.com/blog/crawl-budgets",
    "content": "Why crawl budgets matter Search engines allocate a finite number of requests to each site. See the sitemap guide for the mechanics.",
    "content_length": 131,
    "content_type": "text",
    "title": "Why crawl budgets matter — Acme Blog",
    "author": "Dana Reyes",
    "publish_date": "2026-08-14T09:30:00Z",
    "images": [
      "https://example.com/img/crawl-budget.png"
    ],
    "image_count": 1,
    "links": [
      {
        "href": "https://example.com/docs/sitemaps",
        "text": "sitemap guide"
      }
    ],
    "link_count": 1
  },
  "credits_used": 2,
  "credits_remaining": 998,
  "processing_time": 680
}

Field Descriptions

data.contentThe extracted region. Plain text with runs of whitespace collapsed, or inner HTML when `clean_html` is false

data.content_lengthCharacter count of `content` — a low number usually means the page is client-rendered

data.content_type`text` or `html`, mirroring `clean_html`

data.titleThe `<title>` element of the whole page, not a heading inside the article

data.authorFrom the page's `meta[name=author]` tag. `null` when the page does not declare one

data.publish_dateFrom `meta[property=article:published_time]`, returned verbatim. `null` when absent

data.imagesAbsolute URLs, resolved against the page address. Present when `include_images` is true

data.linksLinks inside the extracted region only. Present when `include_links` is true

credits_usedFixed 2 credits per page

Error Handling

Invalid URL (400 Bad Request)

VALIDATION_ERROR. url is required and must parse as an absolute URL. The same status covers a timeout outside 1000-30000.

Page Too Large (413 Payload Too Large)

RESPONSE_TOO_LARGE. The page exceeded the 25MB read limit and was refused rather than buffered.

Target Timed Out (504 Gateway Timeout)

FETCH_TIMEOUT. The page stopped responding while sending its body. Raise timeout, up to 30000ms.

Fetch Failed (502 Bad Gateway)

FETCH_FAILED. The response body could not be read — connection reset, or a body that is not decodable text.

Extraction Failed (500 Internal Server Error)

TOOL_ERROR. Nothing is charged for a failed call; credits are deducted only after extraction succeeds.

Blocked by robots.txt (403 Forbidden)

The target site's robots.txt disallows this path for CrawlForge. Set respect_robots: false to override if you have your own agreement with the target — the override is recorded against your API key. The override does not reach a host on CrawlForge's permanent opt-out list, which is refused whatever respect_robots is set to.

Note: The page is fetched, not rendered, so a client-rendered article returns a short content rather than an error. If content_length comes back near zero on a page you can see in a browser, reach for scrape_with_actions instead.

Credit Cost

2 credits

2 credits per page

Flat rate regardless of page size or which options you enable. Failed calls are not charged.

What's Included:

Boilerplate removal (nav, header, footer, aside, ads, scripts, styles)

Main-content detection with a <body> fallback

Clean text or article HTML

Title, author and publish date from the page's meta

Absolute image and link URLs from inside the article

Plan Recommendations:

Free Plan: 1,000 one-time trial credits = 500 pages

Hobby Plan: 5,000 credits = 2,500 pages ($19/mo)

Professional Plan: 50,000 credits = 25,000 pages ($99/mo)

Related Tools

extract_text

Plain text off the whole page, no article detection (1 credit)

summarize_content

Condense the article instead of returning it whole (4 credits)

analyze_content

Language, keywords, sentiment and readability (3 credits)

batch_scrape

Do it across a URL list in one call (5 credits per URL)

Ready to try extract_content? Sign up for free and get 1,000 credits to start building.

Use Cases

RAG Ingestion

Turn an article into clean text before chunking and embedding, without the nav menu polluting every chunk.

Reader Views

Render a distraction-free version of a page, keeping the HTML structure by setting clean_html to false.

Byline Extraction

Collect author and publish date from the page's own meta tags for attribution or freshness checks.

Image Harvesting

Get every in-article image as an absolute URL, already resolved against the page's own address.

Citation Graphs

Enable include_links to capture the outbound links inside the article body — not the ones in the site's navigation.

Content Migration

Lift article HTML off an old site and load it somewhere else, without carrying the old template with it.

Parameters

Name	Type	Required	Default	Description
url	string	Required	-	The page to extract from. Must be a valid absolute http or https URL. Example: https://example.com/blog/crawl-budgets
include_images	boolean	Optional	true	Return every `<img>` inside the extracted region as an absolute URL, plus `image_count`. Sources that will not resolve against the page URL are skipped. Set false and both keys are absent. Example: true
include_links	boolean	Optional	false	Return every `<a href>` inside the extracted region as `{ href, text }` with absolute hrefs, plus `link_count`. Off by default — turn it on and both keys appear. Example: true
clean_html	boolean	Optional	true	True returns collapsed plain text and sets `content_type` to `text`. False returns the region's inner HTML and sets `content_type` to `html` — still with the boilerplate elements removed. Example: true
extract_main_content	boolean	Optional	true	Narrow to the article body, trying `<article>`, then `<main>`, then `.content`, then `#content`, and falling back to `<body>` when none match. Set false to keep the whole body. Example: true
timeout	number	Optional	10000	Fetch timeout in milliseconds, between 1000 and 30000. Example: 10000
respect_robots	boolean	Optional	true	Respect the target site's robots.txt. Left at `true`, a path disallowed for `CrawlForge` is refused with 403 before anything is fetched and no credits are charged. Set it to `false` only for a target you have your own agreement with — the response then carries a `warnings` entry and the override is recorded against your API key. Example: true

curl -X POST https://crawlforge.dev/api/v1/tools/extract_content \ -H "X-API-Key: cf_test_YOUR_KEY" \ -H "Content-Type: application/json" \ -d '{ "url": "https://example.com/blog/crawl-budgets", "include_links": true }'

Response Example

200 OK680ms

{
  "success": true,
  "data": {
    "url": "https://example.com/blog/crawl-budgets",
    "content": "Why crawl budgets matter Search engines allocate a finite number of requests to each site. See the sitemap guide for the mechanics.",
    "content_length": 131,
    "content_type": "text",
    "title": "Why crawl budgets matter — Acme Blog",
    "author": "Dana Reyes",
    "publish_date": "2026-08-14T09:30:00Z",
    "images": [
      "https://example.com/img/crawl-budget.png"
    ],
    "image_count": 1,
    "links": [
      {
        "href": "https://example.com/docs/sitemaps",
        "text": "sitemap guide"
      }
    ],
    "link_count": 1
  },
  "credits_used": 2,
  "credits_remaining": 998,
  "processing_time": 680
}

Field Descriptions

data.contentThe extracted region. Plain text with runs of whitespace collapsed, or inner HTML when `clean_html` is false

data.content_lengthCharacter count of `content` — a low number usually means the page is client-rendered

data.content_type`text` or `html`, mirroring `clean_html`

data.titleThe `<title>` element of the whole page, not a heading inside the article

data.authorFrom the page's `meta[name=author]` tag. `null` when the page does not declare one

data.publish_dateFrom `meta[property=article:published_time]`, returned verbatim. `null` when absent

data.imagesAbsolute URLs, resolved against the page address. Present when `include_images` is true

data.linksLinks inside the extracted region only. Present when `include_links` is true

credits_usedFixed 2 credits per page

Error Handling

Invalid URL (400 Bad Request)

VALIDATION_ERROR. url is required and must parse as an absolute URL. The same status covers a timeout outside 1000-30000.

Page Too Large (413 Payload Too Large)

RESPONSE_TOO_LARGE. The page exceeded the 25MB read limit and was refused rather than buffered.

Target Timed Out (504 Gateway Timeout)

FETCH_TIMEOUT. The page stopped responding while sending its body. Raise timeout, up to 30000ms.

Fetch Failed (502 Bad Gateway)

FETCH_FAILED. The response body could not be read — connection reset, or a body that is not decodable text.

Extraction Failed (500 Internal Server Error)

TOOL_ERROR. Nothing is charged for a failed call; credits are deducted only after extraction succeeds.

Blocked by robots.txt (403 Forbidden)

Credit Cost

2 credits

2 credits per page

Flat rate regardless of page size or which options you enable. Failed calls are not charged.

What's Included:

Boilerplate removal (nav, header, footer, aside, ads, scripts, styles)

Main-content detection with a <body> fallback

Clean text or article HTML

Title, author and publish date from the page's meta

Absolute image and link URLs from inside the article

Plan Recommendations:

Free Plan: 1,000 one-time trial credits = 500 pages

Hobby Plan: 5,000 credits = 2,500 pages ($19/mo)

Professional Plan: 50,000 credits = 25,000 pages ($99/mo)