extract_text

Extract clean, readable text from HTML with intelligent parsing. Automatically removes scripts, styles, and boilerplate content while preserving the main text content.

Use Cases

Article Extraction for LLMs

Extract clean article text for summarization, analysis, or AI processing

Content Analysis

Get plain text for word count, readability analysis, or sentiment detection

Clean Text for Summarization

Remove HTML noise before passing to summarization models

Boilerplate Removal

Remove ads, navigation, and other non-content elements

Endpoint

POST/api/v1/tools/extract_text

Auth Required

1 req/s on Free plan

1 credit

Parameters

Note: You must provide either html or url. If both are provided, html takes precedence.

Name	Type	Required	Default	Description
html	string	Optional	-	HTML content to extract text from (provide either html or url) Example: <html><body><h1>Hello World</h1></body></html>
url	string	Optional	-	URL to fetch and extract text from (provide either html or url) Example: https://example.com/article
selector	string	Optional	-	CSS selector to target specific elements (default: entire page) Example: article, .content, #main
clean	boolean	Optional	true	Collapse runs of spaces within a line and drop blank lines; the line breaks between blocks are kept. Applies to `output_format: "text"` only. Example: true
preserve_links	boolean	Optional	false	Append each link as a `text (href)` line after the extracted text. REST API only — the MCP tool has no such option — and applies to `output_format: "text"` only. Example: false
preserve_formatting	boolean	Optional	false	Keep paragraph and heading breaks from the HTML source. REST API only — the MCP tool has no such option — and applies to `output_format: "text"` only. Example: false
max_length	number	Optional	-	Maximum length of the returned text or markdown. A longer result is cut and ends in `...`, and `data.truncated` is `true`; `data.truncated` is always present, `false` when nothing was cut. Example: 5000
output_format	string	Optional	text	`"text"` (default) returns `data.text`, one line per block element — `<h1>Hi</h1><p>there</p>` reads `Hi\nthere`. `"markdown"` returns `data.markdown` instead: the selected elements (or the body) converted to Markdown with Turndown. Either way `data` carries `output_format`, `word_count` and `char_count`, under the names the MCP tool uses; a markdown result is counted without its Markdown syntax characters (`#`, `*`, `_`, `[`, `]` and backticks). `clean`, `preserve_links` and `preserve_formatting` apply to text only; `max_length` applies to both. Example: markdown
respect_robots	boolean	Optional	true	Respect the target site's robots.txt. Left at `true`, a path disallowed for `CrawlForge` is refused with 403 before anything is fetched and no credits are charged. Set it to `false` only for a target you have your own agreement with — the response then carries a `warnings` entry and the override is recorded against your API key. Example: true
escalate	boolean	Optional	false	Opt in to one automatic retry through the stealth browser when the plain fetch of `url` meets a wall instead of the page — a Cloudflare, Amazon, DataDome, PerimeterX, Akamai or Vercel challenge page, a bare 403, 429 or 444, or an empty shell or error placeholder served as 2xx. The plain fetch always runs first; a 404, a 5xx, a binary such as a PDF, or a host whose `Retry-After` window is still open never escalates, and `html` input is never fetched. The text is extracted from the rendered HTML, so the response has the same shape, plus `escalated` (whenever this is set) and `stealth: { engine, vendor_detected }` when the render ran. Escalation reuses the same stealth path as [stealth_mode](/docs/api-reference/tools/stealth-mode) and the same robots.txt gate, and adds no new handling of a bot defence; a wall the render is refused at too still comes back as a blocked page carrying `escalated: true`, charged nothing. Adds 5 credits, charged only when the render actually runs. Example: true
escalate_engine	string	Optional	playwright	Browser engine for the escalation stage: "playwright" (default) or "camoufox" (Firefox-based, stronger anti-detection; available only where installed on the backend). Ignored unless `escalate` is `true` and the plain fetch was walled. Example: camoufox
redact_pii	boolean \| object	Optional	false	Remove personal data from the text this call returns, before the result is stored or sent back. `true` is shorthand for `{ mode: "fast" }` — all four regex classes, tagged. Whenever you ask for redaction the response carries `redaction: { entities, count, mode }` inside `data`, even when nothing matched (`count: 0`), so "found nothing" is never mistaken for "the parameter was ignored"; a class with no hits is left out rather than reported as `0`. Redaction runs before the result is stored, so an oversized result read back later with [read_result](/docs/api-reference/tools/read-result) is already redacted. Two deliberate limits: addresses (`url`, `link`, `href`, `canonical_url`) are never redacted, and counters derived from the text (`content_length`, `word_count`, `character_count`) describe it as extracted, before redaction. Example: true

Fetching a URL

The page is fetched once with a 15-second timeout. A 429 or 503 whose Retry-After is 10 seconds or less is waited out and fetched once more; a longer or missing Retry-After is not retried.
HTML, XHTML, XML and every text/* type are read, and a response with no Content-Type is read as HTML. A PDF, an image or another binary returns 415 UNSUPPORTED_CONTENT_TYPE with next_step: "process_document" and charges nothing: read it with process_document.
A JSON body (application/json or *+json) is returned in text as it is, with the warning the target returned application/json; its body is returned as text. With output_format: "markdown" it is returned in markdown instead.
A page that renders client-side — under 200 characters of visible text and either an empty <div id="root">, "app" or "__next" or text saying the page needs JavaScript ("Enable JavaScript…") — succeeds with rendered: false in data and the warning page renders client-side; use scrape in warnings when no stealth render ran, so an empty shell is not mistaken for a short page. html input is checked the same way.

Request Examples

cURL - Extract from URL

terminalBash

curl -X POST https://www.crawlforge.dev/api/v1/tools/extract_text \
  -H "X-API-Key: cf_test_YOUR_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://example.com/article",
    "selector": "article",
    "clean": true,
    "max_length": 5000
  }'

TypeScript - Extract from HTML

extractText.tsTypescript

// npm install crawlforge-sdk
import { CrawlForge } from 'crawlforge-sdk';

const client = new CrawlForge({ apiKey: process.env.CRAWLFORGE_API_KEY });

const htmlContent =
  '<html><body>' +
  '<article>' +
  '<h1>Article Title</h1>' +
  '<p>This is the main content of the article.</p>' +
  '<a href="/related">Related Article</a>' +
  '</article>' +
  '</body></html>';

const result = await client.extractText({
  html: htmlContent,
  selector: 'article',
  clean: true,
  preserve_links: true,
});

// result.data is untyped in crawlforge-sdk 0.1 — its shape is the Response Example below.
const { text, metadata } = result.data as {
  text: string; metadata: { word_count: number; character_count: number };
};
console.log('Extracted text:', text);
console.log('Word count:', metadata.word_count);
console.log('Character count:', metadata.character_count);

Python - Extract with Selector

extract_text.pyPython

# pip install crawlforge
from crawlforge import CrawlForge

client = CrawlForge()  # reads CRAWLFORGE_API_KEY

result = client.extract_text(
    url='https://example.com/article',
    selector='article, .main-content',
    clean=True,
    preserve_formatting=True,
    max_length=10000,
)

# result.data is a plain dict — its shape is the Response Example below.
text = result.data['text']
metadata = result.data['metadata']

print(f"Title: {metadata['title']}")
print(f"Word count: {metadata['word_count']}")
print(f"Character count: {metadata['character_count']}")
print(f"\nExtracted text:\n{text[:500]}...")

Response Example

200 OK180ms

{
  "success": true,
  "data": {
    "text": "Article Title\nThis is the main content of the article. It contains useful information that has been extracted from the HTML.\nLinks:\nRelated Article (/related)",
    "output_format": "text",
    "word_count": 248,
    "char_count": 1432,
    "truncated": false,
    "metadata": {
      "title": "Article Title - Example Site",
      "description": "Meta description of the article",
      "word_count": 248,
      "character_count": 1432,
      "selector_used": "article",
      "links_preserved": true,
      "formatting_preserved": false
    }
  },
  "credits_used": 1,
  "credits_remaining": 999,
  "processing_time": 180
}

Field Descriptions

data.textThe extracted plain text content

data.output_format"text" or "markdown"; a markdown call returns data.markdown in place of data.text

data.word_countWords in the returned text (named as the MCP tool names it)

data.char_countCharacters in the returned text

data.truncatedtrue when max_length cut the text (it then ends in ...); always present

data.metadata.word_countTotal number of words in the extracted text

data.metadata.character_countTotal number of characters

data.metadata.selector_usedThe CSS selector that was applied

credits_usedCredits deducted for this request (1 per extraction)

Error Handling

Missing Input (400 Bad Request)

Neither html nor url was provided. You must provide at least one.

Invalid Selector (400 Bad Request)

The CSS selector is invalid or matches no elements. Verify your selector syntax.

Blocked by a Bot Wall (403, 444 or 502)

The target answered with a bot-defence challenge page instead of the content, whatever its HTTP status. The response is success: false with error.code BLOCKED, a blocked object naming the vendor (for example cloudflare or akamai) and the evidence it was recognised by, and a next_step: extract_text with escalate:true, or stealth_mode operation:"scrape". The status is the target's own when it is 403, 404, 410, 429, 444 or 451, and 502 otherwise — a wall served as 200 included. No credits are charged. With escalate: true the wall is rendered once in the stealth browser instead; if the render is refused too, the same error carries escalated: true and no next_step.

Target Answered with an Error (502 Bad Gateway)

The target answered with an HTTP error and no recognised bot wall. error.code is UPSTREAM_HTTP and the message is Target answered HTTP <n>. A 403, 404, 410, 429, 444 or 451 is passed through as the response status; every other error — a 5xx, or the target's own 400 or 401 — is a 502, so it cannot be mistaken for a problem with your request or API key. A 403, 429 or 444 also carries next_step. No credits are charged.

Blocked by robots.txt (403 Forbidden)

The target site's robots.txt disallows this path for CrawlForge. Set respect_robots: false to override if you have your own agreement with the target — the override is recorded against your API key. The override does not reach a host on CrawlForge's permanent opt-out list, which is refused whatever respect_robots is set to.

Unsupported Content Type (415 Unsupported Media Type)

The target served a PDF, an image or another binary. error.code is UNSUPPORTED_CONTENT_TYPE, the message is Unsupported content type <type>: extract_text reads HTML and text, and next_step is process_document. The body is not read, nothing is escalated and no credits are charged.

Escalation Backend Not Configured (503 TOOL_NOT_AVAILABLE)

escalate: true runs its stealth stage on the CrawlForge execution backend. When that backend is not configured the stage returns 503 and no credits are charged.

Credit Cost

1 credit

1 credit per request

Each text extraction costs 1 credit, regardless of content size or selector complexity. With escalate: true a call is projected at up to 6 credits and pays the 5-credit escalation add-on only when the plain fetch was walled and the stealth render ran; every failure is charged nothing.

Related Tools

fetch_url

Fetch HTML content before extracting text (1 credit)

scrape_structured

Extract structured data with custom selectors (2 credits)

content_analysis

Analyze extracted text for sentiment and topics (4 credits)

Ready to extract clean text? Sign up for free and get 1,000 credits to start.