Advanced ToolDocuments2 credits

process_document

Point it at a document URL and get structured JSON back. PDFs yield text, embedded metadata, page count and ruled tables; CSVs come back as rows; plain text and HTML return their text. The type is detected from the response headers, the URL and the file's own magic bytes, so you do not have to know it in advance.

Use Cases

Report Ingestion

Turn quarterly PDFs into text and tables you can diff, index or load into a warehouse.

RAG over Documents

Extract paper and manual text for chunking, with the title and author already separated out.

Table Harvesting

Pull ruled tables out of a PDF as row arrays, with the page each one came from.

Filing Monitoring

Watch a regulator's PDF publications and parse each new one the moment it appears.

CSV Endpoints

Read a CSV that is published at a URL as both raw text and parsed rows, without downloading it yourself.

Metadata Audits

Collect embedded title, author, producer and dates across a document set to find stale or mislabelled files.

Endpoint

POST/api/v1/tools/process_document

Auth Required

1 req/s on Free plan

2 credits

Parameters

Name	Type	Required	Default	Description
url	string	Required	-	The document to fetch. Must be a valid absolute http or https URL. Example: https://example.com/reports/q3-infrastructure.pdf
document_type	string	Optional	auto	`auto`, `pdf`, `csv`, `txt`, `docx` or `xlsx`. `auto` decides from the `Content-Type` header, the URL extension and the file's magic bytes. `docx` and `xlsx` are accepted by validation but return 501 on this endpoint. Example: auto
extract_text	boolean	Optional	true	Return the document's text as `text`, with `text_length`. Capped at 200,000 characters and, for PDFs, the first 200 pages — a note says so when either limit bites. Example: true
extract_metadata	boolean	Optional	true	Return embedded document properties. PDFs give title, author, subject, creator, producer and both dates; HTML gives title, description and author; CSV and TXT carry none. Example: true
extract_tables	boolean	Optional	false	Return ruled tables as row arrays. PDFs give up to 20 tables of 1,000 rows each; a CSV comes back as a single table; HTML pages get a note explaining that tables are not extracted here. Example: true
extract_images	boolean	Optional	false	Not available on the hosted REST API. Setting it returns `images: null` and a note pointing at the CrawlForge MCP server, rather than pretending to have found none. Example: false
timeout	number	Optional	30000	Fetch timeout in milliseconds, 1000-60000. The endpoint runs inside a 30-second function, so the fetch is capped at 20,000ms whatever you send. Example: 30000
respect_robots	boolean	Optional	true	Respect the target site's robots.txt. Left at `true`, a path disallowed for `CrawlForge` is refused with 403 before anything is fetched and no credits are charged. Set it to `false` only for a target you have your own agreement with — the response then carries a `warnings` entry and the override is recorded against your API key. Example: true

Limits Worth Knowing

Every one of these announces itself in notes rather than failing quietly.

25MB per document

Checked against Content-Length first, then against the bytes actually received, so a lying header does not get through.

200 pages of text

Text and tables are read from the first 200 pages. page_count still reports the document's real length.

200,000 characters

Longer text is truncated at that point, with a note saying it happened.

Vertical rules define columns

Tables ruled only horizontally — the academic-paper convention — keep their rows but report a single column, and say so in notes.

Request Examples

terminalBash

curl -X POST https://crawlforge.dev/api/v1/tools/process_document \
  -H "X-API-Key: cf_test_YOUR_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://example.com/reports/q3-infrastructure.pdf",
    "extract_tables": true
  }'

Response Example

200 OK4,210ms

{
  "success": true,
  "data": {
    "url": "https://example.com/reports/q3-infrastructure.pdf",
    "document_type": "pdf",
    "content_type": "application/pdf",
    "file_size": 1842665,
    "processed_at": "2026-08-27T02:27:50.833Z",
    "page_count": 14,
    "text": "Q3 Infrastructure Review\n\nRequest volume grew 38% quarter over quarter while p95 latency held flat...",
    "text_length": 101,
    "metadata": {
      "title": "Q3 Infrastructure Review",
      "author": "Dana Reyes",
      "subject": "Quarterly capacity planning",
      "creator": "LaTeX with hyperref",
      "producer": "pdfTeX-1.40.25",
      "creation_date": "D:20260812141055Z",
      "modification_date": "D:20260814093012Z"
    },
    "tables": [
      {
        "page": 4,
        "rows": 3,
        "columns": 3,
        "data": [
          [
            "Region",
            "Requests",
            "p95 ms"
          ],
          [
            "us-east",
            "18,204,551",
            "412"
          ],
          [
            "eu-west",
            "9,118,340",
            "458"
          ]
        ]
      }
    ]
  },
  "credits_used": 2,
  "credits_remaining": 998,
  "processing_time": 4210
}

Field Descriptions

data.document_typeWhat the type detection settled on, which can differ from what you asked for

data.content_typeThe `Content-Type` header the server sent, verbatim — often wrong, which is why magic bytes are checked too

data.file_sizeBytes actually received

data.page_countPages in the document. PDFs only, and reported in full even when only the first 200 were read

data.text_lengthCharacters in `text` after any truncation, not the document's full length

data.metadata.creation_datePDF date strings are returned exactly as embedded, in the `D:YYYYMMDDHHmmSS` form. Parse them yourself

data.tables[].pageThe page the table was found on, so a result can be traced back to the source

data.tables[].columnsDerived from the vertical rules drawn in the PDF. A value of 1 means the table was not ruled into columns

credits_usedFlat 2 credits per document, whatever its page count

Error Handling

Invalid URL (400 Bad Request)

VALIDATION_ERROR. url is required and must parse as an absolute URL. The same status covers a timeout outside 1000-60000, an unknown document_type, and a URL that resolves to a private or local address.

docx or xlsx (501 Not Implemented)

UNSUPPORTED_DOCUMENT_TYPE. Office formats are not parsed on the hosted REST API. The CrawlForge MCP server (npm: crawlforge-mcp-server) handles them with the same API key.

Document Too Large (413 Payload Too Large)

DOCUMENT_TOO_LARGE. Over the 25MB limit. Checked before the download and again after, so an inaccurate Content-Length still gets caught.

Document Unreachable (502 Bad Gateway)

DOCUMENT_FETCH_ERROR when the URL returned a non-2xx status — the message includes it — or FETCH_FAILED when the connection itself failed.

Target Timed Out (504 Gateway Timeout)

FETCH_TIMEOUT. The document did not arrive within the fetch budget.

Processing Failed (500 Internal Server Error)

TOOL_ERROR. A corrupt or encrypted PDF lands here. Nothing is charged for a failed call.

Blocked by robots.txt (403 Forbidden)

The target site's robots.txt disallows this path for CrawlForge. Set respect_robots: false to override if you have your own agreement with the target — the override is recorded against your API key. The override does not reach a host on CrawlForge's permanent opt-out list, which is refused whatever respect_robots is set to.

Note: A scanned PDF is a picture of text, so it parses successfully and returns almost nothing — page_count will look right while text_length is near zero. There is no OCR on this endpoint; check text_length before trusting the result.

Credit Cost

2 credits

2 credits per document

Flat rate: a 200-page PDF costs the same as a one-page one, and enabling tables or metadata costs nothing extra. Failed calls are not charged.

What's Included:

Type detection from headers, URL and magic bytes

PDF text up to 200 pages and 200,000 characters

Embedded PDF metadata, including both timestamps

Up to 20 ruled tables with their page numbers

CSV parsing and plain-text and HTML extraction

Plan Recommendations:

Free Plan: 1,000 one-time trial credits = 500 documents

Hobby Plan: 5,000 credits = 2,500 documents ($19/mo)

Professional Plan: 50,000 credits = 25,000 documents ($99/mo)

Related Tools

extract_content

For HTML articles rather than documents (2 credits)

search_web

Find PDFs to process with a file_type filter (5 credits)

summarize_content

Condense the extracted text afterwards (4 credits)

batch_scrape

Collect many URLs at once, HTML only (5 credits per URL)

Ready to try process_document? Sign up for free and get 1,000 credits to start building.

process_document

Use Cases

Report Ingestion

Turn quarterly PDFs into text and tables you can diff, index or load into a warehouse.

RAG over Documents

Extract paper and manual text for chunking, with the title and author already separated out.

Table Harvesting

Pull ruled tables out of a PDF as row arrays, with the page each one came from.

Filing Monitoring

Watch a regulator's PDF publications and parse each new one the moment it appears.

CSV Endpoints

Read a CSV that is published at a URL as both raw text and parsed rows, without downloading it yourself.

Metadata Audits

Collect embedded title, author, producer and dates across a document set to find stale or mislabelled files.

Parameters

Name	Type	Required	Default	Description
url	string	Required	-	The document to fetch. Must be a valid absolute http or https URL. Example: https://example.com/reports/q3-infrastructure.pdf
document_type	string	Optional	auto	`auto`, `pdf`, `csv`, `txt`, `docx` or `xlsx`. `auto` decides from the `Content-Type` header, the URL extension and the file's magic bytes. `docx` and `xlsx` are accepted by validation but return 501 on this endpoint. Example: auto
extract_text	boolean	Optional	true	Return the document's text as `text`, with `text_length`. Capped at 200,000 characters and, for PDFs, the first 200 pages — a note says so when either limit bites. Example: true
extract_metadata	boolean	Optional	true	Return embedded document properties. PDFs give title, author, subject, creator, producer and both dates; HTML gives title, description and author; CSV and TXT carry none. Example: true
extract_tables	boolean	Optional	false	Return ruled tables as row arrays. PDFs give up to 20 tables of 1,000 rows each; a CSV comes back as a single table; HTML pages get a note explaining that tables are not extracted here. Example: true
extract_images	boolean	Optional	false	Not available on the hosted REST API. Setting it returns `images: null` and a note pointing at the CrawlForge MCP server, rather than pretending to have found none. Example: false
timeout	number	Optional	30000	Fetch timeout in milliseconds, 1000-60000. The endpoint runs inside a 30-second function, so the fetch is capped at 20,000ms whatever you send. Example: 30000
respect_robots	boolean	Optional	true	Respect the target site's robots.txt. Left at `true`, a path disallowed for `CrawlForge` is refused with 403 before anything is fetched and no credits are charged. Set it to `false` only for a target you have your own agreement with — the response then carries a `warnings` entry and the override is recorded against your API key. Example: true

Limits Worth Knowing

Every one of these announces itself in notes rather than failing quietly.

25MB per document

Checked against Content-Length first, then against the bytes actually received, so a lying header does not get through.

200 pages of text

Text and tables are read from the first 200 pages. page_count still reports the document's real length.

200,000 characters

Longer text is truncated at that point, with a note saying it happened.

Vertical rules define columns

Tables ruled only horizontally — the academic-paper convention — keep their rows but report a single column, and say so in notes.

curl -X POST https://crawlforge.dev/api/v1/tools/process_document \ -H "X-API-Key: cf_test_YOUR_KEY" \ -H "Content-Type: application/json" \ -d '{ "url": "https://example.com/reports/q3-infrastructure.pdf", "extract_tables": true }'

Response Example

200 OK4,210ms

{
  "success": true,
  "data": {
    "url": "https://example.com/reports/q3-infrastructure.pdf",
    "document_type": "pdf",
    "content_type": "application/pdf",
    "file_size": 1842665,
    "processed_at": "2026-08-27T02:27:50.833Z",
    "page_count": 14,
    "text": "Q3 Infrastructure Review\n\nRequest volume grew 38% quarter over quarter while p95 latency held flat...",
    "text_length": 101,
    "metadata": {
      "title": "Q3 Infrastructure Review",
      "author": "Dana Reyes",
      "subject": "Quarterly capacity planning",
      "creator": "LaTeX with hyperref",
      "producer": "pdfTeX-1.40.25",
      "creation_date": "D:20260812141055Z",
      "modification_date": "D:20260814093012Z"
    },
    "tables": [
      {
        "page": 4,
        "rows": 3,
        "columns": 3,
        "data": [
          [
            "Region",
            "Requests",
            "p95 ms"
          ],
          [
            "us-east",
            "18,204,551",
            "412"
          ],
          [
            "eu-west",
            "9,118,340",
            "458"
          ]
        ]
      }
    ]
  },
  "credits_used": 2,
  "credits_remaining": 998,
  "processing_time": 4210
}

Field Descriptions

data.document_typeWhat the type detection settled on, which can differ from what you asked for

data.content_typeThe `Content-Type` header the server sent, verbatim — often wrong, which is why magic bytes are checked too

data.file_sizeBytes actually received

data.page_countPages in the document. PDFs only, and reported in full even when only the first 200 were read

data.text_lengthCharacters in `text` after any truncation, not the document's full length

data.metadata.creation_datePDF date strings are returned exactly as embedded, in the `D:YYYYMMDDHHmmSS` form. Parse them yourself

data.tables[].pageThe page the table was found on, so a result can be traced back to the source

data.tables[].columnsDerived from the vertical rules drawn in the PDF. A value of 1 means the table was not ruled into columns

credits_usedFlat 2 credits per document, whatever its page count

Error Handling

Invalid URL (400 Bad Request)

docx or xlsx (501 Not Implemented)

UNSUPPORTED_DOCUMENT_TYPE. Office formats are not parsed on the hosted REST API. The CrawlForge MCP server (npm: crawlforge-mcp-server) handles them with the same API key.

Document Too Large (413 Payload Too Large)

DOCUMENT_TOO_LARGE. Over the 25MB limit. Checked before the download and again after, so an inaccurate Content-Length still gets caught.

Document Unreachable (502 Bad Gateway)

DOCUMENT_FETCH_ERROR when the URL returned a non-2xx status — the message includes it — or FETCH_FAILED when the connection itself failed.

Target Timed Out (504 Gateway Timeout)

FETCH_TIMEOUT. The document did not arrive within the fetch budget.

Processing Failed (500 Internal Server Error)

TOOL_ERROR. A corrupt or encrypted PDF lands here. Nothing is charged for a failed call.

Blocked by robots.txt (403 Forbidden)

Credit Cost

2 credits

2 credits per document

Flat rate: a 200-page PDF costs the same as a one-page one, and enabling tables or metadata costs nothing extra. Failed calls are not charged.

What's Included:

Type detection from headers, URL and magic bytes

PDF text up to 200 pages and 200,000 characters

Embedded PDF metadata, including both timestamps

Up to 20 ruled tables with their page numbers

CSV parsing and plain-text and HTML extraction

Plan Recommendations:

Free Plan: 1,000 one-time trial credits = 500 documents

Hobby Plan: 5,000 credits = 2,500 documents ($19/mo)

Professional Plan: 50,000 credits = 25,000 documents ($99/mo)