extract_embedded_state
Modern sites serialize the data their UI renders straight into the document. This reads it back: one fetch, exact values, and no model anywhere in the extraction path.
Use Cases
Exact prices from JavaScript-rendered pages
Prices, stock levels and IDs come from the site's own state rather than from a model reading rendered text, so there is nothing to fabricate.
Listings a plain scrape cannot see
Search results and product grids that render client-side are usually already present in __NEXT_DATA__ or an RSC payload on the first response.
Cheaper than an LLM extraction
2 credits against 3 for extract_with_llm or extract_structured, with no inference step to wait for.
Auditing what a site publishes about itself
found reports every state source on the page and its size, which is often more than the visible UI shows.
Endpoint
/api/v1/tools/extract_embedded_stateParameters
| Name | Type | Required | Default | Description |
|---|---|---|---|---|
url | string | Required | - | Page to read embedded state from. Example: https://example.com/products/widget |
path | string | Optional | - | Return one subtree instead of the whole payload. Dotted keys and array indexes only - this is not JSONPath, so there are no wildcards, filters, slices or recursive descent. A path that does not resolve comes back as a 400 naming the keys that were available where it stopped, and costs no credits. Example: next_data.props.pageProps |
user_agent | string | Optional | - | Override the User-Agent sent to the target. The request stays signed as CrawlForge; the signature covers the authority, not this header. Example: MyCompanyBot/1.0 |
respect_robots | boolean | Optional | true | Respect the target site's robots.txt. Left at `true`, a path disallowed for `CrawlForge` is refused with 403 before anything is fetched and no credits are charged. Set it to `false` only for a target you have your own agreement with - the response then carries a `warnings` entry and the override is recorded against your API key. Example: true |
timeout | number | Optional | 20000 | Fetch timeout in milliseconds, between 1000 and 60000. State payloads are often megabytes, so the default is higher than on the lighter extraction tools. Example: 20000 |
max_inline_chars | number | Optional | 40000 | Inline size threshold in characters of the result's JSON (1,000-10,000,000). `extract_embedded_state` is never truncated — the full state always arrives inline — but over this size the response also carries `result_handle`, `total_chars` and `truncated: false`, so [read_result](/docs/api-reference/tools/read-result) can search the stored copy or read one `json_path` from it for 1 credit per call. Stored results are kept for 1 hour. Example: 40000 |
Request Examples
cURL - Read the whole state
curl -X POST https://crawlforge.dev/api/v1/tools/extract_embedded_state \
-H "X-API-Key: cf_test_YOUR_KEY" \
-H "Content-Type: application/json" \
-d '{
"url": "https://example.com/products/widget"
}'TypeScript - Scope it with a path
// npm install crawlforge-sdk
import { CrawlForge, ValidationError } from 'crawlforge-sdk';
const client = new CrawlForge({ apiKey: process.env.CRAWLFORGE_API_KEY });
try {
const result = await client.extractEmbeddedState({
url: 'https://example.com/products/widget',
// Dotted keys and array indexes. Not JSONPath: no wildcards or filters.
path: 'next_data.props.pageProps',
});
// result.data is untyped in crawlforge-sdk 0.1 — its shape is the Response Example below.
const { found, data } = result.data as {
found: { name: string; variable: string; bytes: number }[];
data: { product: { price: number } };
};
// Every state source on the page, largest first, whether or not it was scoped.
for (const source of found) {
console.log(source.name, source.variable, source.bytes);
}
// The values are the site's own, so they can be used as they are.
console.log(data.product.price);
} catch (err) {
// A path that does not resolve is a 400 naming the keys that were available.
if (err instanceof ValidationError) {
console.error(err.message);
} else {
throw err;
}
}Python - Discover, then scope
# pip install crawlforge
from crawlforge import CrawlForge
client = CrawlForge() # reads CRAWLFORGE_API_KEY
# 1. Discover what the page carries. Nothing is truncated, so this can be big.
discovery = client.extract_embedded_state(
url='https://example.com/products/widget',
)
# discovery.data is a plain dict — its shape is the Response Example below.
for source in discovery.data['found']:
print(source['name'], source['variable'], source['bytes'])
# This tool's warnings live inside data, next to the state itself.
for warning in discovery.data['warnings']:
print('warning:', warning)
# 2. Ask again for just the branch you want.
scoped = client.extract_embedded_state(
url='https://example.com/products/widget',
path='next_data.props.pageProps.product',
)
print(scoped.data['bytes'], 'bytes')
print(scoped.data['data'])Response Example
{ "success": true, "data": { "url": "https://example.com/products/widget", "found": [ { "name": "next_data", "variable": "__NEXT_DATA__", "bytes": 412880 } ], "path": null, "bytes": 412893, "data": { "next_data": { "buildId": "KfC_3GF1zuM", "props": { "pageProps": { "product": { "sku": "WID-9001", "price": 149.99, "currency": "USD", "inStock": true } } } } }, "warnings": [ "Result is 412893 bytes; \"next_data\" alone is 412880. Re-run with path to scope it, e.g. path:\"next_data.props\"." ] }, "credits_used": 2, "credits_remaining": 998, "processing_time": 980}data.foundEvery state source on the page, with the raw thing it was read from and its serialized sizedata.pathThe path that was applied, or null when the whole payload was returneddata.bytesSerialized size of what is being returned - after scoping, when a path was givendata.dataThe state itself, keyed by source namedata.warningsSources that were present but not parseable as JSON, and the size hint that suggests a pathError Handling
Path did not resolve (400 Bad Request)
The path does not exist in the extracted state. The message names the point it stopped at and the keys that were available there, so a typo comes back fixable. No credits are charged. Run the call once without a path to see what the page actually carries.
Blocked by robots.txt (403 Forbidden)
The target site's robots.txt disallows this path for CrawlForge. Set respect_robots: false to override if you have your own agreement with the target - the override is recorded against your API key. The override does not reach a host on CrawlForge's permanent opt-out list, which is refused whatever respect_robots is set to.
Response too large (413 Payload Too Large)
The page's HTML exceeds the 25MB body limit. The limit is on the page as served, not on the state extracted from it. Nothing is charged.
Target did not respond (504 Gateway Timeout)
The site did not answer within timeout. State-heavy pages are large; raise timeout toward its 60000 ceiling before treating this as a failure. Nothing is charged.