extract_embedded_state

Use Cases

Endpoint

Parameters

What it reads

Each source the page carries is listed in found and keyed in data by its name. A global that is not plain JSON is read as a JavaScript literal evaluated statically - objects, arrays, strings, numbers and Nuxt 2's single-return function - and no page code ever runs; anything that calls code or reads other variables is named in warnings instead.

• next_data - Next.js __NEXT_DATA__.
• next_f - React Server Component flight rows (self.__next_f), keyed by row id, with $<id> references between rows resolved in place (at most doubling the payload).
• data_rows - an index of the next_f rows that carry data rather than markup, largest first, each { id, bytes, keys, length }. Product and listing data usually lives in these rows; read one with path: "next_f.<id>".
• nuxt - Nuxt 2 __NUXT__; nuxt_data - Nuxt 3 __NUXT_DATA__, decoded from its index array into plain objects.
• sveltekit_data - SvelteKit's kit.start data, one entry per route node, root layout first.
• yt_initial_data and yt_initial_player_response - YouTube's ytInitialData and ytInitialPlayerResponse.
• apollo_state, apollo_ssr_transport, initial_state, preloaded_state, preloaded_data, remix_context (__remixContext), tgt_data (__TGT_DATA__), pws_data (__PWS_DATA__), server_data (__SERVER_DATA__), app_state (__APP_STATE__), state (__STATE__) and global_data (__data__) - Apollo, Redux-style and other window globals.
• inertia_page - Inertia's data-page JSON.
• shopify - ShopifyAnalytics.meta and Shopify's product JSON scripts. Meta prices are integers in minor units: 10500 is 105.00 in meta.currency.
• json_scripts - <script type="application/json"> and <script type="text/json"> blocks, each { id, data }.
• json_attributes - data-* attributes holding a JSON object of 2 KB or more, each { tag, attribute, id, data }.

Find a path

On a multi-megabyte payload, ask for the key you want instead of the data. find searches the whole payload, or only inside path when you send one, and returns where the key occurs with the head of each value.

curl -X POST https://crawlforge.dev/api/v1/tools/extract_embedded_state \
  -H "X-API-Key: cf_test_YOUR_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://www.youtube.com/watch?v=dQw4w9WgXcQ",
    "path": "yt_initial_player_response",
    "find": "viewCount"
  }'

Find response

matches replaces data. Each path already starts with the path you sent, so pass it straight back as path in the next call to get the full value. matches_total counts every match; matches_truncated is true when more than the 50 returned were found. A match marked addressable: false sits under a key containing . or [, which a path cannot spell.

{
  "success": true,
  "data": {
    "url": "https://www.youtube.com/watch?v=dQw4w9WgXcQ",
    "found": [
      { "name": "yt_initial_data", "variable": "ytInitialData", "bytes": 21239 },
      { "name": "yt_initial_player_response", "variable": "ytInitialPlayerResponse", "bytes": 8076 }
    ],
    "path": "yt_initial_player_response",
    "bytes": 8076,
    "matches": [
      {
        "path": "yt_initial_player_response.videoDetails.viewCount",
        "preview": "\"1822707973\""
      },
      {
        "path": "yt_initial_player_response.microformat.playerMicroformatRenderer.viewCount",
        "preview": "\"1822707973\""
      }
    ],
    "matches_total": 2,
    "matches_truncated": false,
    "warnings": []
  },
  "credits_used": 2,
  "credits_remaining": 996,
  "processing_time": 640
}

Request Examples

curl -X POST https://crawlforge.dev/api/v1/tools/extract_embedded_state \
  -H "X-API-Key: cf_test_YOUR_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://example.com/products/widget"
  }'

// npm install crawlforge-sdk
import { CrawlForge, ValidationError } from 'crawlforge-sdk';

const client = new CrawlForge({ apiKey: process.env.CRAWLFORGE_API_KEY });

try {
  const result = await client.extractEmbeddedState({
    url: 'https://example.com/products/widget',
    // Dotted keys and array indexes. Not JSONPath: no wildcards or filters.
    path: 'next_data.props.pageProps',
  });

  // result.data is untyped in crawlforge-sdk 0.1 — its shape is the Response Example below.
  const { found, data } = result.data as {
    found: { name: string; variable: string; bytes: number }[];
    data: { product: { price: number } };
  };

  // Every state source on the page, largest first, whether or not it was scoped.
  for (const source of found) {
    console.log(source.name, source.variable, source.bytes);
  }

  // The values are the site's own, so they can be used as they are.
  console.log(data.product.price);
} catch (err) {
  // A path that does not resolve is a 400 naming the keys that were available.
  if (err instanceof ValidationError) {
    console.error(err.message);
  } else {
    throw err;
  }
}

# pip install crawlforge
from crawlforge import CrawlForge

client = CrawlForge()  # reads CRAWLFORGE_API_KEY

# 1. Discover what the page carries. Nothing is truncated, so this can be big.
discovery = client.extract_embedded_state(
    url='https://example.com/products/widget',
)

# discovery.data is a plain dict — its shape is the Response Example below.
for source in discovery.data['found']:
    print(source['name'], source['variable'], source['bytes'])

# This tool's warnings live inside data, next to the state itself.
for warning in discovery.data['warnings']:
    print('warning:', warning)

# 2. Ask again for just the branch you want.
scoped = client.extract_embedded_state(
    url='https://example.com/products/widget',
    path='next_data.props.pageProps.product',
)

print(scoped.data['bytes'], 'bytes')
print(scoped.data['data'])

Response Example

{
  "success": true,
  "data": {
    "url": "https://example.com/products/widget",
    "found": [
      {
        "name": "next_data",
        "variable": "__NEXT_DATA__",
        "bytes": 125
      }
    ],
    "path": null,
    "bytes": 139,
    "data": {
      "next_data": {
        "buildId": "KfC_3GF1zuM",
        "props": {
          "pageProps": {
            "product": {
              "sku": "WID-9001",
              "price": 149.99,
              "currency": "USD",
              "inStock": true
            }
          }
        }
      }
    },
    "warnings": []
  },
  "credits_used": 2,
  "credits_remaining": 998,
  "processing_time": 980
}

Error Handling

Related Tools