extract_embedded_state

Kes Penggunaan

Endpoint

Parameters

Apa yang dibacanya

Setiap sumber yang dibawa halaman disenaraikan dalam found dan dikunci dalam data mengikut namanya. Pemboleh ubah global yang bukan JSON biasa dibaca sebagai literal JavaScript yang dinilai secara statik - objek, tatasusunan, rentetan, nombor dan fungsi satu-return Nuxt 2 - dan tiada kod halaman yang pernah dijalankan; apa-apa yang memanggil kod atau membaca pemboleh ubah lain dinamakan dalam warnings.

• next_data - __NEXT_DATA__ Next.js.
• next_f - baris flight React Server Component (self.__next_f), dikunci mengikut id baris, dengan rujukan $<id> antara baris diselesaikan di tempatnya (paling banyak menggandakan muatan).
• data_rows - indeks baris next_f yang membawa data dan bukan markup, terbesar dahulu, setiap satu { id, bytes, keys, length }. Data produk dan senarai biasanya terdapat dalam baris ini; baca satu dengan path: "next_f.<id>".
• nuxt - __NUXT__ Nuxt 2; nuxt_data - __NUXT_DATA__ Nuxt 3, dinyahkod daripada tatasusunan indeksnya kepada objek biasa.
• sveltekit_data - data kit.start SvelteKit, satu entri bagi setiap nod laluan, susun atur akar dahulu.
• yt_initial_data dan yt_initial_player_response - ytInitialData dan ytInitialPlayerResponse YouTube.
• apollo_state, apollo_ssr_transport, initial_state, preloaded_state, preloaded_data, remix_context (__remixContext), tgt_data (__TGT_DATA__), pws_data (__PWS_DATA__), server_data (__SERVER_DATA__), app_state (__APP_STATE__), state (__STATE__) dan global_data (__data__) - pemboleh ubah global window Apollo, gaya Redux dan lain-lain.
• inertia_page - JSON data-page Inertia.
• shopify - ShopifyAnalytics.meta dan skrip JSON produk Shopify. Harga meta ialah integer dalam unit kecil: 10500 ialah 105.00 dalam meta.currency.
• json_scripts - blok <script type="application/json"> dan <script type="text/json">, setiap satu { id, data }.
• json_attributes - atribut data-* yang memegang objek JSON bersaiz 2 KB atau lebih, setiap satu { tag, attribute, id, data }.

Cari laluan

Pada muatan berbilang megabait, minta kunci yang anda mahu dan bukan datanya. find mencari seluruh muatan, atau hanya di dalam path apabila anda menghantarnya, dan memulangkan tempat kunci itu muncul bersama permulaan setiap nilai.

curl -X POST https://crawlforge.dev/api/v1/tools/extract_embedded_state \
  -H "X-API-Key: cf_test_YOUR_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://www.youtube.com/watch?v=dQw4w9WgXcQ",
    "path": "yt_initial_player_response",
    "find": "viewCount"
  }'

Respons find

matches menggantikan data. Setiap path sudah bermula dengan path yang anda hantar, jadi hantar semula terus sebagai path dalam panggilan seterusnya untuk mendapatkan nilai penuh. matches_total mengira setiap padanan; matches_truncated ialah true apabila lebih daripada 50 yang dipulangkan ditemui. Padanan yang ditanda addressable: false berada di bawah kunci yang mengandungi . atau [, yang tidak dapat dieja oleh laluan.

{
  "success": true,
  "data": {
    "url": "https://www.youtube.com/watch?v=dQw4w9WgXcQ",
    "found": [
      { "name": "yt_initial_data", "variable": "ytInitialData", "bytes": 21239 },
      { "name": "yt_initial_player_response", "variable": "ytInitialPlayerResponse", "bytes": 8076 }
    ],
    "path": "yt_initial_player_response",
    "bytes": 8076,
    "matches": [
      {
        "path": "yt_initial_player_response.videoDetails.viewCount",
        "preview": "\"1822707973\""
      },
      {
        "path": "yt_initial_player_response.microformat.playerMicroformatRenderer.viewCount",
        "preview": "\"1822707973\""
      }
    ],
    "matches_total": 2,
    "matches_truncated": false,
    "warnings": []
  },
  "credits_used": 2,
  "credits_remaining": 996,
  "processing_time": 640
}

Contoh Permintaan

curl -X POST https://crawlforge.dev/api/v1/tools/extract_embedded_state \
  -H "X-API-Key: cf_test_YOUR_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://example.com/products/widget"
  }'

// npm install crawlforge-sdk
import { CrawlForge, ValidationError } from 'crawlforge-sdk';

const client = new CrawlForge({ apiKey: process.env.CRAWLFORGE_API_KEY });

try {
  const result = await client.extractEmbeddedState({
    url: 'https://example.com/products/widget',
    // Dotted keys and array indexes. Not JSONPath: no wildcards or filters.
    path: 'next_data.props.pageProps',
  });

  // result.data is untyped in crawlforge-sdk 0.1 — its shape is the Response Example below.
  const { found, data } = result.data as {
    found: { name: string; variable: string; bytes: number }[];
    data: { product: { price: number } };
  };

  // Every state source on the page, largest first, whether or not it was scoped.
  for (const source of found) {
    console.log(source.name, source.variable, source.bytes);
  }

  // The values are the site's own, so they can be used as they are.
  console.log(data.product.price);
} catch (err) {
  // A path that does not resolve is a 400 naming the keys that were available.
  if (err instanceof ValidationError) {
    console.error(err.message);
  } else {
    throw err;
  }
}

# pip install crawlforge
from crawlforge import CrawlForge

client = CrawlForge()  # reads CRAWLFORGE_API_KEY

# 1. Discover what the page carries. Nothing is truncated, so this can be big.
discovery = client.extract_embedded_state(
    url='https://example.com/products/widget',
)

# discovery.data is a plain dict — its shape is the Response Example below.
for source in discovery.data['found']:
    print(source['name'], source['variable'], source['bytes'])

# This tool's warnings live inside data, next to the state itself.
for warning in discovery.data['warnings']:
    print('warning:', warning)

# 2. Ask again for just the branch you want.
scoped = client.extract_embedded_state(
    url='https://example.com/products/widget',
    path='next_data.props.pageProps.product',
)

print(scoped.data['bytes'], 'bytes')
print(scoped.data['data'])

Contoh Respons

{
  "success": true,
  "data": {
    "url": "https://example.com/products/widget",
    "found": [
      {
        "name": "next_data",
        "variable": "__NEXT_DATA__",
        "bytes": 125
      }
    ],
    "path": null,
    "bytes": 139,
    "data": {
      "next_data": {
        "buildId": "KfC_3GF1zuM",
        "props": {
          "pageProps": {
            "product": {
              "sku": "WID-9001",
              "price": 149.99,
              "currency": "USD",
              "inStock": true
            }
          }
        }
      }
    },
    "warnings": []
  },
  "credits_used": 2,
  "credits_remaining": 998,
  "processing_time": 980
}

Pengendalian Ralat

Alat Berkaitan