Langkau ke kandungan
CrawlForge MCP
Panduan Lanjutan

Teknik Scraping Lanjutan

Kuasai senario scraping kompleks termasuk kandungan dinamik, halaman yang dilindungi pengesahan, pemaparan JavaScript, dan pengendalian AJAX dengan CrawlForge MCP.

Kandungan Dinamik & JavaScript
Pengesahan & Sesi
AJAX & Tatal Tak Terhingga
Pengendalian Had Kadar

1. Kandungan Dinamik & JavaScript

Banyak tapak web moden memaparkan kandungan dengan JavaScript selepas muatan halaman awal. Gunakan scrape_with_actions untuk menunggu elemen dinamik.

Bila Menggunakan Automasi Pelayar
Single-Page Apps (SPAs): Aplikasi React, Vue, Angular yang memuatkan data secara tak segerak
Lazy Loading
Imej, video, atau kandungan yang dimuatkan semasa menatal
Elemen Interaktif
Dropdown, modal, atau tab yang mendedahkan kandungan
HTML Statik
Gunakan fetch_url sebaliknya (5x lebih murah)

Contoh: Scraping SPA React

5 credits

Bash
curl -X POST https://crawlforge.dev/api/v1/tools/scrape_with_actions \
  -H "X-API-Key: cf_test_YOUR_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://spa-example.com/products",
    "actions": [
      {
        "type": "wait",
        "selector": ".product-card",
        "timeout": 5000
      },
      {
        "type": "scroll",
        "direction": "down",
        "amount": 1000
      },
      {
        "type": "wait",
        "duration": 2000
      },
      {
        "type": "extract",
        "selectors": {
          "title": "h1.product-title",
          "price": "span.price",
          "description": "div.product-description"
        }
      }
    ]
  }'

2. Pengesahan & Sesi

Scrape halaman di sebalik borang log masuk atau pengesahan API menggunakan kuki, pengepala, atau penghantaran borang automatik.

Strategi 1: Pengesahan Kuki

Terbaik untuk tapak yang membolehkan anda memperoleh kuki sesi secara manual

Bash
curl -X POST https://crawlforge.dev/api/v1/tools/fetch_url \
  -H "X-API-Key: cf_test_YOUR_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://example.com/dashboard",
    "headers": {
      "Cookie": "session_id=abc123; user_token=xyz789"
    }
  }'

Strategi 2: Log Masuk Automatik dengan Borang

Automasikan keseluruhan proses log masuk dengan form_submit

Bash
curl -X POST https://crawlforge.dev/api/v1/tools/form_submit \
  -H "X-API-Key: cf_test_YOUR_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://example.com/login",
    "fields": {
      "email": "user@example.com",
      "password": "secure_password"
    },
    "submitButton": "button[type=submit]",
    "waitForNavigation": true
  }'

3. AJAX & Tatal Tak Terhingga

Tangkap kandungan yang dimuatkan semasa anda menatal atau mengklik butang "Load More".

Contoh Tatal Tak Terhingga

5 credits

Typescript
const response = await fetch('https://crawlforge.dev/api/v1/tools/scrape_with_actions', {
  method: 'POST',
  headers: {
    'X-API-Key': process.env.CRAWLFORGE_API_KEY!,
    'Content-Type': 'application/json',
  },
  body: JSON.stringify({
    url: 'https://social-media.com/feed',
    actions: [
      // Scroll to load more content (repeat 5 times)
      { type: 'scroll', direction: 'down', amount: 1000 },
      { type: 'wait', duration: 1000 },
      { type: 'scroll', direction: 'down', amount: 1000 },
      { type: 'wait', duration: 1000 },
      { type: 'scroll', direction: 'down', amount: 1000 },
      { type: 'wait', duration: 1000 },
      { type: 'scroll', direction: 'down', amount: 1000 },
      { type: 'wait', duration: 1000 },
      { type: 'scroll', direction: 'down', amount: 1000 },
      { type: 'wait', duration: 1000 },
      // Extract all posts
      {
        type: 'extract',
        selectors: {
          posts: {
            selector: 'article.post',
            multiple: true,
            fields: {
              text: '.post-content',
              author: '.author-name',
              timestamp: 'time'
            }
          }
        }
      }
    ]
  }),
});

const data = await response.json();
console.log(`Loaded ${data.data.extracted.posts.length} posts`);

4. Pengendalian Had Kadar

Laksanakan backoff eksponen dan logik cuba semula apabila menghadapi respons 429.

Contoh Logik Cuba Semula

Typescript
async function scrapeWithRetry(url: string, maxRetries = 3) {
  let retries = 0;
  let delay = 1000; // Start with 1 second

  while (retries < maxRetries) {
    try {
      const response = await fetch('https://crawlforge.dev/api/v1/tools/fetch_url', {
        method: 'POST',
        headers: {
          'X-API-Key': process.env.CRAWLFORGE_API_KEY!,
          'Content-Type': 'application/json',
        },
        body: JSON.stringify({ url }),
      });

      if (response.status === 429) {
        // Rate limited - wait and retry
        console.log(`Rate limited. Waiting ${delay}ms before retry...`);
        await new Promise(resolve => setTimeout(resolve, delay));
        delay *= 2; // Exponential backoff
        retries++;
        continue;
      }

      if (!response.ok) {
        throw new Error(`HTTP ${response.status}: ${response.statusText}`);
      }

      return await response.json();
    } catch (error) {
      if (retries === maxRetries - 1) throw error;
      retries++;
      await new Promise(resolve => setTimeout(resolve, delay));
      delay *= 2;
    }
  }

  throw new Error('Max retries exceeded');
}

// Usage
const data = await scrapeWithRetry('https://example.com');

Langkah Seterusnya

Teruskan perjalanan pembelajaran anda dengan lebih banyak panduan lanjutan

01Pemprosesan Kelompok →Skala ke beribu-ribu URL02Teknik Stealth →Pintas sistem anti-bot