CrawlForge
Beginner Guide

Credit Optimization Guide

Minimize scraping costs by choosing the right tools, implementing smart caching strategies, and optimizing your workflows for maximum value.

Tool Selection Strategy
Caching Strategies
Batch vs Individual
Cost/Benefit Analysis

Quick Wins (Save 50-80%)

Use fetch_url instead of search_web when you know the URL
Saves 4 credits per request (1 credit vs 5 credits)

💰 80% cost reduction

Try static scraping before browser automation
Use fetch_url (1 credit) before scrape_with_actions (5 credits)

💰 80% cost reduction for static content

Cache results locally
Store scraped data in Redis/database to avoid re-scraping same URLs

💰 90%+ reduction on repeated requests

Use batch_scrape for multiple URLs
Same cost (1 credit/URL) but faster and more efficient than individual requests

⚡ 5x faster throughput

1. Tool Selection Strategy

Always start with the cheapest tool that meets your needs, then upgrade only if necessary.

  • 1. Do you know the URL?
    • ✅ Yes → Use fetch_url (1 credit)
    • ❌ No → Use search_web (5 credits)
  • 2. Is the content in the initial HTML?
    • ✅ Yes → Use fetch_url (1 credit) + parse locally (free)
    • ❌ No (JavaScript-rendered) → Use scrape_with_actions (5 credits)
  • 3. Do you need structured extraction?
    • ✅ Yes → Use scrape_structured (2 credits)
    • ❌ No (raw HTML is fine) → Use fetch_url (1 credit)
  • 4. Do you need AI-powered research?
    • ✅ Yes → Use deep_research (10 credits)
    • ❌ No → Use cheaper alternatives

Cost Comparison by Use Case

Use CaseWrong ToolRight ToolSavings
Fetch HTMLsearch_web (5)fetch_url (1)80%
Extract textscrape_with_actions (5)extract_text (1)80%
Get metadatascrape_structured (2)extract_metadata (1)50%
Research topicdeep_research (10)search_web (5) + fetch_url (1×3)20%

Example: Extracting Product Data

Wrong (10 credits)

Typescript
// Using deep_research for simple task
const response = await fetch(
  'https://crawlforge.dev/api/v1/tools/deep_research',
  {
    method: 'POST',
    headers: {
      'X-API-Key': apiKey,
      'Content-Type': 'application/json'
    },
    body: JSON.stringify({
      topic: 'Product details',
      sources: ['https://shop.com/product/123']
    })
  }
);

// Cost: 10 credits 💸

Right (2 credits)

Typescript
// Using scrape_structured for known URL
const response = await fetch(
  'https://crawlforge.dev/api/v1/tools/scrape_structured',
  {
    method: 'POST',
    headers: {
      'X-API-Key': apiKey,
      'Content-Type': 'application/json'
    },
    body: JSON.stringify({
      url: 'https://shop.com/product/123',
      selectors: {
        title: 'h1.product-name',
        price: 'span.price'
      }
    })
  }
);

// Cost: 2 credits ✅ (80% savings)

2. Caching Strategies

Avoid re-scraping the same content by implementing smart caching.

Redis Caching Example

Cache results for 24 hours to eliminate duplicate requests

Typescript
import Redis from 'ioredis';

const redis = new Redis(process.env.REDIS_URL!);

async function fetchWithCache(url: string) {
  // Try cache first
  const cacheKey = `scrape:${url}`;
  const cached = await redis.get(cacheKey);

  if (cached) {
    console.log('Cache hit! Saved 1 credit ✅');
    return JSON.parse(cached);
  }

  // Cache miss - scrape the URL
  console.log('Cache miss - scraping...');
  const response = await fetch('https://crawlforge.dev/api/v1/tools/fetch_url', {
    method: 'POST',
    headers: {
      'X-API-Key': process.env.CRAWLFORGE_API_KEY!,
      'Content-Type': 'application/json',
    },
    body: JSON.stringify({ url }),
  });

  const data = await response.json();

  // Cache for 24 hours
  await redis.setex(cacheKey, 86400, JSON.stringify(data));

  return data;
}

// Usage
const data1 = await fetchWithCache('https://example.com'); // Uses 1 credit
const data2 = await fetchWithCache('https://example.com'); // Uses 0 credits (cached)
const data3 = await fetchWithCache('https://example.com'); // Uses 0 credits (cached)

// Savings: 2 credits (66% reduction)
Cache TTL Strategy: Static content (24h+), Product pages (6h), News (1h), Real-time data (5min or skip cache)

3. Batch vs Individual Requests

Use batch processing for multiple URLs to improve throughput and reduce overhead.

  • Individual Requests
    • ⏱️ Time: ~5 seconds per URL
    • 💰 Cost: 1 credit per URL
    • 📊 Throughput: 12 URLs/minute
    • Use for: <10 URLs
  • Batch Requests (Recommended)
    • ⏱️ Time: ~15 seconds for 50 URLs
    • 💰 Cost: 1 credit per URL
    • 📊 Throughput: 200 URLs/minute
    • ✅ Use for: 10+ URLs (16x faster!)

Code Comparison

Typescript
// Individual requests (SLOW)
const urls = [...]; // 50 URLs

const results = [];
for (const url of urls) {
  const response = await fetch('https://crawlforge.dev/api/v1/tools/fetch_url', {
    method: 'POST',
    headers: {
      'X-API-Key': process.env.CRAWLFORGE_API_KEY!,
      'Content-Type': 'application/json',
    },
    body: JSON.stringify({ url }),
  });

  const data = await response.json();
  results.push(data);
}

// Time: ~250 seconds (4+ minutes)
// Cost: 50 credits

4. Cost/Benefit Analysis

Calculate the ROI of your scraping operations.

Free Plan
$0

1,000 one-time trial credits — = 1,000 fetch_url requests

Hobby Plan
$19

5,000 credits — = $0.0038/credit

Professional
$99

50,000 credits — = $0.002/credit

Cost Per 1,000 URLs (Hobby Plan)

  • ✅ Using fetch_url (1 credit): $3.80
  • ⚠️ Using scrape_structured (2 credits): $7.60
  • ❌ Using scrape_with_actions (5 credits): $19.00
  • ❌ Using deep_research (10 credits): $38.00

Optimization Summary

  1. Always start with fetch_url
  2. Cache results for at least 1 hour
  3. Use batch_scrape for 10+ URLs
  4. Avoid deep_research for simple tasks
  5. Parse HTML locally when possible
  6. Use onlyMainContent: true for batch scraping
  7. Don't scrape same URL twice in 24h
  8. Check if site has API before scraping
  9. Use webhooks for large batches (100+ URLs)
  10. Monitor usage dashboard weekly
Ready to Optimize?
Start implementing these strategies to reduce costs by 50-80%
View API ReferenceView Pricing