Structured Tool2 credits

scrape_structured

Extract structured data from any webpage using custom CSS selectors. Perfect for e-commerce product scraping, news aggregation, and any custom data extraction needs.

Use Cases

E-Commerce Product Scraping

Extract product titles, prices, descriptions, and images from online stores

News Article Extraction

Extract headlines, authors, dates, and content from news sites

Custom Data Transformation

Map any HTML structure to your desired JSON schema

Real Estate Listings

Extract property details, prices, and images from listing sites

Endpoint

POST/api/v1/tools/scrape_structured

Auth Required

1 req/s on Free plan

2 credits

Parameters

Name	Type	Required	Default	Description
url	string	Optional	-	Page to fetch and extract from. Either `url` or `html` is required. Example: https://example.com/product
html	string	Optional	-	Raw HTML to parse instead of fetching. Pair with `base_url` to resolve relative attribute values. Example: <html>...</html>
selectors	object	Required	-	Maps your field names to CSS selectors. At least one entry is required — the field names are yours and come back unchanged in the result. Example: {"title": "h1.product-title", "price": ".price", "description": ".product-desc"}
base_url	string	Optional	-	Base for resolving relative URLs found in extracted attributes. Example: https://example.com
multiple	boolean	Optional	false	When true, extract a list of repeating items instead of a single record. Changes the response shape — see below. Example: true
clean_text	boolean	Optional	true	Collapse whitespace and trim the extracted text. Example: true
include_attributes	array	Optional	-	Also capture these HTML attributes from each matched element, not just its text. Example: ["href", "src", "alt"]
max_items	number	Optional	100	Maximum items returned when `multiple` is true, 1-1000. Example: 100
respect_robots	boolean	Optional	true	Respect the target site's robots.txt. Left at `true`, a path disallowed for `CrawlForge` is refused with 403 before anything is fetched and no credits are charged. Set it to `false` only for a target you have your own agreement with — the response then carries a `warnings` entry and the override is recorded against your API key. Example: true

CSS Selectors:

Use any valid CSS selector syntax. Common patterns:

.className - Select by class
#id - Select by ID
tag.class - Combine tag and class
.parent > .child - Direct child
[data-id="value"] - Attribute selector

Request Examples

cURL - E-Commerce Product

terminalBash

curl -X POST https://crawlforge.dev/api/v1/tools/scrape_structured \
  -H "X-API-Key: cf_test_YOUR_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://example.com/product/123",
    "selectors": {
      "title": "h1.product-title",
      "price": ".price-value",
      "currency": ".price-currency",
      "description": ".product-description",
      "image": "img.main-image",
      "rating": ".rating-value",
      "availability": ".stock-status"
    }
  }'

TypeScript - News Article

scrapeStructured.tsTypescript

const response = await fetch('https://crawlforge.dev/api/v1/tools/scrape_structured', {
  method: 'POST',
  headers: {
    'X-API-Key': process.env.CRAWLFORGE_API_KEY!,
    'Content-Type': 'application/json',
  },
  body: JSON.stringify({
    url: 'https://example.com/news/article-123',
    selectors: {
      headline: 'h1.article-title',
      author: '.author-name',
      publishDate: 'time.publish-date',
      category: '.category-tag',
      content: '.article-body',
      image: '.article-image img'
    }
  }),
});

const data = await response.json();

if (data.success) {
  const article = data.data;
  console.log(`Article: ${article.headline}`);
  console.log(`By: ${article.author}`);
  console.log(`Published: ${article.publishDate}`);
}

Python - Real Estate Listing

scrape_structured.pyPython

import requests
import os

response = requests.post(
    'https://crawlforge.dev/api/v1/tools/scrape_structured',
    headers={
        'X-API-Key': os.environ['CRAWLFORGE_API_KEY'],
        'Content-Type': 'application/json',
    },
    json={
        'url': 'https://example.com/property/456',
        'selectors': {
            'address': '.property-address',
            'price': '.listing-price',
            'bedrooms': '.bed-count',
            'bathrooms': '.bath-count',
            'sqft': '.square-feet',
            'description': '.property-description',
            'images': '.gallery img'
        }
    }
)

data = response.json()

if data['success']:
    property_data = data['data']
    print(f"Property: {property_data['address']}")
    print(f"Price: {property_data['price']}")
    print(f"Beds: {property_data['bedrooms']}")
    print(f"Baths: {property_data['bathrooms']}")

Response Example

200 OK320ms

{
  "success": true,
  "data": {
    "title": "Premium Wireless Headphones",
    "price": "299.99",
    "currency": "USD",
    "description": "High-quality wireless headphones with active noise cancellation and 30-hour battery life.",
    "image": "https://example.com/images/headphones.jpg",
    "rating": "4.7",
    "availability": "In Stock"
  },
  "credits_used": 2,
  "credits_remaining": 998,
  "processing_time": 320
}

Field Descriptions

data.titleExtracted from h1.product-title selector

data.priceExtracted from .price-value selector

data.descriptionExtracted from .product-description selector

credits_usedCredits deducted for this request (2 per scrape)

Error Handling

Blocked by robots.txt (403 Forbidden)

The target site's robots.txt disallows this path for CrawlForge. Set respect_robots: false to override if you have your own agreement with the target — the override is recorded against your API key. The override does not reach a host on CrawlForge's permanent opt-out list, which is refused whatever respect_robots is set to.

Credit Cost