scrape_template

20 pre-built extractors: popular sites, six ATS job boards read through the platform's own documented public API, and keyless government APIs. No schema, no selectors, no LLM — pass a template name with a URL or params. 1 credit per call, however many records come back.

Use Cases

Quick E-Commerce Data

Pull Amazon product price, rating, and stock without writing a single selector.

Developer Profile Aggregation

Combine GitHub repos, npm packages, and Stack Overflow activity into one developer view.

Social Media Monitoring

Track Product Hunt launches and Hacker News with a single endpoint per source; for Reddit, read a post with reddit-thread and its comment tree with reddit_search.

Whole Job Boards in One Call

Read every published role from a company's Greenhouse, Lever, Ashby, Workable, Recruitee or Teamtailor board — exact values from the platform's own API, not parsed out of a careers page.

Endpoint

POST/api/v1/tools/scrape_template

Auth Required

1 req/s on Free plan

1 credit

Parameters

Send a `url` or `params`, not neither. list needs neither and auto always needs a url. A page template needs a url; a platform-API connector takes params, or a board URL it resolves to the API endpoint itself.

Name	Type	Required	Default	Description
template	string	Required	-	Template name — one of the 20 supported templates. Send `"auto"` to have the template picked from the `url`; the response reports the id it chose, never the literal `"auto"`. Send `"list"` to discover them programmatically. Example: github-repo
url	string	Optional	-	URL to scrape. Required unless `template` is `"list"`, or you are calling a connector by `params` instead. Example: https://github.com/mysleekdesigns/crawlforge-mcp
params	object	Optional	-	Inputs for a connector that reads a platform API rather than a page — the board identifier for a job board (`company`), `store` and `collection` for `shopify-collection`, `vin` for `nhtsa-vin`, the registry's own search fields for `npi-provider`. Connector-specific extras go here too: Greenhouse `content: true`, Workable `details: true`, Lever `skip`/`limit`, Teamtailor `per_page`/`offset`. A missing required input is refused with the parameter named. Example: { "company": "stripe" }
timeout	number	Optional	15000	Request timeout in milliseconds (5000–60000). Example: 15000
respect_robots	boolean	Optional	true	Respect the target site's robots.txt. Left at `true`, a path disallowed for `CrawlForge` is refused with 403 before anything is fetched and no credits are charged. Set it to `false` only for a target you have your own agreement with — the response then carries a `warnings` entry and the override is recorded against your API key. Example: true

Templates: pages and products

20 templates in three groups. Each returns a normalized JSON shape tuned for its source.

amazon-product

Product title, price, rating, reviews, stock status

shopify-product

Exact price, compare-at price, per-variant stock and options — read from the store's own JSON, so prices cannot be misread or invented. Works on any Shopify storefront, including custom domains. When a store refuses its JSON endpoint (401, 403, 404 or 410), the record is read from the product page's own schema.org JSON-LD instead, marked source: json-ld with a warning; per-variant stock, compare-at prices and option names are not in JSON-LD, so those fields are null

shopify-collection

Every product in a collection from the store's own products.json, with the same authoritative price and stock shopify-product returns for one item. 30 per page by default, 250 at most

github-repo

Stars, forks, language, last commit, description

youtube-video

Title, channel, views, likes, duration, publish date

reddit-thread

The post record from the Arctic Shift archive — id, title, subreddit, author, score, upvote ratio, comment count, body, flair, removal state (reddit.com blocks direct scraping). Pass the id to reddit_search with mode "thread" for the comment tree

hacker-news-front-page

Templates: job boards

Six applicant tracking systems, each read through the endpoint that platform documents for public, unauthenticated use. All six normalize onto one job shape, so two boards union without per-source mapping — the job board APIs guide covers the shape and the reasoning.

greenhouse-jobs

The Job Board API. Every published job in one request; descriptions opt-in via content: true

lever-postings

The Postings API, with team, commitment and workplace type separated out. skip/limit paging

ashby-jobs

The Public Job Posting API. Department, team, employment and workplace type; descriptions opt-in via descriptions: true

workable-jobs

The public accounts endpoint. Location parts and a telecommuting flag; descriptions opt-in via details: true

recruitee-offers

The Careers Site API. Department, location, employment type code and salary band

teamtailor-jobs

The documented careers-site RSS feed, read with its tt: namespace intact. 100 jobs unless per_page says otherwise

Templates: government APIs

Free, keyless US federal APIs, documented by the agencies that publish them.

nhtsa-vin

VIN decode through NHTSA's vPIC API — make, model, year, trim, body, engine, plant, with the full field set under raw. Partial VINs accepted; vPIC's own error codes are surfaced rather than swallowed

npi-provider

The CMS NPPES registry of US health care providers, searched by number, name, taxonomy or location. One record per NPI, with nothing joined to it

List connectors return many records from one call

A connector that reads a listing endpoint returns items and count under data instead of one record. shopify-collection and all six job-board connectors work this way.

Every job record carries the same twelve fields — id, title, url, location, department, team, employment_type, remote, published_at, updated_at, description, source — and a field the platform does not publish is null, never a guess.
Pass the board identifier in params, or pass a board URL and the connector resolves the API endpoint itself.
Greenhouse, Workable and Ashby make descriptions opt-in because they dominate the payload: a large Greenhouse board goes from 349 KB to 4.2 MB with content: true, and OpenAI's 767-job Ashby board is 5.9 MB with descriptions: true.
Cost is a flat 1 credit per call, however many records come back.

Pass { "template": "list" } (no url required) to fetch the full list with each entry's mode — list for a connector that returns many records from one call, entity for one that returns a single record. Useful for keeping client SDKs in sync.

Deliberately not shipped

smartrecruiters-postings is absent: SmartRecruiters documents its Posting API publicly, but api.smartrecruiters.com/robots.txt disallows everything for every agent except LinkedInBot (verified 2026-08-28), and we did not override robots.txt to reach it. Workday's wday/cxs endpoint is a careers site's own internal endpoint rather than a documented public API, so no connector reads it either. The job board APIs guide sets out what else is out of scope, and why.

Request Examples

cURL — github-repo

terminalBash

curl -X POST https://crawlforge.dev/api/v1/tools/scrape_template \
  -H "X-API-Key: cf_test_YOUR_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "template": "github-repo",
    "url": "https://github.com/mysleekdesigns/crawlforge-mcp"
  }'

TypeScript — amazon-product

scrapeTemplate.tsTypescript

const response = await fetch('https://crawlforge.dev/api/v1/tools/scrape_template', {
  method: 'POST',
  headers: {
    'X-API-Key': process.env.CRAWLFORGE_API_KEY!,
    'Content-Type': 'application/json',
  },
  body: JSON.stringify({
    template: 'amazon-product',
    url: 'https://www.amazon.com/dp/B08XYZ1234',
    timeout: 20000,
  }),
});

const data = await response.json();
if (data.success) {
  const product = data.data.extracted;
  console.log(product.title, product.price, product.rating);
}

cURL — discover templates

terminalBash

curl -X POST https://crawlforge.dev/api/v1/tools/scrape_template \
  -H "X-API-Key: cf_test_YOUR_KEY" \
  -H "Content-Type: application/json" \
  -d '{ "template": "list" }'

cURL — greenhouse-jobs by params

terminalBash

curl -X POST https://crawlforge.dev/api/v1/tools/scrape_template \
  -H "X-API-Key: cf_test_YOUR_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "template": "greenhouse-jobs",
    "params": { "company": "stripe" }
  }'

Response Example

200 OK0.8s

{
  "success": true,
  "data": {
    "template": "greenhouse-jobs",
    "params": {
      "company": "stripe"
    },
    "data": {
      "items": [
        {
          "id": "4012345",
          "title": "Staff Software Engineer",
          "url": "https://job-boards.greenhouse.io/stripe/jobs/4012345",
          "location": "Seattle, Washington, United States",
          "department": null,
          "team": null,
          "employment_type": null,
          "remote": null,
          "published_at": "2026-08-04T17:12:03.000Z",
          "updated_at": "2026-08-21T09:44:51.000Z",
          "description": null,
          "source": "greenhouse-jobs",
          "raw_extra": {
            "internal_job_id": "4098765",
            "requisition_id": "REQ-1234",
            "offices": null
          }
        }
      ],
      "count": 571,
      "company": "Stripe"
    },
    "template_name": "Greenhouse Job Board",
    "extracted_at": "2026-08-28T12:00:00.000Z",
    "fetched_url": "https://boards-api.greenhouse.io/v1/boards/stripe/jobs"
  },
  "credits_used": 1,
  "credits_remaining": 999,
  "processing_time": 812
}

Field Descriptions

data.data.itemsOne record per published job, in the twelve-field shape every job-board connector shares

data.data.items[0].departmentGreenhouse ships departments only with `content: true`, so a summary record reports null rather than a department guessed from the title

data.data.countRecords in this response; a source that declares a larger total also reports `total_available`

data.template_nameHuman-readable name of the template that ran

data.fetched_urlThe endpoint actually read — present whenever it differs from what you passed, which includes every params-only call

credits_usedFlat 1 credit per call, regardless of template or record count

Error Handling

Connector could not build a request (400 Bad Request)

A parameter problem, refused before anything is fetched and never billed. INVALID_TEMPLATE_PARAMS carries the connector's own message naming the parameter it wanted — the board token in job-boards.greenhouse.io/<token>, the subdomain in <company>.recruitee.com. MISSING_URL means you sent params to a template that reads a page and has no params form. NO_TEMPLATE_MATCH means template: "auto" recognised nothing in the URL — name the template yourself, or check it against template: "list".

Blocked by robots.txt (403 Forbidden)

The target site's robots.txt disallows this path for CrawlForge. Set respect_robots: false to override if you have your own agreement with the target — the override is recorded against your API key. The override does not reach a host on CrawlForge's permanent opt-out list, which is refused whatever respect_robots is set to. The check runs against the URL the template actually fetches, which for shopify-product and npm-package is a machine-readable endpoint on the same host rather than the URL you passed.

Credit Cost

1 credit

1 credit per request

Flat 1 credit per call. The cheapest way to pull structured data from a supported site.

Tip: Need a site that isn't supported? Use scrape_structured for custom selectors (2 credits) or extract_with_llm for LLM-powered extraction.

Related Tools

scrape_structured

Custom CSS selector extraction (2 credits)

extract_with_llm

LLM-powered extraction for any site

reddit_search

Reddit posts, comments, and threads via community archives (5 credits)

Ready to scrape popular sites in one call? Sign up for free and get 1,000 credits.

Extraction1 credit

scrape_template

Use Cases

Quick E-Commerce Data

Pull Amazon product price, rating, and stock without writing a single selector.

Developer Profile Aggregation

Combine GitHub repos, npm packages, and Stack Overflow activity into one developer view.

Social Media Monitoring

Track Product Hunt launches and Hacker News with a single endpoint per source; for Reddit, read a post with reddit-thread and its comment tree with reddit_search.

Whole Job Boards in One Call

Read every published role from a company's Greenhouse, Lever, Ashby, Workable, Recruitee or Teamtailor board — exact values from the platform's own API, not parsed out of a careers page.

Endpoint

POST/api/v1/tools/scrape_template

Auth Required

1 req/s on Free plan

1 credit

Parameters

Name	Type	Required	Default	Description
template	string	Required	-	Template name — one of the 20 supported templates. Send `"auto"` to have the template picked from the `url`; the response reports the id it chose, never the literal `"auto"`. Send `"list"` to discover them programmatically. Example: github-repo
url	string	Optional	-	URL to scrape. Required unless `template` is `"list"`, or you are calling a connector by `params` instead. Example: https://github.com/mysleekdesigns/crawlforge-mcp
params	object	Optional	-	Inputs for a connector that reads a platform API rather than a page — the board identifier for a job board (`company`), `store` and `collection` for `shopify-collection`, `vin` for `nhtsa-vin`, the registry's own search fields for `npi-provider`. Connector-specific extras go here too: Greenhouse `content: true`, Workable `details: true`, Lever `skip`/`limit`, Teamtailor `per_page`/`offset`. A missing required input is refused with the parameter named. Example: { "company": "stripe" }
timeout	number	Optional	15000	Request timeout in milliseconds (5000–60000). Example: 15000
respect_robots	boolean	Optional	true	Respect the target site's robots.txt. Left at `true`, a path disallowed for `CrawlForge` is refused with 403 before anything is fetched and no credits are charged. Set it to `false` only for a target you have your own agreement with — the response then carries a `warnings` entry and the override is recorded against your API key. Example: true

Templates: pages and products

20 templates in three groups. Each returns a normalized JSON shape tuned for its source.

amazon-product

Product title, price, rating, reviews, stock status

shopify-product

shopify-collection

Every product in a collection from the store's own products.json, with the same authoritative price and stock shopify-product returns for one item. 30 per page by default, 250 at most

github-repo

Stars, forks, language, last commit, description

youtube-video

Title, channel, views, likes, duration, publish date

reddit-thread

hacker-news-front-page

Templates: job boards

greenhouse-jobs

The Job Board API. Every published job in one request; descriptions opt-in via content: true

lever-postings

The Postings API, with team, commitment and workplace type separated out. skip/limit paging

ashby-jobs

The Public Job Posting API. Department, team, employment and workplace type; descriptions opt-in via descriptions: true

workable-jobs

The public accounts endpoint. Location parts and a telecommuting flag; descriptions opt-in via details: true

recruitee-offers

The Careers Site API. Department, location, employment type code and salary band

teamtailor-jobs

The documented careers-site RSS feed, read with its tt: namespace intact. 100 jobs unless per_page says otherwise

Templates: government APIs

Free, keyless US federal APIs, documented by the agencies that publish them.

nhtsa-vin

npi-provider

The CMS NPPES registry of US health care providers, searched by number, name, taxonomy or location. One record per NPI, with nothing joined to it

List connectors return many records from one call

A connector that reads a listing endpoint returns items and count under data instead of one record. shopify-collection and all six job-board connectors work this way.

Every job record carries the same twelve fields — id, title, url, location, department, team, employment_type, remote, published_at, updated_at, description, source — and a field the platform does not publish is null, never a guess.
Pass the board identifier in params, or pass a board URL and the connector resolves the API endpoint itself.
Greenhouse, Workable and Ashby make descriptions opt-in because they dominate the payload: a large Greenhouse board goes from 349 KB to 4.2 MB with content: true, and OpenAI's 767-job Ashby board is 5.9 MB with descriptions: true.
Cost is a flat 1 credit per call, however many records come back.

Deliberately not shipped

Request Examples

cURL — github-repo

terminalBash

curl -X POST https://crawlforge.dev/api/v1/tools/scrape_template \
  -H "X-API-Key: cf_test_YOUR_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "template": "github-repo",
    "url": "https://github.com/mysleekdesigns/crawlforge-mcp"
  }'

TypeScript — amazon-product

scrapeTemplate.tsTypescript

const response = await fetch('https://crawlforge.dev/api/v1/tools/scrape_template', {
  method: 'POST',
  headers: {
    'X-API-Key': process.env.CRAWLFORGE_API_KEY!,
    'Content-Type': 'application/json',
  },
  body: JSON.stringify({
    template: 'amazon-product',
    url: 'https://www.amazon.com/dp/B08XYZ1234',
    timeout: 20000,
  }),
});

const data = await response.json();
if (data.success) {
  const product = data.data.extracted;
  console.log(product.title, product.price, product.rating);
}

cURL — discover templates

terminalBash

curl -X POST https://crawlforge.dev/api/v1/tools/scrape_template \
  -H "X-API-Key: cf_test_YOUR_KEY" \
  -H "Content-Type: application/json" \
  -d '{ "template": "list" }'

cURL — greenhouse-jobs by params

terminalBash

curl -X POST https://crawlforge.dev/api/v1/tools/scrape_template \
  -H "X-API-Key: cf_test_YOUR_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "template": "greenhouse-jobs",
    "params": { "company": "stripe" }
  }'

Response Example

200 OK0.8s

{
  "success": true,
  "data": {
    "template": "greenhouse-jobs",
    "params": {
      "company": "stripe"
    },
    "data": {
      "items": [
        {
          "id": "4012345",
          "title": "Staff Software Engineer",
          "url": "https://job-boards.greenhouse.io/stripe/jobs/4012345",
          "location": "Seattle, Washington, United States",
          "department": null,
          "team": null,
          "employment_type": null,
          "remote": null,
          "published_at": "2026-08-04T17:12:03.000Z",
          "updated_at": "2026-08-21T09:44:51.000Z",
          "description": null,
          "source": "greenhouse-jobs",
          "raw_extra": {
            "internal_job_id": "4098765",
            "requisition_id": "REQ-1234",
            "offices": null
          }
        }
      ],
      "count": 571,
      "company": "Stripe"
    },
    "template_name": "Greenhouse Job Board",
    "extracted_at": "2026-08-28T12:00:00.000Z",
    "fetched_url": "https://boards-api.greenhouse.io/v1/boards/stripe/jobs"
  },
  "credits_used": 1,
  "credits_remaining": 999,
  "processing_time": 812
}

Field Descriptions

data.data.itemsOne record per published job, in the twelve-field shape every job-board connector shares

data.data.items[0].departmentGreenhouse ships departments only with `content: true`, so a summary record reports null rather than a department guessed from the title

data.data.countRecords in this response; a source that declares a larger total also reports `total_available`

data.template_nameHuman-readable name of the template that ran

data.fetched_urlThe endpoint actually read — present whenever it differs from what you passed, which includes every params-only call

credits_usedFlat 1 credit per call, regardless of template or record count

Error Handling

Connector could not build a request (400 Bad Request)

Blocked by robots.txt (403 Forbidden)

Credit Cost

1 credit

1 credit per request

Flat 1 credit per call. The cheapest way to pull structured data from a supported site.

Tip: Need a site that isn't supported? Use scrape_structured for custom selectors (2 credits) or extract_with_llm for LLM-powered extraction.

Related Tools

scrape_structured

Custom CSS selector extraction (2 credits)

extract_with_llm

LLM-powered extraction for any site

reddit_search

Reddit posts, comments, and threads via community archives (5 credits)

Ready to scrape popular sites in one call? Sign up for free and get 1,000 credits.