Job Board APIs
Six applicant tracking systems publish a company's open roles through an API they document for public, unauthenticated use. scrape_template reads those APIs directly — exact values, no LLM in the path, and permitted by construction rather than by argument.
1. Why the platform's own API beats scraping the board
A company's careers page is a rendering of a database. The applicant tracking system behind it — Greenhouse, Lever, Ashby, Workable, Recruitee, Teamtailor — is the system of record the employer actually publishes to, and every one of those six documents a public endpoint that serves the same records as JSON or RSS.
Reading that endpoint is not a cheaper way to scrape. It is a different operation with different failure modes: there is no markup to misparse, no pagination widget to click through, and no language model deciding what a field meant.
The same board, two ways
| Scraping the rendered board | Reading the platform's API | |
|---|---|---|
| Values | Parsed out of markup; a redesign changes them silently | Exact, as the employer entered them |
| Coverage | Whatever the page renders on first paint | Every published posting on the board |
| Requests | One per page, plus paging | One request for the board, paged only where the platform pages |
| Interpretation | Selectors or an LLM decide what a field meant | Nothing infers; an absent field reads as null |
| Permission | Argued case by case | The platform documents the endpoint for public use |
2. The six job-board connectors
Each connector is pointed at an endpoint the platform itself documents for public, unauthenticated use, on a host whose robots.txt permits it. Pass a board URL and the connector resolves the API endpoint for you, or pass params with the company's board identifier.
greenhouse-jobscontent: true.Board token from job-boards.greenhouse.io/<token>
lever-postingsskip and limit paging.Company from jobs.lever.co/<company>
ashby-jobsdescriptions: true.Jobs page name from jobs.ashbyhq.com/<name>
workable-jobsdetails: true.Account subdomain from apply.workable.com/<subdomain>
recruitee-offersSubdomain from <company>.recruitee.com
teamtailor-jobstt: namespace intact so location, department, role and division survive. Returns 100 jobs unless per_page says otherwise.Subdomain from <company>.teamtailor.com
content: true adds the full HTML description and takes a large board from 349 KB to 4.2 MB (Stripe's 571-job board, measured 2026-08-28). Workable's details: true behaves the same way, and so does Ashby's descriptions: true (OpenAI's 767-job board is 5.9 MB with them, measured 2026-09-01). Ask for descriptions when you need them, not by habit.api.lever.co declares Crawl-delay: 1 in its robots.txt, so lever-postings republishes that as crawlDelaySeconds: 1 for the fetching surface to honour. The connectors never fetch anything themselves — they build the URL and parse the response, which keeps timeouts, SSRF policy and rate limiting with the surface that owns them.3. One job shape across six platforms
All six connectors normalise onto the same twelve fields, so two boards union without a per-source mapping step. A field the platform does not carry comes back null — never a plausible-looking guess.
The shared job shape
| Field | Meaning |
|---|---|
id | The platform's own identifier, stringified so a merged list has one id type |
title | Job title |
url | The public posting |
location | Location as the platform states it |
department | Department, where the platform has that level |
team | Team, where the platform has that level |
employment_type | Passed through in the platform's own words, not collapsed onto one vocabulary |
remote | true, false, or null when the source's word does not answer it |
published_at | ISO 8601, normalised from six different date formats |
updated_at | ISO 8601, where the platform publishes one |
description | Plain text, with the platform's HTML stripped |
source | Which connector produced the record |
- Nothing is inferred. Greenhouse publishes no employment type at all, so a Greenhouse job reports
nullthere rather than a confidentFull-time. - `remote` answers one question: can this job be done from anywhere? Hybrid is genuinely part-remote, so it reads as
nulland the platform's own word is kept alongside it. Reading a hybrid role as remote would be as wrong as reading it as on-site. - `employment_type` keeps the platform's spelling. Deciding what Lever's
Regular Full Time (Salary)really is belongs to you, who can see your own data. - Recruiter contacts are dropped, not mapped. Recruitee stamps a per-job application mailbox on every offer; that field never reaches the output. A job posting is company data, and that is all these connectors return.
Two boards, one list
Because the shape is shared, merging a Greenhouse board and a Lever board takes no mapping code.
// Every job-board connector returns the same twelve fields, so two boards
// concatenate with no per-source mapping step.
const API = 'https://crawlforge.dev/api/v1/tools/scrape_template';
async function board(template: string, params: Record<string, unknown>) {
const response = await fetch(API, {
method: 'POST',
headers: {
'X-API-Key': process.env.CRAWLFORGE_API_KEY!,
'Content-Type': 'application/json',
},
body: JSON.stringify({ template, params }),
});
const payload = await response.json();
return payload.data.data.items;
}
const jobs = [
...(await board('greenhouse-jobs', { company: 'stripe' })),
...(await board('lever-postings', { company: 'matterport' })),
];
// One filter over both boards. `remote` is null wherever the platform's own
// word did not answer the question — hybrid is not remote and not on-site.
const remote = jobs.filter((j) => j.remote === true);
// Cost: 1 credit per call — 2 credits for both boards, however many jobs.4. Calling it: params and auto
A job-board connector needs the company's identifier on that platform, which is a short string in the board URL rather than the whole URL. Pass it in params.
When you already have a URL and would rather not work out which template handles it, send template: "auto" and CrawlForge picks the template from the URL — deterministically, with a host-anchored pattern outranking one that only matches a path shape.
Both call styles
The same board, reached two ways.
# A job-board connector needs the company's identifier on that platform,
# not the whole careers-page URL. That is what `params` carries.
curl -X POST https://crawlforge.dev/api/v1/tools/scrape_template \
-H "X-API-Key: $CRAWLFORGE_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"template": "greenhouse-jobs",
"params": {
"company": "stripe"
}
}'
# Add "content": true to the params for full descriptions — it takes a large
# board past 4 MB, so ask for it only when you need the text.{ "template": "list" } with no URL to get every template with its id, description and mode — list for a connector that returns many records from one call, entity for one that returns a single record.5. The same idea, outside hiring
Job boards are the clearest case, but the principle is the rule everywhere: where an operator publishes its own data, read that.
shopify-collectionproducts.json — the same exact price, compare-at price and stock that shopify-product returns for one item, so a collection and a product page cannot disagree.nhtsa-vinnpi-provider6. What we do not do, and why
The list below is not a roadmap. Each entry is a route we considered, could technically take, and decided against for a stated reason.
Public, unauthenticated sources only
smartrecruiters-postingsapi.smartrecruiters.com/robots.txt disallows everything for every agent, with a single carve-out for LinkedInBot (verified 2026-08-28). Reaching that endpoint as anyone else means overriding robots.txt on every call, which is not a decision a connector gets to make on your behalf. Left out pending an agreement with SmartRecruiters.robots.txt respected — we did not override it
wday/cxs endpoint is the careers site's own internal endpoint, not an API Workday documents for public use. Any connector against it is gated on customer attestation, and none ships today.Not a documented public API
Not offered at any tier
npi-provider is a registry lookup for that reason: one record per NPI, with nothing joined to it.Company data, not people data
The rules these connectors are built to
- Public, unauthenticated pages and endpoints only.
- A documented API is preferred over scraping wherever one exists.
robots.txtis respected by default on every fetching tool. An override is explicit, per request, and recorded against your API key — and it never reaches a host on CrawlForge's permanent opt-out list.- CrawlForge identifies itself honestly: one user agent, the real product name, and a contact URL. See crawler verification.
- Polite rates by default, with
Crawl-delayandRetry-Afterhonoured. - Opt-outs and takedowns are honoured permanently, at the platform layer.
scrape_template costs 1 credit per call, whichever connector you use and however many jobs come back.