Job Listing Aggregation

Collect postings from job boards that paginate, lazy-load, or sit behind a login. Drive a real browser first, then map every board layout onto one JSON schema.

Quick Answer

Use scrape_with_actions (5 credits) to click "load more", scroll, or sign in so the listings actually render, then extract_structured (3 credits) to map each posting onto your own JSON schema -- title, location, salary, remote flag. About 8 credits per board page, and one schema covers boards with different markup.

The Problem

Job boards hide their listings behind "load more" buttons, infinite scroll, and login walls, so a plain HTTP fetch returns an empty shell. Every board also names its fields differently, which breaks selector-based scrapers on the next redesign.

The Solution

CrawlForge scrape_with_actions drives a real browser -- click, type, scroll, wait -- so the listings render before extraction, and extract_structured maps whatever markup a board uses onto one JSON schema you define.

Code Example

// Load a paginated board, then extract each posting as JSON
const board = await mcp.scrape_with_actions({
  url: "https://boards.example.com/jobs?team=engineering",
  actions: [
    { type: "wait", selector: ".job-card", condition: "visible" },
    { type: "click", selector: "button.load-more", clickCount: 3 },
    { type: "scroll", direction: "down", distance: 4000 },
  ],
  formats: ["json", "markdown"],
});

// One schema, any board layout
const posting = await mcp.extract_structured({
  url: "https://boards.example.com/jobs/senior-backend-engineer",
  schema: {
    type: "object",
    properties: {
      title: { type: "string" },
      location: { type: "string" },
      salaryRange: { type: "string" },
      remote: { type: "boolean" },
      postedAt: { type: "string" },
    },
    required: ["title"],
  },
});

console.log(posting.data);

Tools Used

scrape_with_actions5 credits

extract_structured3 credits

Estimated cost: ~8 credits per board page

Frequently Asked Questions

How do I scrape job boards that use infinite scroll?

Use scrape_with_actions to drive a real browser: wait for the listing selector, click the load-more button, and scroll before extraction runs. The postings render first, so you get real content instead of an empty shell.

Can CrawlForge scrape pages behind a login?

Yes. scrape_with_actions supports type and click actions plus formAutoFill, so it can submit credentials and scrape the page that follows. Only sign in to accounts you own or are authorized to access.

How do I normalize listings from boards with different markup?

Define one JSON schema — title, location, salary range, remote flag, posted date — and pass it to extract_structured. It maps each board's markup onto your shape, and falls back to CSS selectors when no LLM is configured.

What does aggregating a job board cost?

About 8 credits covers a board page: 5 for the scrape_with_actions browser pass and 3 for each extract_structured call on a posting. Use batch_scrape when you are pulling hundreds of detail pages a day.

Ready to Get Started?

Every new account gets 1,000 free credits. No credit card required.

Start Free with 1,000 Credits

Related Use Cases

Lead Enrichment

Enrich sales leads with company data, tech stacks, and contact information from the web. Turn a bare domain list into a qualified, sales-ready pipeline.

search_web (5 cr)extract_metadata (1 cr)

E-commerce Price Monitoring

Track competitor prices and product availability across hundreds of e-commerce sites. Detect undercutting the day it happens and reprice with confidence.

batch_scrape (5 cr)track_changes (3 cr)

Quick Answer

Code Example

// Load a paginated board, then extract each posting as JSON const board = await mcp.scrape_with_actions({ url: "https://boards.example.com/jobs?team=engineering", actions: [ { type: "wait", selector: ".job-card", condition: "visible" }, { type: "click", selector: "button.load-more", clickCount: 3 }, { type: "scroll", direction: "down", distance: 4000 }, ], formats: ["json", "markdown"], }); // One schema, any board layout const posting = await mcp.extract_structured({ url: "https://boards.example.com/jobs/senior-backend-engineer", schema: { type: "object", properties: { title: { type: "string" }, location: { type: "string" }, salaryRange: { type: "string" }, remote: { type: "boolean" }, postedAt: { type: "string" }, }, required: ["title"], }, }); console.log(posting.data);