AI-PoweredAutonomous8 credits

agent

Autonomous research and extraction from a natural-language prompt — no URLs required. The agent plans its own steps, finds and reads its own sources, and shapes an answer within the hard maxSteps and maxUrls caps you set.

Use Cases

Open-Ended Research

Answer questions that span sites you haven't identified yet — the agent discovers its own sources instead of taking a URL list.

Competitive Snapshots

Ask for a competitor's current pricing tiers or feature set and get a synthesized answer rather than a pile of raw HTML.

Bounded Autonomy

maxSteps (hard cap 10) and maxUrls (hard cap 20) bound every run, so it can never fetch more than the budget you set.

Machine-Readable Answers

Pass a schema when the result feeds a downstream system rather than a human reader, and the agent returns a structured object instead of prose.

Endpoint

POST/api/v1/tools/agent

Auth Required

1 req/s on Free plan

8 credits

Parameters

maxSteps (hard cap 10) and maxUrls (hard cap 20) are enforced by the tool, not suggestions passed to the model. A run can outlast the REST API's ~50-second window (the underlying tool allows up to 120s), so keep both small on REST — or run longer jobs, and the pro model, on the CrawlForge MCP server.

Name	Type	Required	Default	Description
prompt	string	Required	-	Natural-language task or question the agent should answer. 1-2,000 characters. Example: Find the current pricing tiers for the top 3 MCP web-scraping providers
urls	array	Optional	-	Optional seed URLs to fold into the run — the agent still discovers its own sources beyond these. Up to 20. Example: ["https://example.com/pricing"]
schema	object	Optional	-	Optional JSON schema. Provide one to get a structured object back in `answer` instead of prose.
model	string	Optional	"default"	`default` runs the built-in planning loop. `pro` is rejected by the REST API — it needs interactive confirmation, so run pro on the CrawlForge MCP server. Example: default
maxSteps	number	Optional	5	Maximum fetch iterations the agent may run. Hard cap 10. Example: 5
maxUrls	number	Optional	10	Maximum URLs the agent may fetch. Hard cap 20. Example: 10

Request Examples

cURL

terminalBash

curl -X POST https://crawlforge.dev/api/v1/tools/agent \
  -H "X-API-Key: cf_test_YOUR_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "prompt": "Find the current pricing tiers for the top 3 MCP web-scraping providers",
    "maxSteps": 5,
    "maxUrls": 10
  }'

TypeScript

agent.tsTypescript

// npm install crawlforge-sdk
import { CrawlForge } from 'crawlforge-sdk';

const client = new CrawlForge({ apiKey: process.env.CRAWLFORGE_API_KEY });

const result = await client.agent({
  prompt: 'Find the current pricing tiers for the top 3 MCP web-scraping providers',
  maxSteps: 5, // fetch iterations, hard cap 10
  maxUrls: 10, // URLs to fetch, hard cap 20
  // Optional: seed the run with URLs you already trust (max 20)
  // urls: ['https://example.com/pricing'],
  // Optional: pass a JSON schema to get a structured answer instead of prose
  // schema: { type: 'object', properties: { /* ... */ } },
});

// result.data is untyped in crawlforge-sdk 0.1 — its shape is the Response Example below.
const { answer, steps_taken, sources } = result.data as {
  answer: string; steps_taken: number; sources: { url: string; title: string }[];
};
console.log('Answer:', answer);
console.log('Steps taken:', steps_taken);
console.log('Sources read:', sources);

console.log('Credits used:', result.creditsUsed);
console.log('Credits remaining:', result.creditsRemaining);

Python

agent.pyPython

# pip install crawlforge
from crawlforge import CrawlForge

client = CrawlForge()  # reads CRAWLFORGE_API_KEY

result = client.agent(
    prompt='Find the current pricing tiers for the top 3 MCP web-scraping providers',
    maxSteps=5,  # fetch iterations, hard cap 10
    maxUrls=10,  # URLs to fetch, hard cap 20
    # Optional: seed the run with URLs you already trust (max 20)
    # urls=['https://example.com/pricing'],
    # Optional: pass a JSON schema to get a structured answer instead of prose
    # schema={'type': 'object', 'properties': {}},
)

# result.data is a plain dict — its shape is the Response Example below.
print(f"Answer: {result.data['answer']}")
print(f"Steps taken: {result.data['steps_taken']}")
print(f"Sources read: {result.data['sources']}")

print(f"Credits used: {result.credits_used}")
print(f"Credits remaining: {result.credits_remaining}")

Response Example

200 OK13820ms

{
  "success": true,
  "data": {
    "answer": "## Pricing comparison\n\n- **CrawlForge** — Free (1,000 credits), Hobby $19/mo, Professional $99/mo...",
    "sources": [
      {
        "url": "https://example.com/pricing",
        "title": "Example — Pricing"
      },
      {
        "url": "https://example.org/plans",
        "title": "Example Org — Plans"
      }
    ],
    "steps_taken": 3,
    "urls_fetched": 5
  },
  "credits_used": 8,
  "credits_remaining": 992,
  "processing_time": 13820
}

Field Descriptions

data.answerThe synthesized answer — prose by default, or a structured object when you pass a `schema`

data.sourcesEvery source the agent read while answering — use it to audit provenance

data.steps_takenHow many fetch iterations the run actually used

data.urls_fetchedHow many URLs the agent fetched while answering

credits_usedCredits deducted for this run (8 per run, regardless of steps taken)

credits_remainingYour remaining credit balance

Error Handling

Invalid Input (400 Bad Request)

The prompt is missing or outside 1-2,000 characters, maxSteps/maxUrls are out of range, or model is set to pro — which the REST API rejects because it needs interactive confirmation.

Agent Run Failed (500 Internal Server Error)

The run could not be completed. Credits are not deducted for a failed run — retry with a narrower prompt or a smaller maxSteps.

Insufficient Credits (402 Payment Required)

Your account doesn't have enough credits. Purchase more credits or upgrade your plan.

Rate Limit Exceeded (429 Too Many Requests)

You've exceeded your plan's rate limit. Wait a moment or upgrade your plan for higher limits.

Pro Tip: A run can outlast the REST API's ~50-second window, and it costs 8 credits whether it finishes or times out. Keep maxSteps and maxUrls small on REST; for long runs or the pro model, use the CrawlForge MCP server.

Credit Cost

8 credits

8 credits per run

Each agent run costs a flat 8 credits regardless of how many steps it takes or how many URLs it visits — the caps you set bound how much the agent fetches, not the price.

Free Plan: 1,000 one-time credits = 125 runs

Hobby Plan: 5,000 credits/month = 625 runs ($19/mo)

Professional Plan: 50,000 credits/month = 6,250 runs ($99/mo)

Business Plan: 250,000 credits/month = 31,250 runs ($399/mo)

Related Tools

deep_research

Multi-stage research with source verification and synthesis (10 credits)

search_web

Structured web search when you want to pick the sources yourself (5 credits)

extract_with_llm

LLM extraction from a page you already have (3 credits)

scrape

Multi-format extraction from a single known URL (2 credits)

Ready to try agent? Sign up for free and get 1,000 credits to start building.

Use Cases

Open-Ended Research

Answer questions that span sites you haven't identified yet — the agent discovers its own sources instead of taking a URL list.

Competitive Snapshots

Ask for a competitor's current pricing tiers or feature set and get a synthesized answer rather than a pile of raw HTML.

Bounded Autonomy

maxSteps (hard cap 10) and maxUrls (hard cap 20) bound every run, so it can never fetch more than the budget you set.

Machine-Readable Answers

Pass a schema when the result feeds a downstream system rather than a human reader, and the agent returns a structured object instead of prose.

Parameters

Name	Type	Required	Default	Description
prompt	string	Required	-	Natural-language task or question the agent should answer. 1-2,000 characters. Example: Find the current pricing tiers for the top 3 MCP web-scraping providers
urls	array	Optional	-	Optional seed URLs to fold into the run — the agent still discovers its own sources beyond these. Up to 20. Example: ["https://example.com/pricing"]
schema	object	Optional	-	Optional JSON schema. Provide one to get a structured object back in `answer` instead of prose.
model	string	Optional	"default"	`default` runs the built-in planning loop. `pro` is rejected by the REST API — it needs interactive confirmation, so run pro on the CrawlForge MCP server. Example: default
maxSteps	number	Optional	5	Maximum fetch iterations the agent may run. Hard cap 10. Example: 5
maxUrls	number	Optional	10	Maximum URLs the agent may fetch. Hard cap 20. Example: 10

Request Examples

cURL

terminalBash

curl -X POST https://crawlforge.dev/api/v1/tools/agent \
  -H "X-API-Key: cf_test_YOUR_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "prompt": "Find the current pricing tiers for the top 3 MCP web-scraping providers",
    "maxSteps": 5,
    "maxUrls": 10
  }'

TypeScript

agent.tsTypescript

// npm install crawlforge-sdk
import { CrawlForge } from 'crawlforge-sdk';

const client = new CrawlForge({ apiKey: process.env.CRAWLFORGE_API_KEY });

const result = await client.agent({
  prompt: 'Find the current pricing tiers for the top 3 MCP web-scraping providers',
  maxSteps: 5, // fetch iterations, hard cap 10
  maxUrls: 10, // URLs to fetch, hard cap 20
  // Optional: seed the run with URLs you already trust (max 20)
  // urls: ['https://example.com/pricing'],
  // Optional: pass a JSON schema to get a structured answer instead of prose
  // schema: { type: 'object', properties: { /* ... */ } },
});

// result.data is untyped in crawlforge-sdk 0.1 — its shape is the Response Example below.
const { answer, steps_taken, sources } = result.data as {
  answer: string; steps_taken: number; sources: { url: string; title: string }[];
};
console.log('Answer:', answer);
console.log('Steps taken:', steps_taken);
console.log('Sources read:', sources);

console.log('Credits used:', result.creditsUsed);
console.log('Credits remaining:', result.creditsRemaining);

Python

agent.pyPython

# pip install crawlforge
from crawlforge import CrawlForge

client = CrawlForge()  # reads CRAWLFORGE_API_KEY

result = client.agent(
    prompt='Find the current pricing tiers for the top 3 MCP web-scraping providers',
    maxSteps=5,  # fetch iterations, hard cap 10
    maxUrls=10,  # URLs to fetch, hard cap 20
    # Optional: seed the run with URLs you already trust (max 20)
    # urls=['https://example.com/pricing'],
    # Optional: pass a JSON schema to get a structured answer instead of prose
    # schema={'type': 'object', 'properties': {}},
)

# result.data is a plain dict — its shape is the Response Example below.
print(f"Answer: {result.data['answer']}")
print(f"Steps taken: {result.data['steps_taken']}")
print(f"Sources read: {result.data['sources']}")

print(f"Credits used: {result.credits_used}")
print(f"Credits remaining: {result.credits_remaining}")

Response Example

200 OK13820ms

{
  "success": true,
  "data": {
    "answer": "## Pricing comparison\n\n- **CrawlForge** — Free (1,000 credits), Hobby $19/mo, Professional $99/mo...",
    "sources": [
      {
        "url": "https://example.com/pricing",
        "title": "Example — Pricing"
      },
      {
        "url": "https://example.org/plans",
        "title": "Example Org — Plans"
      }
    ],
    "steps_taken": 3,
    "urls_fetched": 5
  },
  "credits_used": 8,
  "credits_remaining": 992,
  "processing_time": 13820
}

Field Descriptions

data.answerThe synthesized answer — prose by default, or a structured object when you pass a `schema`

data.sourcesEvery source the agent read while answering — use it to audit provenance

data.steps_takenHow many fetch iterations the run actually used

data.urls_fetchedHow many URLs the agent fetched while answering

credits_usedCredits deducted for this run (8 per run, regardless of steps taken)

credits_remainingYour remaining credit balance

Error Handling

Invalid Input (400 Bad Request)

The prompt is missing or outside 1-2,000 characters, maxSteps/maxUrls are out of range, or model is set to pro — which the REST API rejects because it needs interactive confirmation.

Agent Run Failed (500 Internal Server Error)

The run could not be completed. Credits are not deducted for a failed run — retry with a narrower prompt or a smaller maxSteps.

Insufficient Credits (402 Payment Required)

Your account doesn't have enough credits. Purchase more credits or upgrade your plan.

Rate Limit Exceeded (429 Too Many Requests)

You've exceeded your plan's rate limit. Wait a moment or upgrade your plan for higher limits.

Credit Cost

8 credits

8 credits per run

Each agent run costs a flat 8 credits regardless of how many steps it takes or how many URLs it visits — the caps you set bound how much the agent fetches, not the price.

Free Plan: 1,000 one-time credits = 125 runs

Hobby Plan: 5,000 credits/month = 625 runs ($19/mo)

Professional Plan: 50,000 credits/month = 6,250 runs ($99/mo)

Business Plan: 250,000 credits/month = 31,250 runs ($399/mo)