Skip to content
AI-PoweredAutonomous8 credits

agent

Autonomous research and extraction from a natural-language prompt — no URLs required. The agent plans its own steps, finds and reads its own sources, and shapes an answer within the hard maxSteps and maxUrls caps you set.

Use Cases

Open-Ended Research

Answer questions that span sites you haven't identified yet — the agent discovers its own sources instead of taking a URL list.

Competitive Snapshots

Ask for a competitor's current pricing tiers or feature set and get a synthesized answer rather than a pile of raw HTML.

Bounded Autonomy

maxSteps (hard cap 10) and maxUrls (hard cap 20) bound every run, so it can never fetch more than the budget you set.

Machine-Readable Answers

Pass a schema when the result feeds a downstream system rather than a human reader, and the agent returns a structured object instead of prose.

Endpoint

POST/api/v1/tools/agent
Auth Required
1 req/s on Free plan
8 credits

Parameters

NameTypeRequiredDefaultDescription
prompt
stringRequired-
Natural-language task or question the agent should answer. 1-2,000 characters.
Example: Find the current pricing tiers for the top 3 MCP web-scraping providers
urls
arrayOptional-
Optional seed URLs to fold into the run. The agent still discovers its own sources beyond these, unless the prompt points at them — "this page", "these URLs", "the given site" — in which case no web search runs and the answer comes from the seed URLs alone. Up to 20.
Example: ["https://example.com/pricing"]
schema
objectOptional-
Optional JSON schema. Provide one to get a structured object back in `answer` instead of prose.
model
stringOptional"default"
`default` runs the built-in planning loop. `pro` is **rejected by the REST API** — it needs interactive confirmation, so run pro on the CrawlForge MCP server.
Example: default
maxSteps
numberOptional5
Maximum fetch iterations the agent may run. Hard cap 10.
Example: 5
maxUrls
numberOptional10
Maximum URLs the agent may fetch. Hard cap 20.
Example: 10
max_inline_chars
numberOptional40000
Largest result to return inline, in characters of its JSON (1,000-10,000,000). Over it the response carries `preview` (the first `max_inline_chars` characters of `answer`, or of the pretty-printed JSON when a `schema` makes `answer` an object), `result_handle`, `total_chars`, `truncated: true` and `expires_at`, and [read_result](/docs/api-reference/tools/read-result) reads the rest for 1 credit per call. Stored results are kept for 1 hour.
Example: 40000

Request Examples

cURL

terminalBash
curl -X POST https://crawlforge.dev/api/v1/tools/agent \
  -H "X-API-Key: cf_test_YOUR_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "prompt": "Find the current pricing tiers for the top 3 MCP web-scraping providers",
    "maxSteps": 5,
    "maxUrls": 10
  }'

TypeScript

agent.tsTypescript
// npm install crawlforge-sdk
import { CrawlForge } from 'crawlforge-sdk';

const client = new CrawlForge({ apiKey: process.env.CRAWLFORGE_API_KEY });

const result = await client.agent({
  prompt: 'Find the current pricing tiers for the top 3 MCP web-scraping providers',
  maxSteps: 5, // fetch iterations, hard cap 10
  maxUrls: 10, // URLs to fetch, hard cap 20
  // Optional: seed the run with URLs you already trust (max 20)
  // urls: ['https://example.com/pricing'],
  // Optional: pass a JSON schema to get a structured answer instead of prose
  // schema: { type: 'object', properties: { /* ... */ } },
});

// result.data is untyped in crawlforge-sdk 0.1 — its shape is the Response Example below.
const { answer, steps_taken, sources } = result.data as {
  answer: string; steps_taken: number; sources: { url: string; title: string }[];
};
console.log('Answer:', answer);
console.log('Steps taken:', steps_taken);
console.log('Sources read:', sources);

console.log('Credits used:', result.creditsUsed);
console.log('Credits remaining:', result.creditsRemaining);

Python

agent.pyPython
# pip install crawlforge
from crawlforge import CrawlForge

client = CrawlForge()  # reads CRAWLFORGE_API_KEY

result = client.agent(
    prompt='Find the current pricing tiers for the top 3 MCP web-scraping providers',
    maxSteps=5,  # fetch iterations, hard cap 10
    maxUrls=10,  # URLs to fetch, hard cap 20
    # Optional: seed the run with URLs you already trust (max 20)
    # urls=['https://example.com/pricing'],
    # Optional: pass a JSON schema to get a structured answer instead of prose
    # schema={'type': 'object', 'properties': {}},
)

# result.data is a plain dict — its shape is the Response Example below.
print(f"Answer: {result.data['answer']}")
print(f"Steps taken: {result.data['steps_taken']}")
print(f"Sources read: {result.data['sources']}")

print(f"Credits used: {result.credits_used}")
print(f"Credits remaining: {result.credits_remaining}")

Response Example

200 OK13820ms
{
"success": true,
"data": {
"answer": "## Pricing comparison\n\n- **CrawlForge** — Free (1,000 credits), Hobby $19/mo, Professional $99/mo...",
"sources": [
{
"url": "https://example.com/pricing",
"title": "Example — Pricing"
},
{
"url": "https://example.org/plans",
"title": "Example Org — Plans"
}
],
"steps_taken": 3,
"urls_fetched": 5
},
"credits_used": 8,
"credits_remaining": 992,
"processing_time": 13820
}
Field Descriptions
data.answerThe synthesized answer — prose by default, or a structured object when you pass a `schema`
data.sourcesEvery source the agent read while answering — use it to audit provenance
data.steps_takenHow many fetch iterations the run actually used
data.urls_fetchedHow many URLs the agent fetched while answering
credits_usedCredits deducted for this run: 8, plus 5 for each stealth retry that got the page (18 at most), regardless of steps taken
credits_remainingYour remaining credit balance

Error Handling

Invalid Input (400 Bad Request)

The prompt is missing or outside 1-2,000 characters, maxSteps/maxUrls are out of range, or model is set to pro — which the REST API rejects because it needs interactive confirmation.

Agent Run Failed (500 Internal Server Error)

The run could not be completed. Credits are not deducted for a failed run — retry with a narrower prompt or a smaller maxSteps.

Insufficient Credits (402 Payment Required)

Your account doesn't have enough credits. Purchase more credits or upgrade your plan.

Rate Limit Exceeded (429 Too Many Requests)

You've exceeded your plan's rate limit. Wait a moment or upgrade your plan for higher limits.

Credit Cost

8 credits
8 credits per run, plus 5 per stealth retry
Each agent run costs 8 credits regardless of how many steps it takes or how many URLs it visits — the caps you set bound how much the agent fetches, not the price. When a page it needs is walled (a challenge page, 403/429, an empty shell or a timeout), the agent retries it in a stealth browser without being asked, and each retry that gets the page adds 5 credits. A retry that is blocked again costs nothing. There are at most 2 retries a run, so a run never costs more than 18 — which is also what is reserved before it starts.

Free Plan: 1,000 one-time credits = 125 runs

Hobby Plan: 5,000 credits/month = 625 runs ($19/mo)

Professional Plan: 100,000 credits/month = 12,500 runs ($99/mo)

Business Plan: 500,000 credits/month = 62,500 runs ($399/mo)

Related Tools