crawl_deep
Walk a site outward from a starting URL, breadth-first, following links until it hits your page or depth limit. Returns each page it reached with its title, its depth from the start, and how many links it found there.
Use Cases
Map a documentation site
Start at the docs root and discover every reachable page, with depth showing how the information is nested.
Build a URL list before batch scraping
Crawl to discover URLs, then feed them to batch_scrape to extract content — cheaper than crawling with full extraction.
Audit internal linking
The per-page link counts and the depth distribution show which sections are richly linked and which are nearly orphaned.
Check how deep your content really sits
pages_per_depth reveals whether important pages are three or four clicks from the entry point.
Endpoint
/api/v1/tools/crawl_deepParameters
start_url, not url. Every parameter uses snake_case — unknown keys are silently discarded rather than rejected, so a camelCase key like maxDepth will be ignored and the default used instead.| Name | Type | Required | Default | Description |
|---|---|---|---|---|
start_url | string | Required | - | The URL the crawl begins from. Required. Example: https://example.com/docs |
max_pages | number | Optional | 10 | Maximum pages to visit, 1-100. The crawl stops as soon as this many pages have been visited, so it also caps cost and duration. Example: 25 |
max_depth | number | Optional | 3 | Maximum link depth from `start_url`, 1-5. The starting page is depth 0. Example: 2 |
same_domain_only | boolean | Optional | true | When true, only links whose hostname matches `start_url` are followed. External links still count toward the link totals but are never visited. Example: true |
respect_robots_txt | boolean | Optional | true | When true, each origin's robots.txt is fetched once per crawl and any URL it disallows for `CrawlForge` is skipped. A missing or unreachable robots.txt is treated as no restrictions. Older alias for `respect_robots`; both names set the same thing. Example: true |
respect_robots | boolean | Optional | true | Respect each origin's robots.txt. This is the canonical name, shared with the CrawlForge MCP server; `respect_robots_txt` is the older alias and still works — set either one. Left at `true`, a disallowed `start_url` is refused with 403 before anything is fetched and disallowed pages are skipped mid-crawl. Set it to `false` only for a site you have your own agreement with — the response then carries a `warnings` entry and the override is recorded against your API key. Example: true |
crawl_delay | number | Optional | 1000 | Milliseconds to wait between page fetches, 0-5000. Applied from the second page onward. Raise it for small or rate-limited sites. Example: 1000 |
timeout | number | Optional | 30000 | Total crawl budget in milliseconds, 1000-60000, divided evenly across `max_pages` to give each fetch its own timeout. Raising `max_pages` therefore shortens the time any single page is allowed. Example: 30000 |
CrawlForge product token, and robots.txt is respected by default. Each origin's rules are fetched once per crawl and cached. Disallowed URLs are skipped without spending your max_pages budget, and a disallowed start_url returns 403 before anything is fetched — so a blocked crawl costs no credits.How the crawl behaves
Worth knowing before you read the numbers it returns.
max_pages you get a broad shallow map rather than one deep branch.pages and the crawl continues. pages_crawled can be lower than max_pages with no indication of which URLs failed.links is a count, not a listpages reports how many followable links were found there. Use extract_links on a specific page if you need the URLs themselves.Request Examples
# The starting URL parameter is start_url, not url.
curl -X POST https://crawlforge.dev/api/v1/tools/crawl_deep \
-H "X-API-Key: cf_test_YOUR_KEY" \
-H "Content-Type: application/json" \
-d '{
"start_url": "https://example.com/docs",
"max_pages": 25,
"max_depth": 2,
"same_domain_only": true,
"crawl_delay": 1000,
"timeout": 30000
}'Response Example
{ "success": true, "data": { "start_url": "https://example.com/docs", "pages_crawled": 12, "max_depth_reached": 2, "total_links_found": 184, "pages": [ { "url": "https://example.com/docs", "depth": 0, "title": "Documentation", "links": 24 }, { "url": "https://example.com/docs/quickstart", "depth": 1, "title": "Quickstart", "links": 18 }, { "url": "https://example.com/docs/api", "depth": 1, "title": "API Reference", "links": 31 } ], "crawl_stats": { "completed": true, "duration_ms": 8380, "pages_per_depth": { "0": 1, "1": 6, "2": 5 } } }, "credits_used": 4, "credits_remaining": 996, "processing_time": 8420}data.pages_crawledPages actually visited and parsed. Lower than max_pages when pages failed or the site ran out of reachable links.data.max_depth_reachedDeepest level reached. Lower than max_depth means the crawl exhausted the site or hit max_pages first.data.total_links_foundSum of the per-page link counts. Counts duplicates across pages, so it is not a count of distinct URLs.data.pagesOne entry per visited page, in visit order — breadth-first, so depth 0 first, then all of depth 1.data.pages.linksNumber of followable links found on that page, not the URLs. Use extract_links for the list.data.crawl_stats.duration_msTime spent crawling, in milliseconds. Slightly lower than the envelope's processing_time, which also covers request handling.data.crawl_stats.pages_per_depthHow many pages were visited at each depth — the shape of the site as reached from start_url.processing_timeTotal wall-clock time for the crawl, in milliseconds.Error Handling
Missing or invalid start_url (400 VALIDATION_ERROR)
The most common cause is sending url instead of start_url. Unknown keys are discarded, so the request then arrives with no starting URL at all. The details array names the failing field.
Parameter out of range (400 VALIDATION_ERROR)
max_pages must be 1-100, max_depth 1-5, crawl_delay 0-5000, timeout 1000-60000. Values outside those bounds are rejected rather than clamped.
start_url disallowed by robots.txt (403 ROBOTS_DISALLOWED)
The target's robots.txt disallows CrawlForge for that URL. Nothing is fetched and no credits are charged. Set respect_robots: false — or its older alias respect_robots_txt — only where you have your own agreement with the site; the override is recorded against your API key. The override does not reach a host on CrawlForge's permanent opt-out list, which is refused whatever respect_robots is set to.
Crawl failed (500 TOOL_ERROR)
An unexpected failure during the crawl. Individual page failures do not cause this — they are skipped silently — so a 500 means the crawl itself could not proceed.
max_pages is the lever that matters. It caps the work directly, and because timeout is divided across it, a high max_pages with a low timeout gives each page very little time and quietly increases the number that fail.Credit Cost
Cost Breakdown:
Any crawl, 1 to 100 pages: 4 credits
Plan Recommendations:
Free Plan: 1,000 one-time trial credits = 250 crawls
Hobby Plan: 5,000 credits/mo = 1,250 crawls ($19/mo)
Professional Plan: 50,000 credits/mo = 12,500 crawls ($99/mo)
Because the cost is flat, prefer one crawl with a high max_pages over several small crawls of the same site.